Immersive media data processing method and related equipment

Abstract

An embodiment of the present application provides a method and related apparatus for processing immersive media data. The method comprises: obtaining a media file for the immersive media, the immersive media comprising N interchangeable streams, the media file comprising relationship indication information, wherein the relationship indication information is used to indicate the interchangeable relationship between the N streams, where N is an integer greater than 1; and decoding the media file based on the relationship indication information to present the immersive media. Through the embodiment of the present application, the interchangeable relationship between the N streams of the immersive media can be clearly indicated, thereby guiding the accurate decoding and presentation of the immersive media based on the interchangeable relationship, thereby improving the presentation quality of the immersive media.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to an immersive media data processing method, an immersive media data processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Immersive media can be encoded into interchangeable code streams to meet different presentation requirements for immersive media. For example, two code streams with different encoding qualities but the same content can be interchangeable; another example: two code streams with different encoding types but the same content can be interchangeable. For multiple interchangeable code streams, corresponding instructions are required on the decoding side to guide the decoding and presentation process of the immersive media. However, the existing coding standards for immersive media are not clear and specific enough in terms of the instructions for interchangeable code streams, which affects the presentation effect of the immersive media. Summary of the Invention

[0003] The embodiments of the present application provide an immersive media data processing method and related devices, which can clearly indicate the replaceable relationship between code streams and improve the presentation effect of the immersive media.

[0004] In one aspect, an embodiment of the present application provides a method for processing immersive media data, the method comprising:

[0005] Obtaining a media file of immersive media; the immersive media includes N replaceable code streams; the media file includes relationship indication information, the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1;

[0006] The media file is decoded according to the relationship indication information to present the immersive media.

[0007] In one aspect, an embodiment of the present application provides another method for processing immersive media data, the method comprising:

[0008] Encoding the immersive media to obtain N replaceable code streams;

[0009] generating relationship indication information according to the replaceable relationship between the N code streams, where the relationship indication information is used to indicate the replaceable relationship between the N code streams;

[0010] The relationship indication information and the N code streams are encapsulated to obtain a media file of the immersive media.

[0011] In one aspect, an embodiment of the present application provides an immersive media data processing device, the device comprising:

[0012] An acquisition unit is configured to acquire a media file of immersive media; the immersive media includes N replaceable code streams; the media file includes relationship indication information, the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1;

[0013] The processing unit is configured to decode the media file according to the relationship indication information to present the immersive media.

[0014] In one aspect, an embodiment of the present application provides another immersive media data processing device, the device comprising:

[0015] An encoding unit, configured to encode the immersive media to obtain N replaceable code streams;

[0016] a processing unit, configured to generate relationship indication information according to the replaceable relationship between the N code streams, wherein the relationship indication information is used to indicate the replaceable relationship between the N code streams;

[0017] The processing unit is further configured to encapsulate the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0018] In one aspect, an embodiment of the present application provides a computer device, comprising:

[0019] a processor suitable for executing a computer program;

[0020] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned immersive media data processing method is implemented.

[0021] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is loaded by a processor and executes the above-mentioned immersive media data processing method.

[0022] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the above-mentioned immersive media data processing method is implemented.

[0023] In an embodiment of the present application, a media file of immersive media is obtained. The immersive media includes N interchangeable streams. The media file includes relationship indication information, where the relationship indication information is used to indicate the interchangeable relationship between the N streams, where N is an integer greater than 1. The media file is decoded based on the relationship indication information to present the immersive media. The relationship indication information clearly indicates the interchangeable relationship between the N streams of the immersive media, thereby guiding accurate decoding and presentation of the immersive media based on the interchangeable relationship, thereby improving the presentation quality of the immersive media. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1a is a schematic diagram of a 6DoF provided by an exemplary embodiment of the present application;

[0026] Figure 1b is a schematic diagram of a 3DoF provided by an exemplary embodiment of the present application;

[0027] Figure 1c is a schematic diagram of a 3DoF+ provided by an exemplary embodiment of the present application;

[0028] Figure 2 is an architectural diagram of a data processing system provided by an exemplary embodiment of the present application;

[0029] Figure 3a is a schematic diagram of a packaging result based on single-track packaging provided by an exemplary embodiment of the present application;

[0030] Figure 3b is a schematic diagram of a packaging result of a component-based multi-track packaging provided by an exemplary embodiment of the present application;

[0031] Figure 3c is a schematic diagram of a packaging result of a multi-track packaging based on slicing provided by an exemplary embodiment of the present application;

[0032] Figure 3d is a schematic diagram of another packaging result of multi-track packaging based on slicing provided by an exemplary embodiment of the present application;

[0033] Figure 4a is a flow chart of data processing for immersive media provided by an exemplary embodiment of the present application;

[0034] Figure 4b is a schematic diagram of an immersive media packaging result provided by an exemplary embodiment of the present application;

[0035] Figure 5 This is a flowchart of a data processing method for immersive media provided by an exemplary embodiment of the present application;

[0036] Figure 6aThis is a flowchart of another immersive media data processing method provided by an exemplary embodiment of the present application;

[0037] Figure 6b This is a schematic diagram of the content of a media file provided by an exemplary embodiment of the present application;

[0038] Figure 7 is a schematic diagram of the content of another media file provided by an exemplary embodiment of the present application;

[0039] Figure 8a This is a schematic structural diagram of an immersive media data processing device provided by an exemplary embodiment of the present application;

[0040] Figure 8b is a schematic structural diagram of another immersive media data processing device provided by an exemplary embodiment of the present application;

[0041] Figure 9 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0044] In this application, the term "at least one" means one or more, and "multiple" means two or more; for example, multiple code streams means two or more code streams, and at least one media track means one or more media tracks.

[0045] The following is an introduction to other technical terms involved in this application:

[0046] 1. Immersive Media

[0047] Immersive media refers to media files that can provide immersive media content, allowing viewers immersed in the media content to obtain visual, auditory and other sensory experiences in the real world. Immersive media can also be called immersive media. According to the degree of freedom of viewers when consuming media content, immersive media can be divided into: 6DoF (Degree of Freedom) immersive media, 3DoF immersive media, and 3DoF+ immersive media. Among them, Figure 1a As shown, 6DoF means that viewers of immersive media can freely translate along the X, Y, and Z axes. For example, viewers can move freely within 360-degree VR content. Similar to 6DoF are 3DoF and 3DoF+ production technologies. Figure 1b A schematic diagram of a 3DoF provided in an embodiment of the present application; Figure 1b As shown, 3DoF means that the viewer of immersive media is fixed at the center point of a three-dimensional space, and the viewer's head rotates along the X-axis, Y-axis, and Z-axis to view the images provided by the media content. Figure 1c A schematic diagram of 3DoF+ provided in an embodiment of the present application is shown as follows: Figure 1c As shown, 3DoF+ means that when the virtual scene provided by the immersive media has certain depth information, the head of the immersive media viewer can move within a limited space based on 3DoF to view the images provided by the media content.

[0048] Based on the temporal characteristics of immersive media, immersive media includes sequential and non-sequential immersive media. Sequential immersive media has a temporal order between signals, while non-sequential immersive media does not. Based on the signal characteristics of immersive media, immersive media includes, but is not limited to, volumetric media, volumetric video media, multi-view video media, subtitle media, and audio media. Volumetric media refers to media with three-dimensional content, such as point cloud media (a typical 6DoF immersive media).

[0049] 2. Substitutable Relationship

[0050] Immersive media can be encoded into multiple replaceable code streams. There is a replaceable relationship between different replaceable code streams, and the replaceable relationship refers to a relationship that can replace each other. Based on the replaceable relationship between the replaceable code streams, the N replaceable code streams are allowed to replace each other during presentation. Different replaceable code streams can have the same content and different quality, or the same content and different encoding types. For example, there is a replaceable relationship between code streams of different resolutions obtained by encoding point cloud media. For another example, the code stream obtained by encoding point cloud media using a lossy encoding method and the code stream obtained by using a lossless encoding method are mutually replaceable code streams.

[0051] 3. Point Cloud and Point Cloud Media

[0052] A point cloud is a set of randomly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. Each point in a point cloud includes at least geometric data, which represents the point's three-dimensional position. Depending on the application scenario, a point in a point cloud may also include one or more sets of attribute data. Each set of attribute data reflects a specific attribute of the point, such as color, material, or other information. Typically, each point in a point cloud has the same number of sets of attribute data.

[0053] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes, and are therefore widely used in scenarios such as virtual reality (VR) games, computer-aided design (CAD), geographic information systems (GIS), autonomous navigation systems (ANS), digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive telepresence, and three-dimensional reconstruction of biological tissues and organs.

[0054] Point clouds are primarily acquired through computer generation, three-dimensional (3D) laser scanning, and 3D photogrammetry. Specifically, point clouds can be obtained by capturing real-world visual scenes using a collection device (a set of cameras or a camera device with multiple lenses and sensors). 3D laser scanning can generate point clouds of static real-world 3D objects or scenes, with millions of point clouds acquired per second. 3D photography can generate point clouds of dynamic real-world 3D objects or scenes, with tens of millions of point clouds acquired per second. Furthermore, in the medical field, point clouds of biological tissues and organs can be obtained through magnetic resonance imaging (MRI), computed tomography (CT), and electromagnetic positioning information. Furthermore, point clouds can also be generated directly by computers based on virtual 3D objects and scenes. For example, computers can generate point clouds of virtual 3D objects and scenes. With the continuous accumulation of large-scale point cloud data, efficient storage, transmission, publication, sharing, and standardization of point cloud data have become key to point cloud applications.

[0055] Point cloud media includes a point cloud sequence composed of one or more point cloud frames in sequence, and each point cloud frame is composed of geometric data and attribute data of one or more points in the point cloud. A point in the point cloud may include one or more sets of attribute data, and each set of attribute data is used to reflect an attribute of the point. For example, a point in the point cloud has a set of color attribute data, and the color attribute data is used to reflect the color attribute of the point (such as red, yellow, etc.); for another example, a point in the point cloud has a set of reflectivity attribute data, and the reflectivity attribute data is used to reflect the laser reflection intensity attribute of the point. When a point in the point cloud has multiple sets of attribute data, the types of the multiple sets of attribute data can be the same or different. For example, a point in the point cloud can have a set of color attribute data and a set of reflectivity attribute data; for another example, a point in the point cloud can have two sets of color attribute data, and the two sets of color attribute data are used to reflect the color attributes of the point at different times.

[0056] 4. Track

[0057] A track refers to a collection of media data in the process of media file encapsulation, and a track consists of multiple time-sequential samples. A media file may contain one or more tracks. Schematically, for example, a video media file may include but is not limited to: video media track, audio media track, and subtitle media track. In particular, metadata information can also be included in a media file as a media type in the form of a metadata track. The so-called metadata information is a general term for information related to the presentation of immersive media, and the metadata information may include descriptive information about the media content of the immersive media. In an embodiment of the present application, the time-sequential immersive media is included in the media file of the immersive media in the form of a track, and the track may also be called a media track.

[0058] 5. Sample

[0059] A sample is a unit of encapsulation in a media file. A track is composed of many samples. For example, a video media track can be composed of many samples, with a sample typically being a video frame. In embodiments of the present application, as described above, sequential immersive media can be included in a media file of the sequential immersive media in the form of a track. The track contains one or more samples, and each sample can contain one or more haptic signals within the sequential immersive media.

[0060] 6. Sample Entry

[0061] Sample entries are used to indicate metadata related to all samples in a track. For example, a sample entry for a video media track typically contains metadata related to initializing the consumer device. Another example is a sample entry for a volumetric media track that can contain relationship information indicating the interchangeability between streams.

[0062] 7. Item

[0063] A project is a unit of encapsulation for non-sequential media data during the media file encapsulation process. For example, a static image can be encapsulated as a project. In embodiments of the present application, non-sequential immersive media can be encapsulated as one or more projects. In embodiments of the present application, a project can also be referred to as a media project.

[0064] 8. ISOBMFF (ISO Based Media File Format)

[0065] ISOBMFF is a packaging standard for media files, and a typical ISOBMFF file is an MP4 file.

[0066] 9. DASH (Dynamic Adaptive Streaming over HTTP) DASH is an adaptive bitrate technology that enables high-quality streaming media to be delivered over the Internet through traditional HTTP network servers.

[0067] 10. MPD (Media Presentation Description, Media Presentation Description Signaling in DASH): MPD is used to describe the media segment information in a media file.

[0068] 11. Representation

[0069] A Representation is a combination of one or more media components in DASH. For example, a video file of a certain resolution or time domain level can be considered a Representation.

[0070] 12. Adaptation Sets: Adaptation Sets refers to a set of one or more video streams in DASH. An Adaptation Set can contain multiple Representations. In the embodiments of this application, an Adaptation Set can be referred to as Adaptation.

[0071] Based on the above description, an embodiment of the present application provides a data processing solution for immersive media, which includes a processing flow at an encoding end of the immersive media and a processing flow at a decoding end of the immersive media.

[0072] (1) The processing flow of the encoding end is roughly as follows:

[0073] ① Encode the immersive media to obtain N replaceable code streams of the immersive media, where N is an integer greater than 1; ② Generate relationship indication information based on the replaceable relationship between the N code streams of the immersive media, where the relationship indication information is used to indicate the replaceable relationship between the N code streams; ③ Encapsulate the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0074] (2) The processing flow of the decoding end is roughly as follows:

[0075] ① Obtain a media file of immersive media, where the immersive media includes N replaceable code streams, and the media file includes relationship indication information, where the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1; ② Decode the media file according to the relationship indication information to present the immersive media.

[0076] It can be seen from the above scheme that the present application can add relationship indication information to the media file of the immersive media during the encoding process of the immersive media. The relationship indication information can indicate the replaceable relationship between multiple replaceable code streams of the immersive media. Based on the replaceable relationship, the decoding end can be guided to accurately decode the immersive media, thereby ensuring the presentation accuracy of the immersive media and improving the presentation effect of the immersive media.

[0077] Based on the above description, the following Figure 2 The data processing system suitable for implementing the immersive media provided by the embodiment of the present application is introduced. Figure 2As shown, the data processing system 20 of the immersive media may include a service device 201 and a consumer device 202. The service device 201 may serve as the encoding end of the immersive media; the service device 201 may be a terminal device or a server. The consumer device 202 may serve as the decoding end of the immersive media; the consumer device 202 may be a terminal device or a server. A communication connection may be established between the service device 201 and the consumer device 202. The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, car terminal, smart TV, etc., but is not limited thereto. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0078] The specific process of the data processing of the immersive media by the service device 201 and the consumer device 202 is as follows: For the service device 201, the data processing mainly includes the following processes: (1) the acquisition process of the immersive media; (2) the encoding and file packaging process of the immersive media. For the consumer device 202, the data processing mainly includes the following processes: (3) the decapsulation and decoding process of the immersive media file; (4) the presentation process of the immersive media.

[0079] In addition, the transmission process of immersive media between the service device 201 and the consumer device 202 can be carried out based on various transmission protocols (or transmission signaling). The transmission protocols here may include but are not limited to: DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming media transmission) protocol, HLS (HTTP Live Streaming, dynamic bit rate adaptive transmission) protocol, SMTP (Smart Media Transport Protocol, smart media transmission protocol), TCP (Transmission Control Protocol, transmission control protocol), etc.

[0080] The following is a detailed description of the data processing process of immersive media:

[0081] (1) The acquisition process of immersive media.

[0082] Service device 201 can acquire immersive media, which can be obtained through scene capture or device generation. Scene-captured immersive media refers to immersive media obtained by capturing real-world visual scenes using a capture device associated with service device 201. The capture device is used to provide immersive media acquisition services for service device 201. The capture device may include, but is not limited to, any of the following: a camera, a sensor, or a scanning device. Cameras may include standard cameras, stereo cameras, and light field cameras. Sensors may include lasers and radars. Scanning devices may include 3D laser scanning devices. The capture device associated with service device 201 may refer to a hardware component within service device 201, such as a terminal camera or sensor. It may also refer to a hardware device connected to service device 201, such as a camera connected to service device 201. Device-generated immersive media refers to the service device 201 generating immersive media based on virtual objects (e.g., virtual 3D objects and virtual 3D scenes obtained through 3D modeling). The aforementioned immersive media can be point cloud media or other media, such as multi-view video media, volumetric video media, audio media, tactile media, subtitle media, etc. Among them, tactile media refers to immersive media of the tactile type, which can provide consumers with a tactile sensory experience in the real world.

[0083] (2) The process of encoding and file packaging of immersive media.

[0084] ① The service device 201 may encode the immersive media to obtain N interchangeable codestreams of the immersive media, where N is an integer greater than 1. In one embodiment, the immersive media is point cloud media, and point cloud compression (PCC) may be used to encode the acquired point cloud media to obtain N interchangeable codestreams of the point cloud media. For example, G-PCC (Geometry-based Point Cloud Compression) may be used to encode the geometric data and attribute data in the acquired point cloud media to obtain different versions of the geometry codestream and attribute codestream of the point cloud media.

[0085] ② The service device 201 generates relationship indication information based on the replaceable relationship between the N streams of the immersive media. The replaceable relationship refers to the relationship between any two streams that can replace each other. The generated relationship indication information is used to indicate the replaceable relationship between the N streams.

[0086] Furthermore, after generating the relationship indication information, the service device 201 may encapsulate the relationship indication information and the N code streams of the immersive media to obtain a media file of the immersive media.

[0087] The encapsulation of N streams of immersive media may include the following methods:

[0088] a. If the immersive media is time-sequential immersive media, any of the N code streams of the immersive media may be encapsulated into one or more media tracks.

[0089] The encapsulation of any of the N streams of immersive media can be done using a single-track encapsulation method (one stream is encapsulated into one media track) or a multi-track encapsulation method (one stream is encapsulated into multiple media tracks). The following uses point cloud media as an example to illustrate the encapsulation methods of point cloud streams (i.e., the streams of point cloud media). There are three types of encapsulation methods: (1) single-track encapsulation; (2) component-based multi-track encapsulation; and (3) fragment-based multi-track encapsulation.

[0090] (1) Single-track package

[0091] A media track can be obtained by encapsulating a code stream using a single track encapsulation method. The media track contains a sample entry and at least one sample. Each sample contains parameter information, geometric data, and attribute data. Figure 3a The schematic diagram of the encapsulation result based on single-track encapsulation is shown, and the media track obtained by encapsulating the point cloud code stream stores samples of geometric point clouds.

[0092] (2) Multi-track packaging

[0093] Using multi-track encapsulation to encapsulate a bitstream can yield multiple media tracks. Depending on the encapsulation unit, multi-track encapsulation can include component-based and fragment-based.

[0094] In the component-based multi-track packaging method, the media tracks into which the code stream is packaged include the geometry track and the attribute track. The geometry track contains a sample entry and at least one sample, each sample contains parameter information and geometry data; the attribute track contains a sample entry and at least one sample, each sample contains parameter information and attribute data. Schematically, as shown in Figure 3b A schematic diagram shows the packaging results of a component-based multi-track packaging. Here, the point cloud media is packaged into one geometry component track and two attribute component tracks. Different attribute component tracks contain different attribute data, and the geometry component track is associated with both attribute component tracks.

[0095] In the slice-based multi-track packaging method, the code stream can be encapsulated into slice-based media tracks, including a slice base track and multiple slice tracks. Each sample in the slice base track contains a geometry header and an attribute header; each sample in the slice track contains one or more slices. In one implementation, each slice contains a geometry header, geometry data, an attribute header, and attribute data. Schematically, as shown in the following example: Figure 3c The schematic diagram of the packaging result of the multi-track packaging based on sharding is shown. It includes a sharding base track and two sharding tracks, and the sharding base track is associated with both sharding tracks. In another implementation, the sharding track may include a geometry track and an attribute track. Each shard in the geometry track includes a geometry header and geometry data, and each shard in the attribute track includes an attribute header and attribute data. The geometry track and the attribute track are associated, and the sharding base track may be associated with at least one geometry sharding track. Schematically, as shown in FIG. Figure 3d A schematic diagram of another encapsulation result of multi-track encapsulation based on slicing is shown. It includes 1 slice base track, 2 geometry component tracks, and 2 attribute component tracks. The geometry data and attribute data are encapsulated in slices of different tracks respectively. A geometry component track (such as slice track 1, slice track 1) and an attribute component track (slice track 2, slice track 2) are associated. The association is reflected in the fact that the data in their samples come from the same slice (such as slice 3); the slice base track (slice base track) is associated with each geometry component track (including slice track 1 and slice track 3).

[0096] Any bitstream of immersive media can be encapsulated using a single track or multiple tracks, which is not limited in this application. After encapsulating the media track, the relationship indication information can be added to the corresponding media track to form a media file of the immersive media. Schematically, the relationship indication information can be added to the sample entry of the corresponding media track. Specifically, the setting of the relationship indication information may include the following:

[0097] ① If any of the N codestreams is encapsulated into multiple media tracks, then relationship indication information can be added to any of the multiple media tracks to indicate that the media track and the other media tracks correspond to one codestream. ② If any of the N codestreams is encapsulated into one media track, then relationship indication information can be added to the media track to indicate the replaceable relationship between the codestream to which the media track belongs and the other codestreams in the N codestreams. ③ If at least two of the N codestreams are encapsulated into multiple media tracks, and the media tracks corresponding to any two codestreams contain the same media track, then relationship indication information can be added to the same media track to indicate that the media track is shared by different codestreams.

[0098] b. If the immersive media is non-sequential, any of the N streams in the immersive media can be encapsulated into one or more media items. Furthermore, relationship indication information can be added to the corresponding media items to form a media file for the immersive media. Adding relationship indication information to the corresponding media items is similar to adding relationship indication information to the media track described above and will not be detailed here.

[0099] After obtaining the media file of the immersive media, the service device 201 may send the media file to the consumption device 202 .

[0100] (3) The file decapsulation and decoding process of immersive media.

[0101] The consumption device 202 may obtain the media file of the immersive media and the media presentation description information through the service device 201 . The media presentation description information includes relevant information of the media file of the immersive media.

[0102] The decoding process of the consumer device 202 is the inverse of the encoding process of the server device 201. The consumer device 202 decapsulates the media file according to the immersive media file format requirements, obtaining multiple interchangeable codestreams of the encapsulated immersive media. The consumer device 202 then determines a codestream from the encapsulated immersive media and decodes it to restore the immersive media. As one possible implementation, during the decoding process, the consumer device 202 can obtain relationship indication information from the media file. Based on the interchangeable relationships indicated by the relationship indication information, the consumer device 202 selects the codestream to be presented. Specifically, the consumer device organizes the media tracks / media items corresponding to the codestream and then decodes the media tracks / media items to present the immersive media. As another possible implementation, the immersive media can be transmitted using a streaming method. In this case, the consumer device 202 can obtain transmission signaling (such as DASH or SMT), which contains description information of the relationship indication information. Based on the transmission signaling, the consumer device 202 can determine the media file segments (including one or more media tracks / one or more media items) of the immersive media to be decoded and consumed, and then decode the segments to present the immersive media.

[0103] (4) The presentation process of immersive media.

[0104] The consumer device 202 can render the decoded bitstream. Based on the temporal characteristics of the immersive media, for sequential immersive media, this specifically involves rendering the decoded media tracks. For non-sequential immersive media, this specifically involves rendering the decoded media items. For example, if the immersive media is volumetric video media and includes three media tracks, after decoding each media track, the content of the volumetric video media can be rendered according to the geometric and attribute data contained in the media tracks, allowing playback of the volumetric video media.

[0105] In one embodiment, the present application also provides a flowchart of an immersive media data processing method. Figure 4a The data processing method of the immersive media includes the following steps:

[0106] Service device 201: First, the real-world visual scene A can be sampled through an acquisition device (such as a set of cameras or a camera device with multiple lenses and sensors) to obtain the source data B of the immersive media corresponding to the real-world visual scene. Schematically, if the immersive media is point cloud media, the source data B is a frame sequence composed of a large number of point cloud frames; the acquired immersive media is then encoded to obtain a code stream E, which contains N replaceable code streams; then, relationship indication information can be generated based on the replaceable relationship between the N code streams, and the code stream E and the relationship indication information can be encapsulated to obtain a media file corresponding to the immersive media. In one implementation, during the encapsulation process, the code stream of the immersive media can be encapsulated into one or more media tracks (or media items), and the relationship indication information can be added to the corresponding media tracks (or media items) to form a media file of the immersive media. The service device 201 can synthesize one or more encoded bit streams into a media file F for file playback, or an initialization segment and a sequence of media segments (FS) for streaming according to a specific media container file format; wherein the media container file format can refer to the ISO basic media file format specified in the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) 14496-12.

[0107] In one embodiment, service device 201 may further generate description information of relationship indication information based on the replaceable relationship between the N streams. This description information of the relationship indication information may be sent to consumer device 202 via transmission signaling. Consumer device 202 may then determine whether to use this transmission signaling to obtain the immersive media file based on the transmission method of the media file. Optionally, this transmission signaling may be in the form of a signaling description file.

[0108] Consumer device 202 first receives the immersive media file sent by service device 201. This media file may include: media file F' for file playback, or initialization segment and sequence Fs' of media segments for streaming. The media file is then decapsulated to obtain a codestream E'. Relationship information is then obtained from the media file. Based on the alternative relationship indicated by the relationship information, the desired codestream is determined from N codestreams. This codestream is then decoded to obtain immersive media D'. Consumer device 202 can obtain the streamed initialization segment and media segment Fs' based on transmission signaling. Decoding the codestream specifically involves decoding the corresponding media track / media item.

[0109] In a specific implementation, the consumer device can also determine the media files or media segment sequences required to present the immersive media based on the viewing requirements of the current object (including viewing position / viewing direction); and decode the media files or media segment sequences required to present the immersive media to obtain the immersive media required for presentation. Finally, based on the viewing (viewing window) direction of the current object, the decoded immersive media is rendered to obtain the media frame A' of the immersive media, and the immersive media is presented on the screen of the head-mounted display or any other display device carried by the consumer device according to the presentation time of the media frame. It should be noted that the viewing window of the current object can be determined by various types of sensors (such as head tracking sensors, position tracking sensors, and eye tracking sensors). Among them, in the window-based transmission process, the current viewing position and viewing direction are also passed to the strategy module to determine the track to be received.

[0110] It is understandable that the data processing technology for immersive media involved in this application can be implemented by relying on cloud technology; for example, using a cloud server as a service device. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing. The data processing technology for immersive media provided in this application can be applied to point cloud compression-related products and various links such as the service device end, playback device end, and intermediate nodes in the immersive system.

[0111] In an embodiment of the present application, a service device can obtain immersive media and encode the immersive media to obtain N replaceable code streams, and encapsulate the N code streams and relationship indication information (used to indicate the replaceable relationship between the code streams) to obtain a media file of the immersive media; then, a consumer device can obtain the media file of the immersive media, and according to the replaceable relationship indicated by the relationship indication information in the media file, determine the code stream to be presented from the N code streams of the immersive media, perform decoding processing, and present the immersive media. It can be seen that during the encoding process of the immersive media, relationship indication information can be added to the media file. In this way, the replaceable relationship between the code streams can be clearly indicated through the relationship indication information, thereby effectively guiding the decoding end to more accurately decode and present the immersive media, thereby improving the presentation effect of the immersive media.

[0112] In currently achievable methods, temporal immersive media can be encapsulated as one or more media tracks, and the media tracks include component tracks. The replaceable relationship between code streams can be indicated by the replaceable relationship between component tracks. Component tracks that can be replaced with each other can constitute a track replaceable group. For example, taking the temporal immersive media as volumetric video as an example, when there are replaceable component tracks in the component track of the volumetric video, the replaceable information structure (V3CAlternativeInfoStruct) of the volumetric video can be used to indicate the differences between multiple replaceable component tracks in a track replaceable group. The syntax of the replaceable information structure is shown in Table 1 below.

[0113] Table 1

[0114]

[0115]

[0116] The meanings of the fields in Table 1 above are as follows:

[0117] Quality ranking identification field (quality_ranking_flag): When the value is 1, it indicates that there is a quality replaceable relationship between the component tracks in the track replaceable group; when the value is 0, it indicates that there is no quality replaceable relationship between the replaceable component tracks.

[0118] Coding type identification field (codec_type_flag): When the value is 1, it indicates that there is a coding type replaceable relationship between the component tracks in the track replaceable group; when the value is 0, it indicates that there is no coding type replaceable relationship between the component tracks in the track replaceable group.

[0119] Quality ranking field (quality_ranking): used to indicate quality ranking information. The smaller the value of the quality ranking field, the higher the quality of the corresponding component track.

[0120] Coding type field (codec_type): used to indicate the coding type of the corresponding component track.

[0121] Volumetric video content can be encoded as different versions. These alternative contents are indicated by the alternate group mechanism defined in ISO / IEC 14496-12 (the alternate group field in the TrackHeaderBox). If atlas tracks in different volumetric videos have the same alternate_group value, the volumetric video content corresponding to those atlas tracks is interchangeable.

[0122] Due to the interchangeable relationships between component tracks in volumetric video, interchangeable component tracks belong to the same track interchangeability group. Only one component track in a track interchangeability group can be indexed by the corresponding atlas track or atlas fragment track. A component track contains various data for a video frame, such as geometry and attributes; an atlas track contains images; for example, a video frame is encapsulated as an atlas track in the form of an image. The definition and syntax of a track interchangeability group are shown in Table 2 below.

[0123] Table 2

[0124]

[0125]

[0126] The track alternative group can be indicated by a track group type data box (TrackGroupTypeBox), the type of which is 'valg' and is included in a track group data box (TrackGroupBox). Zero or more TrackGroupTypeBoxes can be set in a media track.

[0127] For non-temporal immersive media, there may be a replaceable relationship between its component items. For example, if the non-temporal immersive media is non-temporal volumetric media, then there is a replaceable relationship between the component items of the non-temporal volumetric media. V3CAlternativeEntityToGroupBox is used to indicate the difference information (such as quality difference information) between the replaceable component items. Among the mutually replaceable component items, only one can be indexed by the corresponding atlas item or atlas slice item. For the definition and grammatical representation of the item replaceable group, please refer to Table 3 below.

[0128] Table 3

[0129]

[0130] The project replaceable group is indicated by an entity group data box (EntityToGroupBox), the type of which is 'valy', and 0, 1 or more entity group data boxes can be set in a component project.

[0131] Based on the above example content related to track replaceable groups in timed immersive media and item replaceable groups in non-timed immersive media, corresponding data boxes can be set according to the corresponding syntax and included in the media file to indicate the replaceable relationship between items / tracks.

[0132] However, in some special scenarios, the replaceable relationship between projects or tracks alone cannot clearly indicate the replaceable relationship between streams, thus affecting the presentation of immersive media. For example, see Figure 4b The encapsulation diagram of the code stream is shown in FIG. Figure 4b As shown in (1), the media file contains two code streams, each of which uses a multi-track encapsulation method to obtain multiple media tracks. When the geometric information in bitstream1 and bitstream2 is completely consistent, that is, the geometric information uses exactly the same encoding method, there is only one geometric track track1 in the media file, and track2 and track3 are interchangeable with each other. Track1+track2 corresponds to bitstream1, and track1+track3 corresponds to bitstream2. At this time, the interchangeable relationship between track2 and track3 and the sharing of geometric tracks indicate the interchangeable relationship between the code streams. However, if there is bitstream3 and the geometric track of bitstream3 is not repeated (that is, it is not shared by multiple code streams), as shown in Figure 4b As shown in (2), it only indicates the replaceable relationship between track2 and track3, and cannot indicate the replaceable relationship between bitstream3 and other code streams. At this time, the indication of the replaceable relationship between code streams is not clear enough.

[0133] It can be seen that simply indicating the interchangeable relationship between tracks cannot cover the interchangeable relationship between codestreams in any scenario, and the indication of the interchangeable relationship between codestreams is not clear enough. Therefore, this application expands the relationship indication information during the encoding process to indicate the interchangeable relationship at the codestream level, supporting use in various file encapsulation scenarios, flexibly organizing the media tracks / media items corresponding to the codestreams, and clearly indicating the interchangeable relationship between codestreams, with high versatility.

[0134] It should be noted that the embodiment of the present application can add several descriptive fields at the system level, including field extensions at the file encapsulation level and field extensions at the signaling message level, to support the implementation steps of the present application. The following examples are given in the form of extending the existing ISOBMFF data box and DASH signaling. Figure 5 , Figure 5 This is a flow chart of an immersive media data processing method provided in an embodiment of the present application. The immersive media data processing method can be executed by a consumer device 202 in an immersive media data processing system. The method includes the following steps S501-S502:

[0135] S501, obtaining a media file of immersive media; the immersive media includes N replaceable code streams; the media file includes relationship indication information, where the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1.

[0136] According to the timing characteristics of the immersive media, the immersive media can be time-sequential immersive media or non-time-sequential immersive media; according to the signal characteristics of the immersive media, the immersive media can be point cloud media or other media, such as any one of multi-perspective video media, audio media, subtitle media, tactile media, volumetric video media, etc. The N code streams of the immersive media have a replaceable relationship. Based on this replaceable relationship, any two code streams can replace each other, that is, they are replaceable code streams. Therefore, each of the N code streams can also be called a replaceable code stream. For example, the immersive media includes 3 replaceable code streams, namely bitstream1, bitstream2 and bitstream3, and the 3 code streams are replaceable versions of the same content but different qualities. Any code stream can be a binary code stream or other base code stream (such as a quaternary code stream, a hexadecimal code stream, etc.). This application does not impose any restrictions on this.

[0137] Next, the configuration of the relationship indication information in the media file and the content indicated by the relationship indication information are explained by taking the case where the time-sequential immersive media is encapsulated as a media track in the media file and the non-time-sequential immersive media is encapsulated as a media item in the media file.

[0138] (1) The immersive media is time-sequential immersive media. The N code streams of the immersive media are encapsulated as M media tracks in the media file, where M is an integer and is greater than or equal to N.

[0139] Since a code stream of time-series immersive media can be encapsulated as one or more media tracks in a media file, the number of media tracks M ≥ the number of replaceable code streams N. The M media tracks include a corresponding number of different media tracks corresponding to the N code streams, and all media tracks corresponding to the N code streams are contained in one media file. For example, the immersive media contains three replaceable code streams, namely bitstream1, bitstream2 and bitstream3, and the media file contains eight media tracks, namely, the three media tracks into which bitstream1 is encapsulated, the four media tracks into which bitstream2 is encapsulated, and the one media track into which bitstream3 is encapsulated. There is no identical media track in the media tracks of the various code streams, that is, the various code streams do not share media tracks.

[0140] Relationship indication information can be set in the media track, specifically within the sample entry of the media track. This information can be considered as metadata for the media track's replaceable information, indicating the replaceable relationship between codestreams. For ease of description, any one of the N codestreams is denoted as codestream i. Any two of the N codestreams can be denoted as codestream i and codestream j, respectively, where i and j are positive integers and both less than or equal to N.

[0141] Based on different encapsulation methods, the relationship indication information settings may include the following (1.1)-(1.3):

[0142] (1.1) Stream i is encapsulated into one media track M among M media tracks i In the example, the relationship indication information is set in the media track M i middle.

[0143] For the code stream i of time-series immersive media, the encoding end can use single-track encapsulation to encapsulate it into a single media track M i . M media tracks contain this media track M i , the media track M i Can be used to represent the code stream i. The relationship indication information is set in the media track M i In the media track M i The replaceable relationship between code stream i and other code streams, where other code streams refer to the code streams other than code stream i among the N code streams.

[0144] (1.2) The code stream i is encapsulated into multiple media tracks in M ​​media tracks, and the relationship indication information is set in the media track M i Medium; Media Track M i Refers to any one of the multiple media tracks into which the codestream i is encapsulated.

[0145] For a stream i of sequential immersive media, a multi-track encapsulation method can also be used at the encoding end to encapsulate it into multiple media tracks. The M media tracks contain multiple media tracks corresponding to the stream i. The encapsulated multiple media tracks can be combined to represent the stream i. Each media track in the multiple media tracks belongs to the stream i. The relationship indication information can be set on any media track in the multiple media tracks, that is, media track M. i middle.

[0146] In a feasible implementation, when the relationship indication information is set in the media track M among the multiple media tracks, i In the case of the relationship indication information, the relationship indication information can not only indicate the replaceable relationship between the code stream i and other code streams to which the media track belongs, but also be used to indicate the relationship between the media track M and the code stream. iThe relationship between other media tracks corresponding to the code stream i; wherein, other media tracks refer to the multiple media tracks to which the code stream i is encapsulated except the media track M i The association relationship is used to indicate the media track M i The above association relationship may refer to the combination relationship between multiple media tracks belonging to the same code stream, through the media track M i The combination of the media track of the code stream i and other media tracks can completely represent the code stream i, and the relationship indication information can include an indication of the association relationship.

[0147] (1.3) Code stream i is encapsulated into multiple media tracks in M ​​media tracks; code stream j is encapsulated into multiple media tracks in M ​​media tracks; if the media tracks in which code stream i is encapsulated and the media tracks in which code stream j is encapsulated both contain media tracks M ij , then the relationship indication information is also used to indicate the media track M ij shared ownership.

[0148] For any two codestreams i and j among N codestreams that use multi-track encapsulation, the number of media tracks encapsulated into codestream i and codestream j can be the same or different, but the number of media tracks encapsulated into codestream i and codestream j is greater than 1. Illustratively, codestream i is encapsulated into 3 of the 8 media tracks, and codestream j is encapsulated into 2 of the 8 media tracks. Furthermore, because some information between different codestreams may be completely consistent, the media tracks obtained by encapsulating different codestreams may contain the same media tracks. For identical media tracks, one can be retained in the media file. In this way, there are no duplicate media tracks among the M media tracks, effectively saving storage resources. Schematically, the three media tracks corresponding to codestream i include geometry track 1, and the two media tracks corresponding to codestream j include geometry track 2; the geometric data in codestream i and codestream j are obtained in exactly the same encoding method, so codestream i and codestream j have the same geometry track, that is, geometry track 1 and geometry track 2 belong to the same media track, then only one geometry track (that is, geometry track 1 or 2) can be retained in the media file.

[0149] Relationship indication information can be set in the media track M ij Indicates the media track M ij The shared ownership relationship is used to represent the media track M ij It is the media track shared by stream i and stream j. ijA media track that belongs to both codestream i and codestream j is also called a shared media track. It is understood that any shared media track can be shared by at least two codestreams. For example, if a media track in a media file is shared by three codestreams, the number of shared media tracks included in the M media tracks can be zero or greater.

[0150] In addition, since the media track M ij It is one of the multiple media tracks corresponding to stream i or stream j. ij The relationship indication information may indicate the following multiple relationships: media track M ij The replaceable relationship between the code stream and other code streams, media track M ij The relationship between the media track M and other media tracks corresponding to the same stream i / stream j. ij As mentioned in (1.2) above, the media track M i When it is a media track shared between different code streams, it is set to the media track M i The relationship indication information may also indicate the above-mentioned multiple relationships.

[0151] As can be seen, the relationship indication information set in the media track can indicate any one or more of the following relationships: the interchangeability between the codestream to which the media track belongs and other codestreams, the association relationship between the media track and other media tracks belonging to the same codestream, and the shared ownership relationship of the media tracks. Indications based on these relationships clearly indicate the interchangeability between codestreams, facilitating the flexible and accurate organization of the media tracks corresponding to the interchangeable codestreams and decoding the media tracks corresponding to the codestream to be presented to present immersive media.

[0152] In one achievable manner, when there are multiple media tracks among the M media tracks that need to be played together, the multiple media tracks that need to be played together belong to the same playback track group.

[0153] According to different joint playback requirements, media tracks can be played together. Media tracks belonging to the same playback track group may belong to the same code stream or to different code streams. Schematically, for code streams that use multi-track encapsulation, a combination of multiple media tracks is required to fully represent the code stream. In this way, multiple media tracks that exist for M media tracks need to be played together. These multiple media tracks that need to be played together belong to the same code stream and can be classified into a playback track group. The combination between media tracks can be indicated by the playback track group. Taking immersive media as time-series volumetric media (such as volumetric video) as an example, the playback track group of volumetric video has the definition and grammatical representation shown in Table 4 below.

[0154] Table 4

[0155]

[0156] For media tracks of volumetric video, only certain combinations of media tracks should be played together. The playback track group of volumetric video can be used to indicate the combinations of media tracks required for joint playback.

[0157] For each media track in a playback track group, the TrackGroupBox (track group data box) of the media track contains a PlayoutTrackGroupBox (extended from TrackGroupTypeBox in ISO / IEC 14496-12, i.e., playback track group data box) carrying a unique track_group_id (track group identifier, used to indicate the identifier of the playback track group). PlayoutTrackGroupBox indicates that the corresponding media track belongs to a media track in a playback track group. For all media tracks in a playback track group, their joint quality level can be optionally defined to indicate media content of different qualities. Among them, the meanings of the various fields involved in the syntax part of Table 4 above are as follows:

[0158] Quality level identification field (quality_ranking_flag): When the value is 1, it indicates that all media tracks in the playback track group of the volumetric video have a joint quality level; when the value is 0, it indicates that all media tracks in the playback track group of the volumetric video do not have a joint quality level.

[0159] Quality ranking field (quality_ranking): used to indicate the joint quality ranking of all media tracks in a playback track group of a volumetric video. The smaller the value of this quality ranking field, the higher the joint quality ranking.

[0160] (2) The immersive media is non-sequential immersive media; N code streams are encapsulated as P media items in the media file; P is an integer and P is greater than or equal to N.

[0161] A code stream of non-time-sequential immersive media can be encapsulated as one or more media items in a media file. P media items include a corresponding number of different media items corresponding to N code streams, and the P media items are included in one media file.

[0162] Relationship indication information can be set in the media item. For ease of description, any one of the N streams is represented as stream i, and any two of the N streams can be represented as stream i and stream j, respectively, where i and j are positive integers and both i and j are less than or equal to N. Based on different encapsulation methods, the relationship indication information settings may include the following (2.1)-(2.3):

[0163] (2.1) The code stream i is encapsulated into one media item P among P media items i In the example, the relationship indication information is set on the media item P i middle.

[0164] The stream i is encapsulated into a single media item P i , and is included in P media items. The media item P i Can be used to represent the code stream i. In this case, the relationship indication information can be set in the media item and can be used to indicate the media track M i The replaceable relationship between code stream i and other code streams, where other code streams refer to the code streams other than code stream i among the N code streams.

[0165] (2.2) The code stream i is encapsulated into multiple media items among P media items, and the relationship indication information is set in the media item P. i In; Media Project P i It refers to any one of the multiple media items into which the code stream i is encapsulated.

[0166] In the case where the code stream i is encapsulated into multiple media items, the relationship indication information can be set in any media item among the multiple media items, that is, the media item P i middle.

[0167] In a feasible implementation, the relationship indication information is also used to indicate the media item P i The relationship between other media items corresponding to the code stream i. Among them, other media items refer to the multiple media items into which the code stream i is encapsulated, except for the media item P i Media items other than P; the association relationship is used to represent the media item P i The above association relationship may refer to the combination relationship between multiple media items corresponding to the same stream. i The combination of the media items and the other media items belonging to the same codestream i (ie, including all media items belonging to the same codestream) can completely represent the codestream i. The relationship indication information includes an indication of the association relationship.

[0168] Optionally, the media item P with relationship indication information is set i It can belong only to stream i, or it can belong to stream i and at least one stream among N streams except stream i. i Belongs to at least two streams, set in the media project P i The relationship indication information in also has the relationship indication in (2.3) below.

[0169] (2.3) Code stream i is encapsulated into multiple media items in P media items; code stream j is encapsulated into multiple media items in P media items. If the media items into which code stream i is encapsulated and the media items into which code stream j is encapsulated both contain media items P ij , then the relationship indication information is also used to indicate the media item P ij Shared ownership relationship; shared ownership relationship is used to represent the media item P ij It is a media item shared by streams i and j.

[0170] For code stream i and code stream j, the number of media items into which code stream i is encapsulated and the number of media items into which code stream j is encapsulated may be the same or different. Schematically, code stream i is encapsulated into 3 media items, and code stream j is encapsulated into 2 media items. Moreover, since some information between different code streams may be completely consistent, each media track obtained by encapsulating different code streams may contain the same media items. For media items that are repeated between different code streams, one can also be retained in the media file, which can effectively save storage resources. For example, a media file includes 7 media items, 3 of which are corresponding to code stream i and 5 of which are corresponding to code stream j. The media items corresponding to code stream i and the media items corresponding to code stream j both contain media item x. Only one of this media item x is retained in the media file and belongs to both code stream i and code stream j.

[0171] Relationship indication information can be set on the media item P ij and can be used to indicate media items P ij The shared ownership relationship is used to represent the media item P ij It is the media track shared by streams i and j. That is, media project P ij Media items belonging to both stream i and stream j are also referred to as shared media items. It is understood that any shared media item can be shared by at least two of the N streams, and the number of shared media items included in the M media items can be zero or greater.

[0172] In addition, due to the media project P ij It is also one of the multiple media items corresponding to stream i or stream j. Therefore, by setting ij The relationship indication information may indicate the following multiple relationships: media item P ij The replaceable relationship between the code stream and other code streams, media item P ij The relationship between other media items corresponding to stream i / stream j, media item P ij As mentioned in (2.2) above, the media item P iFor media tracks shared between different code streams, set the media item P i The relationship indication information may also indicate the above relationship.

[0173] Relationship indications set within media items can indicate any one or more of the following relationships: the interchangeability of the media item's codestream with other codestreams, the association between the media item and other media items belonging to the same codestream, and the shared ownership of the media item. These relationship indications clearly demonstrate the interchangeability between codestreams, facilitating flexible and accurate organization of media items corresponding to interchangeable codestreams and their decoding for immersive media presentation.

[0174] In a feasible implementation, when there are multiple media items among the P media items that need to be played together, the multiple media items that need to be played together belong to the same playback entity group.

[0175] Depending on the different joint playback requirements, different media items can be played together. The media items of the same playback entity group can belong to the same or different code streams. Schematically, there are multiple media items in P media items that can completely represent the code stream. Therefore, multiple media items representing the code stream need to be played together. These multiple media tracks that need to be played together can be classified into a playback entity group. The playback entity group can indicate the combination of media items to be played together. Taking non-sequential immersive media as non-sequential volumetric media as an example, the playback entity group of non-sequential volumetric media is introduced as follows. The playback entity group of non-sequential volumetric media has the definition shown in Table 5 below.

[0176] Table 5

[0177]

[0178] For media items of volumetric media, when only certain specific combinations of media items should be played together, a playback entity group of volumetric media can be used to indicate the combination of these jointly played media items. The playback entity group is represented by a playback entity group data box of the 'eply' type, PlayoutEntityToGroupBox, which can be set in the media item. For all media items of a playback entity group, their joint quality level can be optionally defined to indicate media content of different qualities. The meanings of the various fields involved in the syntax part of Table 5 above are as follows:

[0179] Quality ranking flag field (quality_ranking_flag): When the value is 1, it indicates that all media items of the playback entity group of volumetric media have a joint quality ranking; when the value is 0, it indicates that all media items of the playback entity group of volumetric media do not have a joint quality ranking.

[0180] Quality ranking field (quality_ranking): used to indicate the joint quality ranking of all media items in a playback entity group of a volumetric media. The smaller the value of the quality ranking field, the higher the joint quality ranking.

[0181] In one embodiment, N code streams with an alternative relationship belong to the same alternative group. Different code streams in the same alternative group can be replaced with each other during presentation. The relationship indication information includes an alternative information data box (AlternativeInfoBox). The alternative information data box has the definition shown in Table 6 below.

[0182] Table 6

[0183]

[0184] The Alternative Information Data Box is a newly added data box of type 'alif' and can be set in the sample entry of a media item or media track. That is, the sample entry of a media item or media track can contain an Alternative Information Data Box. The number of Alternative Information Data Boxes can be greater than or equal to 0, that is, 0, 1, or more Alternative Information Data Boxes can be set in a media track / media item, depending on the characteristics of the media track / media item. For example, if media track 1 has two code streams, then two Alternative Information Data Boxes can be set.

[0185] The Alternative Information Data Box can be used to indicate the alternative group to which the corresponding codestream of a media track / media item belongs: a. If the Alternative Information Data Box is set in the current media track, the Alternative Information Data Box contains information about the alternative group to which the corresponding codestream of the current media track belongs. The current media track refers to the media track currently being decoded. b. If the Alternative Information Data Box is set in the current media item, the Alternative Information Data Box contains information about the alternative group to which the corresponding codestream of the current media item belongs. The current media item refers to the media item currently being decoded.

[0186] It is understood that if the immersive media is time-sequential, one or more media tracks in the media file may contain a replacement information data box. If the immersive media is non-time-sequential, one or more media items in the media file may contain a replacement information data box. For any media track or media item being decoded in the media file, if a replacement information data box is present, the replacement information data box indicates the replacement group to which the corresponding bitstream belongs, thereby indicating the replacement relationship between the corresponding bitstreams.

[0187] In one implementation, the alternative information data box includes an alternative group identifier flag field (alternative_group_id_flag) and an alternative group identifier field (alternative_group_id). If the alternative information data box is set in the current media track, the alternative group identifier flag field is used to indicate whether the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs.

[0188] The alternative information data box can be specifically set in the sample entry of the current media track. When the value of the alternative group identifier field is the first preset value (such as "0"), it means that the alternative information data box in the current media track indicates the alternative group identifier to which the corresponding code stream of the current media track belongs. When the value of the alternative group identifier flag field is the second preset value (such as "1"), it means that the alternative information data box in the current media track does not indicate the alternative group identifier to which the corresponding code stream of the current media track belongs. Optionally, if the value of the alternative group identifier flag field is the first preset value (such as "0"), then the alternative group identifier exists in the track header data box (TrackHeaderBox) of the current media track. The alternative group identifier field (alternative_group_id) is used to indicate the alternative group identifier to which the corresponding code stream of the current media track belongs; the alternative group identifier corresponding to different code streams of the same alternative group is the same, and the alternative group identifier can be a value, such as 1.

[0189] If the replaceable information data box is set in the current media project, the replaceable group identification flag field is used to indicate whether the replaceable information data box in the current media project indicates the replaceable group identifier to which the corresponding code stream of the current media project belongs. Based on the different values ​​of the replaceable group identification flag field, the replaceable group identification flag field can indicate different contents. Among them, when the value of the replaceable group identification flag field is a first preset value (such as "0"), it means that the replaceable information data box in the current media project indicates the replaceable group identifier to which the corresponding code stream of the current media project belongs; when the value of the replaceable group identification flag field is a second preset value (such as "1"), it means that the replaceable information data box in the current media project does not indicate the replaceable group identifier to which the corresponding code stream of the current media project belongs; the replaceable group identification field is used to indicate the replaceable group identifier to which the corresponding code stream of the current media project belongs; different code streams of the same replaceable group have the same replaceable group identifier, and the replaceable group identifier can be a value, such as 1; the replaceable group identifier can also be a string of characters, such as aabbxx.

[0190] For alternative codestreams, a media track / media item with an alternative information data box configured with an alternative information data box can indicate the alternative group identifier (AGI) to which the codestream corresponding to the media track / media item belongs, based on the AGI flag field and AGI field contained in the AGI data box. The same AGI field value in the AGI data boxes of different media tracks / media items indicates that the corresponding codestreams belong to the same AGI.

[0191] In one embodiment, the relationship indication information is also used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item. The shared ownership relationship can be indicated in two ways: one is based on the fields in the replaceable information data box, and the other is based on the number of replaceable information data boxes.

[0192] Method 1: indicating based on a field in a replaceable information data box.

[0193] The alternative information data box includes a multi-alternative bitstream flag field (multi_alternative_bitstream_flag) and a bitstream number field (num_bitstream).

[0194] If the alternative information data box is set in the current media track, the multiple alternative code stream flag field is used to indicate whether the current media track belongs to multiple code streams. The above-mentioned alternative information data box can be specifically set in the sample entry of the current media track.

[0195] Among them, when the value of the multiple replacement codestream flag field is a first preset value (such as "0"), it indicates that the current media track belongs to only one codestream; the current media track is a component of a codestream. The current media track can represent a codestream alone, or be combined with other media tracks to represent a codestream. When the value of the multiple replacement codestream flag field is a second preset value (such as "1"), it indicates that the current media track belongs to multiple codestreams. The current media track is also a component of multiple codestreams. The current media track is shared by at least two codestreams. For example, the current media track track1 is both a media track among the multiple media tracks corresponding to the codestream bitstream1, and a media track among the multiple media tracks corresponding to the codestream bitstream2. The codestream number field (num_bitstream) is used to indicate the number of codestreams to which the current media track belongs. That is, when the value of the multiple replacement codestream flag field indicates that the current media track belongs to multiple codestreams, the number of codestreams to which the current media track belongs can be indicated by the codestream number field (num_bitstream). The value of the Number of Streams field is the same as the number of streams in the current media track. For example, if the current media track has K (greater than 1) streams, the value of the Number of Streams field can be K. For example, if the value of multi_alternative_bitstream_flag in the Alternative Information Data Box in the current media track track1 is 1 and num_bitstream is 3, then track1 has three streams, that is, track1 is a media track shared by three streams.

[0196] If the alternative information data box is set in the current media item, the multi-alternative bitstream flag field is used to indicate whether the current media item belongs to multiple bitstreams. When the value of the multi-alternative bitstream flag field (multi_alternative_bitstream_flag) is a first preset value (such as "0"), it means that the current media item belongs to only one bitstream and the current media item is a component of one bitstream. When the value of the multi-alternative bitstream flag field is a second preset value (such as "1"), it means that the current media item belongs to multiple bitstreams and the current media track is a component of multiple bitstreams. The bitstream number field (num_bitstream) is used to indicate the number of bitstreams belonging to the current media item. The value of the bitstream number field is the same as the number of bitstreams belonging to the current media item.

[0197] The multiple replacement codestream flag field and the codestream quantity field can indicate whether the current media track or current media item is shared by multiple codestreams, thereby clarifying the sharing relationship of the current media track / current media item and the number of codestreams belonging to the current media track / current media item.

[0198] Method 2: Based on the indication of the number of replaceable information data boxes.

[0199] If the current media track contains only one replaceable information data box, it indicates that the current media track belongs to only one code stream; if the current media track contains multiple replaceable information data boxes, it indicates that the current media track belongs to multiple code streams, and the number of replaceable information data boxes in the current media track should be equal to the number of code streams to which the current media track belongs.

[0200] The number of alternative information data boxes in the current media track is the same as the number of codestreams to which the current media track belongs, indicating the number of codestreams to which the current media track belongs. By adding multiple alternative information data boxes (AlternativeInfoBox) in the current media track, it can be indicated that the current media track has a shared ownership relationship, that is, the current media track can be shared by multiple codestreams. For a current media track that belongs to only one of the N replaceable codestreams, the current media track contains at most one alternative information data box (AlternativeInfoBox). Here, at most one means 0 or 1. This is because: although the current media track belongs to one codestream, it may be one of the multiple media tracks corresponding to the multi-track encapsulated codestream, rather than a single media track corresponding to the single-track encapsulated codestream. In this case, the current media track may not need to have an alternative information data box.

[0201] If the current media item contains only one Alternative Information Data Box, it indicates that the current media item belongs to only one stream. If the current media item contains multiple Alternative Information Data Boxes, it indicates that the current media item belongs to multiple streams. The number of Alternative Information Data Boxes in the current media item should be equal to the number of streams to which the current media item belongs. For example, if the current media item contains two Alternative Information Data Boxes, it indicates that the current media track belongs to two streams.

[0202] In this approach, the alternative information data box does not include the multiple alternative codestream flag field and / or the codestream quantity field described in approach 1. The shared ownership relationship of the current media track / current media item is indicated based on the number of alternative information data boxes, thereby identifying the shared media track / media item and facilitating the setting of corresponding information in different alternative information data boxes of the current media track / current media item to further indicate the association relationship between the current media track / current media item and the corresponding media track / corresponding media item.

[0203] In one embodiment, the relationship indication information is further used to indicate the association relationship between the current media track and other media tracks belonging to the same code stream as the current media track, or to indicate the association relationship between the current media item and other media items belonging to the same code stream as the current media item.

[0204] Based on the association relationship, the current media track and other media tracks belonging to the same codestream can be associated with each other to represent the corresponding codestream. The Replaceable Information Data Box includes a component reference type field (components_ref_type), which is used to indicate the association between the current media track and other media tracks belonging to the same codestream as the current media track, or the association between the current media item and other media items belonging to the same codestream as the current media item.

[0205] (1) The replaceable information data box is set in the current media track.

[0206] ① If the value of the component reference type field (components_ref_type) is a first preset value (such as "0"), it means that the current media track is associated with other media tracks belonging to the same code stream as the current media track through a track reference, and the replaceable information data box also includes: a track reference type field (track_ref_type), which is used to indicate the type of track reference.

[0207] The track reference is used to associate the current media track with other media tracks. The type of the track reference is indicated by the track reference type field (track_ref_type). If the track reference type fields of different media tracks have the same value, it can indicate that the media tracks are associated with each other.

[0208] ② If the value of the component reference type field (components_ref_type) is the second preset value (such as "1"), it means that the current media track is associated with other media tracks belonging to the same code stream as the current media track through the track group, and the replaceable information data box also includes: a track group type field (track_group_type) and a track group identification field (track_group_id). The track group type field is used to indicate the type of the track group to which the current media track belongs, and the track group identification field is used to indicate the identifier of the track group to which the current media track belongs.

[0209] The track group contains multiple media tracks. Schematically, according to the code stream affiliation, the above track groups can be media tracks corresponding to the same code stream. The media tracks contained in each track group can be combined to represent a code stream. M media tracks can correspond to N track groups. The type of the track group to which the current media track belongs is indicated by the track group type field (track_group_type) contained in the replaceable data box, and the identifier of the track group to which the current media track belongs is indicated by the track group identifier field (track_group_id). The identifier and type of the same track group are the same. The identifier can be a number or a string to indicate that the media tracks in the track group are interrelated.

[0210] (2) A new data box can be replaced and set in the current media project.

[0211] ① If the value of the component reference type field (components_ref_type) is the third preset value (such as "2"), it means that the current media item is associated with other media items belonging to the same code stream as the current media item through an item reference, and the replaceable information data box also includes: an item reference type field (item_ref_type), which is used to indicate the type of the item reference.

[0212] The item reference is used to associate the current media item with other media items. The type of the item reference is indicated by the item_ref_type field. Different media items have the same item_ref_type field value, indicating that the media items can be associated with each other.

[0213] ② If the value of the component reference type field (components_ref_type) is the fourth preset value (such as "3"), it means that the current media item is associated with other media items belonging to the same code stream as the current media item through the entity group, and the replaceable information data box also includes: an entity group type field (track_group_type) and an entity group identification field (track_group_id), the entity group type field is used to indicate the type of the entity group to which the current media item belongs; the entity group identification field is used to indicate the identifier of the entity group to which the current media item belongs.

[0214] The entity group contains one or more media items. Schematically, according to the code stream attribution characteristics, an entity group contains all media items corresponding to a code stream, so that the code stream is represented by the entity group. The type of the entity group to which the current media item belongs is indicated by the entity group type field (track_group_type) contained in the replaceable data box, and the identifier of the entity group to which the current media item belongs is indicated by the entity group identification field (track_group_id). The identifier and type of the same entity group are the same to indicate the association relationship between the media items in the entity group.

[0215] The component reference type field (components_ref_type) allows for the association of the current media track with other media tracks, or the current media item with other media items, based on the values ​​of certain fields. This clearly indicates the combination of media tracks or media items corresponding to the same stream. Furthermore, by combining the indication of associations and shared ownership within the Replaceable Information Data Box, it is possible to indicate the interchangeability between media track combinations, and thus, the interchangeability at the stream level.

[0216] In one embodiment, the replaceable information data box further includes a multi-components flag field (multi_components_flag).

[0217] If the Alternative Information Data Box is set in the current media track, the multi-component flag field (multi_components_flag) indicates whether the codestream to which the current media track belongs is encapsulated into multiple media tracks. The Alternative Information Data Box can be set in the sample entry of the current media track. The multi-component flag field can be used to determine the encapsulation method of the codestream to which the current media track belongs, and thus the component attributes of the codestream to which the current media track belongs. The component attributes indicate whether the current media track is one of the multiple media tracks to which the codestream is encapsulated, or whether the codestream is encapsulated into a single media track.

[0218] When the value of the multi-component flag field (multi_components_flag) is the first preset value (such as "0"), it indicates that the code stream to which the current media track belongs is encapsulated into one media track; the current media track is the media track to which the code stream to which it belongs is encapsulated. The current media track is a component of the code stream encapsulated by a single track, and the corresponding code stream can be represented by the current media track alone. When the value of the multi-component flag field (multi_components_flag) is the second preset value (such as "1"), it indicates that the code stream to which the current media track belongs is encapsulated into multiple media tracks, and the current media track is any one of the multiple media tracks to which the code stream to which it belongs is encapsulated. That is, the code stream to which the current media track belongs adopts a multi-track encapsulation method, and the current media track is one of the multiple media tracks to which it is encapsulated. At this time, the current media track needs to be combined with other media tracks belonging to the same code stream to represent the code stream.

[0219] If the replaceable information data box is set in the current media item, the multi-component flag field (multi_components_flag) is used to indicate whether the code stream belonging to the current media item is encapsulated into multiple media items; when the value of the multi-component flag field is a first preset value (such as "0"), it indicates that the code stream belonging to the current media item is encapsulated into one media item; the current media item is the media item obtained by encapsulating the code stream, and the corresponding code stream can be represented by the current media item alone. The current media item is the media item into which its code stream is encapsulated; the current media item is the media item obtained by encapsulating the code stream, and the corresponding code stream can be represented by the current media item alone. When the value of the multi-component flag field is a second preset value (such as "1"), it indicates that the code stream belonging to the current media item is encapsulated into multiple media items, and the current media item is any one of the multiple media items into which its code stream is encapsulated. At this time, the current media item can be combined with other media items belonging to the same code stream to represent the code stream.

[0220] Based on the above introduction, the syntax representation in the replaceable information data box can be shown in Table 7 below.

[0221] Table 7

[0222]

[0223]

[0224] The meanings of the various fields in the replaceable information data box shown in Table 7 are as follows:

[0225] Alternative Group Identification Flag Field (alternative_group_id_flag): If the alternative information data box is set in the current media track, it is used to indicate whether the alternative information data box in the current media track indicates the alternative group identifier to which the code stream corresponding to the current media track belongs. If the alternative information data box is set in the current media item, it is used to indicate whether the alternative information data box in the current media item indicates the alternative group identifier to which the code stream corresponding to the current media item belongs. Schematically, when the value of alternative_group_id_flag is 1, it indicates that the alternative information data box in the current media track indicates the alternative group identifier to which the code stream corresponding to the current media track belongs. When the value of alternative_group_id_flag is 0, it indicates that the alternative information data box in the current media track does not indicate the alternative group identifier to which the code stream corresponding to the current media track belongs.

[0226] Alternative Group Identifier (alternative_group_id): If the Alternative Information Data Box is set in the current media track, this field indicates the Alternative Group Identifier (AGI) to which the corresponding stream of the current media track belongs. If the Alternative Information Data Box is set in the current media item, this field indicates the Alternative Group Identifier (AGI) to which the corresponding stream of the current media item belongs. Illustratively, this AGI can be 1.

[0227] Multi-alternative stream identification field (multi_alternative_bitstream_flag): If the alternative information data box is set in the current media track, it is used to indicate whether the current media track belongs to multiple bitstreams; if the alternative information data box is set in the current media item, it is used to indicate whether the current media item belongs to multiple bitstreams.

[0228] Number of bitstreams field (num_bitstream): If the alternative information data box is set in the current media track, it is used to indicate the number of bitstreams belonging to the current media track; if the alternative information data box is set in the current media item, it is used to indicate the number of bitstreams belonging to the current media item.

[0229] Schematically, the value of multi_alternative_bitstream_flag is 1, the current media track belongs to multiple streams, and the number of streams is indicated by num_bitstream.

[0230] Component reference field (components_ref_type): If the replaceable information data box is set in the current media track, it is used to indicate the association between the current media track and other media tracks belonging to the same codestream as the current media track; illustratively, the value of components_ref_type is 0, indicating that the current media track is associated with other media tracks belonging to the same codestream through a track reference, and the track reference type field (track_ref_type) is used to indicate the type of track reference. The value of components_ref_type is 1, indicating that the current media track is associated with other media tracks belonging to the same codestream through a track group, and the type of the track group is indicated by the track group type field (track_group_type), and the identifier of the track group is indicated by the track group identifier field (track_group_id). If the replaceable information data box is set in the current media item, it is used to indicate the association between the current media item and other media items belonging to the same codestream as the current media item. Illustratively, a components_ref_type value of 2 indicates that the current media item is associated with other media items belonging to the same stream via an item reference, and the type of the item reference is indicated by the item_ref_type field. A components_ref_type value of 3 indicates that the current media item is associated with other media items belonging to the same stream via an entity group, and the type of the entity group is indicated by the entity group type field (entity_group_type), and the entity group identifier is indicated by the entity group identifier field (entity_group_id).

[0231] Multi-components_flag: If the Alternative Information Data Box is set in the current media track, it indicates whether the codestream of the current media track is encapsulated into multiple media tracks. If the Alternative Information Data Box is set in the current media item, it indicates whether the codestream of the current media item is encapsulated into multiple media items.

[0232] Illustratively, when the multi_alternative_bitstream_flag indicates that the current media track belongs to multiple codestreams (i.e., multi_alternative_bitstream_flag == 1), the codestream number field (num_bitstream) and the component reference field (components_ref_type) are defined. For each codestream belonging to the current media track, the components_ref_type field indicates how the current media track is associated with other media tracks belonging to the same codestream. When the multi_alternative_bitstream_flag indicates that the current media track belongs to a single codestream, the multi_components_flag field further indicates whether the codestream belonging to the current media track is encapsulated into multiple media tracks. When the multi_components_flag indicates that the codestream belonging to the current media track is encapsulated into multiple media tracks (i.e., multi_components_flag == 1), the component reference field (components_ref_type) is defined to further indicate how the current media track and other media tracks belonging to the same codestream are associated.

[0233] In another implementation, if the shared ownership relationship of the current media track / current media item is indicated by the number of replaceable information data boxes, each replaceable information data box has the syntax shown in Table 8 below.

[0234] Table 8

[0235]

[0236]

[0237] Compared with Table 7, the alternative data box includes an alternative group identifier flag field (alternative_group_id_flag), a multi-component flag field (multi_components_flag), and a component reference field (components_ref_type). The relevant descriptions can be found above and are not repeated here. However, the alternative data box does not include a multi-alternative stream flag field (multi_alternative_bitstream_flag) or the corresponding determination content. Illustratively, the alternative information data box shown in Table 8 is set in the current media track and can directly indicate whether the codestream belonging to the current media track is encapsulated into multiple media tracks through the multi-component flag field (multi_components_flag). Furthermore, if the codestream belonging to the current media track is encapsulated into multiple media tracks (multi_components_flag == 1), the component reference field (components_ref_type) is further defined to indicate the current media track and other media tracks belonging to the same codestream as the current media track.

[0238] Optionally, if the immersive media is point cloud media, and the code stream of the point cloud media is a point cloud code stream, the corresponding replaceable information data box can be further simplified by limiting the type. If the code stream to which the current media track / current media project belongs is a point cloud code stream, and the point cloud code stream is encapsulated in a multi-track encapsulation manner, the value of the multi-component flag field is a second preset value (such as "1"). That is to say, when the point cloud media adopts a multi-track encapsulation manner, it can be directly considered that the value of multi_components_flag is 1. Furthermore, when there is a shared media track for multiple replaceable point cloud code streams, it can be directly considered that components_ref_type=1. Thus, multiple media tracks corresponding to a point cloud code stream can be organized through a specific track group. Based on the above content, the judgment when the field values ​​in the replaceable information data box are different can be omitted, thereby simplifying the content in the replaceable data box, improving the efficiency of organizing the media tracks corresponding to multiple replaceable code streams, and saving resources spent on searching media tracks / media projects.

[0239] Depending on the immersive media's transmission method, the method for obtaining the immersive media's media files varies. In one implementation, a consumer device can receive a complete immersive media file, which encapsulates multiple interchangeable streams and relationship information. In another implementation, the immersive media is transmitted using a streaming transmission method. Obtaining the immersive media file includes the following steps: obtaining immersive media transmission signaling, which includes description information of the relationship information; and obtaining the immersive media file based on the transmission signaling.

[0240] The transmission signaling may be DASH signaling, MPD signaling, etc., and the transmission signaling can be obtained by the consumer device in the form of a signaling description file. The description information is used to define the N code streams with an alternative relationship indicated by the relationship indication information.

[0241] Optionally, the description information includes N preselection identifiers (Preselection), each preselection identifier is used to represent one of the N codestreams; each preselection identifier has the same coding identifier (such as @gpccId=1); each preselection identifier corresponds to one or more adaptation sets (Adaptation), and an adaptation set represents a media track or a media item in the codestream represented by each preselection identifier; or each preselection identifier corresponds to one or more representations (Representation), and a representation represents a media track or a media item in the codestream represented by each preselection identifier.

[0242] Schematically, the transmission signaling is DASH signaling, and the preselection identifier Preselection included in the description information can be defined for different components of the same code stream (such as different media tracks / different media items) using the Preselection tool in the DASH signaling, and the Preselection is used to represent one code stream among N code streams. The N preselection identifiers are different to represent different code streams, for example, the preselection identifier Preselection1 corresponds to the code stream bitstream1, and the preselection identifier Preselection2 corresponds to the code stream bitstream2. The interchangeable relationship between the code streams represented by different preselection identifiers can be indicated by the same coding identifier (@gpccId=1). For the code stream represented by the preselection identifier, it can be represented by a combination of the adaptation sets (Adaptation) corresponding to the preselection identifier. An adaptation set contains one identifier. The number of adaptation sets (Adaptation) or the number of representations (Representation) corresponding to a preselection identifier is equal to the number of media tracks / media items of the code stream represented by the preselection identifier. Illustratively, if the pre-selected identifier corresponds to a Representation / Adaptation set, it means that the components of the code stream represented by the pre-selected identifier include a media track or a media item.

[0243] After receiving the transmission signaling, the consumer device can request corresponding media file segments based on its own performance and immersive media presentation requirements, then decode and decapsulate the acquired media file segments to present the immersive media. The performance of the consumer device includes, but is not limited to, supported encoding methods, supported bandwidth, supported processing power of the consumer device's CPU, supported rendering power of the consumer device's GPU, and so on. Presentation requirements include, but are not limited to, presentation clarity, resolution, bitrate, size, viewing angle, viewing direction, and so on.

[0244] S502: Decode the media file according to the relationship indication information to present immersive media.

[0245] In one embodiment, decoding a media file according to the relationship indication information to present immersive media may include the following steps: first, according to the replaceable relationship indicated by the relationship indication information, determining a code stream to be presented from N replaceable code streams; and then decoding and presenting the code stream to be presented.

[0246] As an implementation method, the consumer device can obtain the complete media file. Schematically, the immersive media is a time-sequential immersive media, which contains M media tracks corresponding to N code streams, and the relationship indication information is set in the corresponding media track. The consumer device can first decapsulate the media file to obtain M media tracks, and then determine the code stream to be presented and decode it according to the replaceable relationship indicated by the relationship indication information set in the media track. Here, the code stream to be presented is determined specifically based on the relationship indication information to filter out all media tracks that can represent the code stream, which may be a combination of multiple media tracks or a single media track. In addition, the corresponding media track can also be determined based on the device performance of the consumer device and the presentation requirements of the immersive media. Then, the code stream to be presented is decoded, specifically the selected media track is decoded, so as to present the immersive media. It can be understood that if the code stream to be presented is represented by a media item, then the object of decoding is specifically the media item.

[0247] As another possible implementation, a consumer device can obtain immersive media files based on transmission signaling. These immersive media files are obtained in the form of fragments. Illustratively, a fragment of the media file includes one or more media tracks, which can represent the desired streams from N streams. By decapsulating the fragment of the media file, the one or more media tracks and their relationship indicators can be obtained. Based on this relationship indicator, the media tracks can then be decoded and the immersive media presented. If the fragment of the media file includes a media item, the media item can be decoded to present the immersive media.

[0248] The immersive media data processing method provided in an embodiment of the present application can obtain a media file of immersive media, wherein the immersive media includes N interchangeable streams, the media file includes relationship indication information, and the relationship indication information is used to indicate the interchangeable relationship between the N streams, where N is an integer greater than 1; and the media file is decoded and processed according to the relationship indication information to present the immersive media. The relationship indication information can clearly indicate the interchangeable relationship between any two streams of the immersive media, thereby guiding the accurate decoding and presentation of the immersive media based on the interchangeable relationship, thereby improving the presentation effect of the immersive media. On this basis, based on the encapsulation of the streams, the relationship indication information can be set in the corresponding media track / media item, not only indicating the interchangeable relationship at the stream level, but also further supporting the combination relationship between multiple media tracks (or multiple media items) corresponding to the same stream and / or the shared ownership relationship of media tracks (or media items) shared by different streams. Based on the above relationship indication, for any stream, the media track / media item corresponding to the stream to be presented can be accurately obtained based on the relationship indication information, thereby enabling the presentation of any version of the immersive media content, thereby improving the presentation effect of the immersive media.

[0249] See Figure 6a , Figure 6a This is a flow chart of a method for processing immersive media data provided in an embodiment of the present application. The method can be executed by the service device 201 in the immersive media data processing system. The method includes the following steps S601-S602:

[0250] S601: Encode the immersive media to obtain N replaceable code streams of the immersive media.

[0251] In one implementation, the service device may encode the immersive media using different encoding methods to obtain N replaceable code streams of the immersive media. In this case, the N code streams have the same content and different encoding types. In another implementation, the immersive media may also be encoded according to different quality standards to obtain N replaceable code streams of the immersive media. In this case, the N code streams have the same content and different qualities. The N replaceable code streams can be regarded as different versions of code streams, and any two of the N code streams have a replaceable relationship. Based on the replaceable relationship, different code streams are allowed to replace each other during presentation. The replaceability here includes but is not limited to any one or more of the following: replaceable quality, replaceable encoding type, and replaceable content.

[0252] S602: Generate relationship indication information according to the replaceable relationship between N code streams of the immersive media.

[0253] S603: Encapsulate the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0254] The generated relationship indication information can be used to indicate the replaceable relationship between N code streams. The service device can encapsulate each of the N code streams and add corresponding relationship indication information based on the relationship between the components (such as media tracks / media items) obtained from the encapsulation process and the relationship with the code stream. In one implementation, the immersive media is time-series immersive media, and the service device can encapsulate the N replaceable code streams into M media tracks (M>=N), where each code stream can be encapsulated into one or more media tracks among the M media tracks.

[0255] For ease of description, any of the N code streams can be represented as code stream i. Based on different encapsulation methods of code stream i, the setting of the relationship indication information at the encoding end and the content of the relationship indication information are shown as follows (1)-(3).

[0256] (1) When the code stream i is encapsulated into a media track M i When the media track M i Relationship indication information is added to the , where the relationship indication information can be used to indicate the replaceable relationship between the code stream i and other code streams, where the other code streams refer to the code streams other than the code stream i among the N code streams.

[0257] (2) When codestream i is encapsulated as multiple media tracks, relationship indication information may be added to any of the multiple media tracks. This relationship indication information may also be used to indicate the association between the media track and other media tracks in codestream i, and the relationship indication information may include an indication of this association. In this case, the relationship indication information added to the corresponding media track may be used not only to indicate the interchangeable relationship between codestream i and other codestreams to which the media track belongs, but also to indicate the association between the media track and other media tracks belonging to codestream i, thereby indicating that the combination of multiple media tracks can represent codestream i.

[0258] (3) When at least two replaceable code streams among N code streams are encapsulated as multiple media tracks, and the multiple media tracks encapsulated by different code streams contain the same media track, for the media tracks repeated between different code streams, only one media track may be retained in the media file, and relationship indication information may be added to the media track shared by at least two code streams to indicate that the media track belongs to multiple code streams at the same time. For example, Figure 6bIn the media file shown, when the geometric information in bitstream1 and bitstream2 is completely consistent, that is, the geometric information adopts exactly the same encoding method, a repeated geometric track can be omitted in the media file, that is, only one geometric track track1 is included in the media file, and relationship indication information can be added to the geometric track track1 to identify the geometric track track1 as a shared media track.

[0259] It is understandable that if the above relationship indication information is set in a media track shared by multiple code streams, it can not only indicate that the media track is shared by multiple code streams, but also indicate that the media track and other media tracks belong to the same code stream i. Based on the replaceable relationship between code stream i and other code streams, it can be known that the combination of media tracks and the combination of media tracks representing other code streams or a single media track can be replaced with each other. Figure 6b In the media file shown, tracks track2 and track3 have an interchangeable relationship, and based on the relationship indication information, the interchangeable relationship between any two combinations of track1+track2, track1+track3, and track4+track5 can also be known, thereby achieving the purpose of indicating the interchangeable relationship at the bitstream level and realizing a more accurate and clear indication of the interchangeable relationship between bitstreams.

[0260] By encapsulating the code stream into a media track and adding the relationship indication information to the corresponding media track in the above manner, an immersive media media file can be formed.

[0261] In another implementation, the immersive media is non-sequential, and the service device may encapsulate N interchangeable streams into P media items, where each stream may be encapsulated into one or more of the P media items. Similarly, for stream i of the non-sequential immersive media, the following (4)-(6) relationship indication information settings and content indications apply.

[0262] (4) When the code stream i is encapsulated as a media item M i When you can use the media project M i Relationship indication information is added to the , where the relationship indication information can be used to indicate the replaceable relationship between the code stream i and other code streams, where the other code streams refer to the code streams other than the code stream i among the N code streams.

[0263] (5) When codestream i is encapsulated as multiple media items, relationship indication information may be added to any of the multiple media items. This relationship indication information may also be used to indicate the association between the media item and other media items in codestream i. The relationship indication information includes an indication of this association. In this case, the relationship indication information added to the corresponding media item may be used not only to indicate the interchangeable relationship between codestream i and other codestreams to which the media item belongs, but also to indicate the association between the media item and other media items belonging to the same codestream i, thereby indicating that the combination of multiple media items can represent codestream i.

[0264] (6) When at least two replaceable codestreams among N codestreams are encapsulated into multiple media items, and the multiple media items not encapsulated by the codestreams contain the same media item, for the duplicate media items corresponding to the codestreams, one media item may be retained in the media file, and relationship indication information may be added to the media item to indicate that the media item belongs to multiple codestreams at the same time.

[0265] In an embodiment of the present application, the immersive media is encoded to obtain N code streams of the immersive media, and the N code streams have an interchangeable relationship. Relationship indication information can be generated based on the interchangeable relationship, and the relationship indication information and the N code streams are encapsulated to obtain a media file of the immersive media. It can be seen that in the encoding process of the immersive media, relationship indication information can be added to the media file to indicate the interchangeable relationship between different code streams. In this way, the purpose of indicating the interchangeable relationship at the code stream level is achieved through the relationship indication information. Through the indication of the interchangeable relationship at the code stream level, no matter how many interchangeable code streams are encapsulated in the media file, or what encapsulation method is used for the code stream, the interchangeable relationship between any two interchangeable code streams can be clearly indicated by the relationship indication information set in the corresponding media track / media project. In this way, the number of interchangeable code streams of the immersive media is sufficiently compatible, and the versatility and scalability are strong. Based on this relationship indication information, the decoding end can flexibly organize the media tracks / media items corresponding to the code stream, accurately select the corresponding media tracks / media items, and then guide the decoding and accurate presentation of the immersive media, thereby improving the presentation effect of the immersive media.

[0266] The following is a complete example to explain in detail the data processing method of immersive media provided by this application:

[0267] 1. The service device can obtain immersive media and encode the immersive media to obtain N replaceable code streams.

[0268] 2. The service device generates relationship indication information according to the replaceable relationship between the N code streams.

[0269] 3. The service device encapsulates the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0270] It is assumed here that the immersive media is point cloud media, the bitstream of the immersive media is the point cloud bitstream, and N takes the value of 3. The point cloud media encoding can obtain three point cloud bitstreams, which are recorded as bitstream1, bitstream2, and bitstream3, and the three point cloud bitstreams are replaceable bitstreams with the same content but different quality. Among them, the geometric data of bitstream1 and bitstream2 bitstreams are obtained in exactly the same encoding method, and bitstream1 and bitstream2 bitstreams are encapsulated in a component-based multi-track encapsulation method. In addition, a certain attribute in bitstream1 and bitstream2 (such as reflectivity) is also obtained in exactly the same encoding method, and bitstream3 is encapsulated in a single-track encapsulation method. Then the geometric track and reflectivity attribute track in bitstream1 and bitstream2 are the same media tracks, so only one geometric track and one reflectivity attribute track are retained in the media file. A schematic diagram of the encapsulation results of the media tracks contained in the media file is shown as follows. Figure 7 As shown in the figure, bitstream1 and bitstream2 share Track1's geometry track (GeometryTrack) and Track4's attribute track (AttributeTrack). Bitstream1 and bitstream2 have different color attribute tracks (AttributeTrack(color)). Bitstream1 is represented by Track1, Track2, and Track4, bitstream2 is represented by Track1, Track3, and Track4, and bitstream3 is represented by Track5. Track5 is a track containing geometry and attribute data (Geometry+Attributes Track) obtained by encapsulating bitstream3 in a single track.

[0271] The use of the alternative information metadata and related data boxes contained in the media track can be seen in Table 9 below.

[0272] Table 9

[0273]

[0274]

[0275] Among them, based on the settings of Table 11 above, for Track1, two alternative information data boxes AlternativeInfoBox are set to indicate that Track1 is shared by two code streams (i.e., bitstream1 and bitstream2), and the values ​​of the track group identification field track_group_id in different alternative data boxes are different. In addition, a playback track group data box PlayoutTrackGroupBox is also provided to indicate that it needs to be played in conjunction with other media tracks. The PlayoutTrackGroupBox contains the track group type field track_group_type and the track group identification field track_group_id, and the values ​​are the same as the values ​​of the corresponding fields in the alternative information data box. The values ​​of the track group identification fields of the media tracks belonging to the same track group are the same. Whether a single-track encapsulation method or a multi-track encapsulation method is adopted can be known by the value of the multi-component flag field multi_components_flag (such as a value of 0 indicates single-track encapsulation, and a value of 1 indicates multi-track encapsulation). Track2, Track3, and Track4 all have a PlayoutTrackGroupBox set, indicating they need to be played in conjunction with other media tracks. The specific method of combining tracks is indicated by track_group_id. For example, the track_group_id in Track1, Track2, and Track4 is the same, indicating that these tracks need to be played in conjunction. Similarly, Track1, Track3, and Track4 need to be played in conjunction. Track5 only has one AlternativeInfoBox set, and the Alternative Group Identifier is the same as the Alternative Group Identifier contained in Track1's AlternativeInfoBox, indicating that the corresponding streams are interchangeable.

[0276] 4. The service device can transmit the immersive media files to the consumer device. There are two ways to transmit the media files:

[0277] A. The service device can directly transmit the complete media file F to the consumption device, and the media file includes the relationship indication information.

[0278] B. The service device may transmit one or more segments Fs of the media file (eg, one or more media tracks of the media file) to the consumption device in a streaming transmission manner.

[0279] During streaming, the server generates description information for relationship indication information based on the interchangeability between the streams. This description information may define the N streams with interchangeability indicated by the relationship indication information. The server then transmits this description information to the consumer device via transmission signaling, which may be in the form of a signaling description file. The consumer device determines the interchangeability between the streams based on the description information and then obtains the stream to be presented based on the transmission signaling.

[0280] As in the above example, the service device can generate a signaling description file based on the sharing and replaceable relationship between the geometric track and the attribute track. The signaling description file contains descriptive information of the relationship indication information. Taking DASH as an example, the existing Preselection tool in the DASH signaling can be used to define the different components of the same code stream (corresponding to the media track here) as a Preselection, and add the same encoding identifier @gpccId to the same point cloud content to represent different versions of the point cloud code stream. Specifically, the tracks Track1 to Track5 contained in the media file correspond to the adaptation set / representation Adaptation1 / Representation1 to Adaptation5 / Representation5, respectively. The descriptive information of the relationship indication information is as follows:

[0281] Preselection1:Adaptation1+Adaptation2+Adaptation4;@gpccId=1

[0282] Preselection2:Adaptation1+Adaptation3+Adaptation4;@gpccId=1

[0283] Preselection3:Adaptation5;@gpccId=1

[0284] Adaptation1 is the adaptation set corresponding to track 1, Adaptation2 is the adaptation set corresponding to track 2, Adaptation3 is the adaptation set corresponding to track 3, Adaptation4 is the adaptation set corresponding to track 4, and Adaptation5 is the adaptation set corresponding to track 5. The preselection identifier Preselection corresponds to one bitstream. Different preselection identifiers with @gpccId=1 indicate that different bitstreams can be interchanged.

[0285] 5. The consumption device receives the media file of the immersive media, which contains the relationship indication information.

[0286] 6. The consumption device decodes the media file according to the relationship indication information to present the immersive media.

[0287] Based on different media file transmission methods, the consumer device can receive the complete media file F, or obtain a fragment Fs of the media file based on the transmission signaling. The point cloud file F1 in the above example of the media file is used as an example for explanation.

[0288] (1) The consumer device receives a complete point cloud file F1, which contains all media tracks corresponding to N replaceable code streams. The consumer device can first decapsulate the point cloud file to obtain the media tracks contained therein. Then, based on the data box information set in the media track, it can be learned that there are three options for representing the code stream: ① Track1+Track2+Track4; ② Track1+Track3+Track4; ③ Track5. Then, based on the performance of the consumer device and the presentation requirements for the point cloud media, and combined with the replaceable relationship indicated by the relationship indication information in the point cloud file, the required code stream can be selected for presentation. Specifically, the track in the point cloud file F1 is selected, and then the selected track is decoded based on the corresponding metadata information in the point cloud file to present the point cloud media. In this way, since the complete media file has been obtained in advance, when the consumer device needs to switch between different versions of the code stream, it can directly decode the required code stream for presentation, achieving more efficient switching, thereby improving the presentation efficiency when switching immersive media.

[0289] (2) The consumer device first receives the signaling description file and parses the signaling description file to obtain the description information of the relationship indication information. Based on the description information, it can be known that the consumer device has the following options for representing the code stream:

[0290] Representation1+Representation2+Representation4;

[0291] Representation1+Representation3+Representation4;

[0292] Representation5.

[0293] Here, Representation1 is the representation corresponding to track1, Representation2 is the representation corresponding to track2, Representation3 is the representation corresponding to track3, Representation4 is the representation corresponding to track4, and Representation5 is the representation corresponding to track5. Representation1+Representation2+Representation4 corresponds to bitstream1, Representation1+Representation3+Representation4 corresponds to bitstream2, and Representation5 corresponds to bitstream3.

[0294] Based on the device's capabilities and presentation requirements, the consumer device can request the corresponding transport stream (corresponding to one or more tracks in the point cloud file, i.e., file fragments) Fs based on transmission signaling. The consumer device can then decapsulate the received file fragments and decode the media tracks, ultimately presenting the point cloud media. In this way, the consumer device does not need to receive the entire media file, but instead accurately obtains the required stream based on transmission signaling, thereby reducing the resource consumption of the immersive media presentation.

[0295] It should be understood that for non-temporal immersive media, the media tracks in the examples of the above embodiments are replaced with media items, and the same is applicable.

[0296] In an embodiment of the present application, a service device can obtain immersive media and encode the immersive media to obtain multiple replaceable code streams; then, relationship indication information is generated based on the replaceable relationship between the code streams, and the relationship indication information and the code stream are encapsulated to obtain a media file of the immersive media, and the media file is transmitted to a consumer device. The consumer device can receive the media file, decode the immersive media according to the relationship indication information contained in the media file, and present the immersive media. It can be seen that in the process of encoding the immersive media (specifically, in the process of encapsulating the media file), by adding relationship indication information to the media file of the immersive media, the replaceable relationship between different code streams of the immersive media can be effectively indicated through the relationship indication information, thereby guiding the decoding end to accurately present the immersive media based on its own needs, thereby improving the presentation accuracy and presentation effect of the immersive media.

[0297] Compared to the method of encapsulating mutually replaceable code streams into different media files (such as media file F1 contains the media track corresponding to code stream 1, and media file F2 contains the media track corresponding to code stream 2), the embodiment of the present application is more concise and efficient by encapsulating all the media tracks corresponding to the N code streams of immersive media in one media file, and indicating the replaceable relationship between each code stream based on the relationship indication information. In addition, compared to the method of encapsulating different code streams in one media file, and the same media tracks of the media tracks encapsulated by the tracks of different code streams will appear repeatedly in the media file, this solution only retains one media track shared by at least two code streams in the media file, and indicates the relationship between the media track and the code stream through the relationship indication information. This saves storage resources and can accurately find the media file corresponding to the code stream for decoding. Furthermore, compared to encapsulating different code streams in one media file, omitting repeated media tracks in the media file, and indicating the replaceable relationship between media tracks. This solution is based on the relationship indication information that indicates the replaceable relationship at the bitstream level, rather than the replaceable relationship at the track level / project level. In this way, no matter how many bitstreams are encapsulated in the media file, the corresponding media track / media project can be accurately selected based on the relationship indication information, and then the corresponding bitstream can be decoded to present the immersive media, which has better presentation effects and greater versatility.

[0298] Next, the data processing device for immersive media involved in the embodiments of the present application will be described.

[0299] See Figure 8a , Figure 8a This is a structural diagram of an immersive media data processing device provided in an embodiment of the present application. The immersive media data processing device can be set in the computer device provided in an embodiment of the present application. The computer device can be the consumer device mentioned in the above method embodiment. Figure 8a The data processing device for immersive media shown may be a computer program (including program code) running on a computer device, and the data processing device for immersive media may be used to execute Figure 5 Some or all of the steps in the method embodiment shown. Figure 8a The data processing device of the immersive media may include the following units:

[0300] An acquisition unit 801 is configured to acquire a media file of immersive media; the immersive media includes N replaceable streams; the media file includes relationship indication information, the relationship indication information being used to indicate the replaceable relationship between the N streams, where N is an integer greater than 1;

[0301] The processing unit 802 is configured to decode the media file according to the relationship indication information to present immersive media.

[0302] In one embodiment, the immersive media is time-sequential immersive media; N code streams are encapsulated as M media tracks in the media file; M is an integer and M is greater than or equal to N; and the relationship indication information is set in the media track.

[0303] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into one media track M among the M media tracks. i Relationship indication information is set in the media track M i middle.

[0304] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into multiple media tracks in M ​​media tracks; the relationship indication information is set in media track M i Medium; Media Track M i Refers to any one of the multiple media tracks into which the codestream i is encapsulated.

[0305] In one embodiment, the relationship indication information is also used to indicate the media track M i The relationship between other media tracks corresponding to the code stream i; wherein, other media tracks refer to the multiple media tracks to which the code stream i is encapsulated except the media track M i Media tracks other than M; the association relationship is used to indicate the media track M i Belongs to the same codestream i as other media tracks.

[0306] In one embodiment, any two code streams among N code streams are represented as code stream i and code stream j, where i and j are positive integers and both are less than or equal to N; code stream i is encapsulated into multiple media tracks among M media tracks; code stream j is encapsulated into multiple media tracks among M media tracks; if the media track into which code stream i is encapsulated and the media track into which code stream j is encapsulated both contain media track M ij , then the relationship indication information is also used to indicate the media track M ij Shared ownership relationship; shared ownership relationship is used to represent the media track M ij It is the media track shared by streams i and j.

[0307] In one embodiment, the immersive media is non-sequential immersive media; N code streams are encapsulated as P media items in the media file; P is an integer and P is greater than or equal to N; and the relationship indication information is set in the media item.

[0308] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into multiple media items in P media items; relationship indication information is set in media item P. i In; Media Project P i It refers to any one of the multiple media items into which the code stream i is encapsulated.

[0309] In one embodiment, the relationship indication information is also used to indicate the media item P i The relationship between the other media items corresponding to the code stream i; wherein the other media items refer to the multiple media items into which the code stream i is encapsulated except the media item P i Media items other than P; the association relationship is used to represent the media item P i Belong to the same stream i as other media items.

[0310] In one embodiment, the relationship indication information is also used to indicate the media item P i The association relationship between other media items corresponding to stream i;

[0311] Among them, other media items refer to the multiple media items into which the code stream i is encapsulated except the media item P i Media items other than P; the association relationship is used to represent the media item P i Belong to the same stream i as other media items.

[0312] In one embodiment, any two code streams among the N code streams are represented as code stream i and code stream j, where i and j are positive integers and both are less than or equal to N; code stream i is encapsulated into multiple media items among P media items; code stream j is encapsulated into multiple media items among P media items; if the media items into which code stream i is encapsulated and the media items into which code stream j is encapsulated both contain media item P, ij , then the relationship indication information is also used to indicate the media item P ij Shared ownership relationship; shared ownership relationship is used to represent the media item P ij It is a media item shared by streams i and j.

[0313] In one embodiment, if the immersive media is time-series immersive media, N streams are encapsulated as M media tracks in the media file; M is an integer and is greater than or equal to N; wherein, when multiple media tracks among the M media tracks need to be played together, the multiple media tracks that need to be played together belong to the same playback track group;

[0314] If the immersive media is non-sequential immersive media, N code streams are encapsulated as P media items in the media file; P is an integer and P is greater than or equal to N; among them, when there are multiple media items among the P media items that need to be played jointly, the multiple media items that need to be played jointly belong to the same playback entity group.

[0315] In one embodiment, N code streams with a replaceable relationship belong to the same replaceable group, and different code streams in the same replaceable group can be replaced with each other during presentation; the relationship indication information includes a replaceable information data box;

[0316] If the alternative information data box is set in the current media track, the alternative information data box contains information about the alternative group to which the code stream corresponding to the current media track belongs;

[0317] If the replaceable information data box is set in the current media item, the replaceable information data box contains information about the replaceable group to which the corresponding code stream of the current media item belongs;

[0318] The current media track refers to the media track being decoded; the current media item refers to the media item being decoded.

[0319] In one embodiment, the replaceable information data box includes a replaceable group identification flag field and a replaceable group identification field;

[0320] If the alternative information data box is set in the current media track, the alternative group identifier flag field is used to indicate whether the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs. When the value of the alternative group identifier flag field is a first preset value, it means that the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs. When the value of the alternative group identifier flag field is a second preset value, it means that the alternative information data box in the current media track does not indicate the alternative group identifier to which the codestream corresponding to the current media track belongs. The alternative group identifier field is used to indicate the alternative group identifier to which the codestream corresponding to the current media track belongs.

[0321] If the replaceable information data box is set in the current media item, the replaceable group identification flag field is used to indicate whether the replaceable information data box in the current media item indicates the replaceable group identifier to which the corresponding code stream of the current media item belongs; when the value of the replaceable group identification flag field is a first preset value, it means that the replaceable information data box in the current media item indicates the replaceable group identifier to which the corresponding code stream of the current media item belongs; when the value of the replaceable group identification flag field is a second preset value, it means that the replaceable information data box in the current media item does not indicate the replaceable group identifier to which the corresponding code stream of the current media item belongs; the replaceable group identification field is used to indicate the replaceable group identifier to which the corresponding code stream of the current media item belongs.

[0322] In one embodiment, the relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item; the alternative information data box includes a multiple alternative code stream flag field and a code stream quantity field;

[0323] If the alternative information data box is set in the current media track, the multiple alternative codestream flag field is used to indicate whether the current media track belongs to multiple codestreams. When the value of the multiple alternative codestream flag field is a first preset value, it indicates that the current media track belongs to only one codestream. When the value of the multiple alternative codestream flag field is a second preset value, it indicates that the current media track belongs to multiple codestreams. The codestream number field is used to indicate the number of codestreams belonging to the current media track.

[0324] If the alternative information data box is set in the current media item, the multiple alternative codestream flag field is used to indicate whether the current media item belongs to multiple codestreams. When the value of the multiple alternative codestream flag field is a first preset value, it means that the current media item belongs to only one codestream. When the value of the multiple alternative codestream flag field is a second preset value, it means that the current media item belongs to multiple codestreams. The codestream number field is used to indicate the number of codestreams belonging to the current media item.

[0325] In one embodiment, the relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item;

[0326] If the current media track contains only one alternative information data box, it indicates that the current media track belongs to only one codestream; if the current media track contains multiple alternative information data boxes, it indicates that the current media track belongs to multiple codestreams, and the number of alternative information data boxes in the current media track should be equal to the number of codestreams to which the current media track belongs; or

[0327] If the current media item contains only one replaceable information data box, it indicates that the current media item belongs to only one code stream; if the current media item contains multiple replaceable information data boxes, it indicates that the current media item belongs to multiple code streams; the number of replaceable information data boxes in the current media item should be equal to the number of code streams to which the current media item belongs.

[0328] In one embodiment, the relationship indication information is further used to indicate the association relationship between the current media track and other media tracks belonging to the same code stream as the current media track, or to indicate the association relationship between the current media item and other media items belonging to the same code stream as the current media item.

[0329] The replaceable information data box includes a component reference type field, which is used to indicate the association between the current media track and other media tracks belonging to the same codestream as the current media track, or to indicate the association between the current media item and other media items belonging to the same codestream as the current media item.

[0330] When the value of the component reference type field is the first preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track reference, and the replaceable information data box also includes a track reference type field, which is used to indicate the type of the track reference;

[0331] When the value of the component reference type field is the second preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track group, and the replaceable information data box further includes a track group type field and a track group identifier field. The track group type field is used to indicate the type of track group to which the current media track belongs; and the track identifier field is used to indicate the identifier of the track group to which the current media track belongs.

[0332] When the value of the component reference type field is a third preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an item reference, and the replaceable information data box further includes an item reference type field, which is used to indicate the type of the item reference;

[0333] When the value of the component reference type field is the fourth preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an entity group, and the replaceable information data box also includes an entity group type field and an entity group identification field. The entity group type field is used to indicate the type of the entity group to which the current media item belongs; the entity group identification field is used to indicate the identifier of the entity group to which the current media item belongs.

[0334] In one embodiment, the replaceable information data box further includes a multi-component flag field;

[0335] If the replaceable information data box is set in the current media track, the multi-component flag field is used to indicate whether the code stream belonging to the current media track is encapsulated into multiple media tracks. When the value of the multi-component flag field is a first preset value, it indicates that the code stream belonging to the current media track is encapsulated into one media track, and the current media track is the media track into which the code stream belonging to the current media track is encapsulated. When the value of the multi-component flag field is a second preset value, it indicates that the code stream belonging to the current media track is encapsulated into multiple media tracks, and the current media track is any one of the multiple media tracks into which the code stream belonging to the current media track is encapsulated.

[0336] If the replaceable information data box is set in the current media item, the multi-component flag field is used to indicate whether the code stream belonging to the current media item is encapsulated into multiple media items. When the value of the multi-component flag field is a first preset value, it indicates that the code stream belonging to the current media item is encapsulated into one media item, and the current media item is the media item into which the code stream belonging to the current media item is encapsulated. When the value of the multi-component flag field is a second preset value, it indicates that the code stream belonging to the current media item is encapsulated into multiple media items, and the current media item is any one of the multiple media items into which the code stream belonging to the current media item is encapsulated.

[0337] Among them, if the code stream belonging to the current media track / current media project is a point cloud code stream, and the point cloud code stream is encapsulated in a multi-track encapsulation manner, the value of the multi-component flag field is the second preset value.

[0338] In one embodiment, the immersive media is transmitted in a streaming manner. The acquisition unit 801 is specifically configured to: acquire transmission signaling of the immersive media, where the transmission signaling includes description information of the relationship indication information; and acquire a media file of the immersive media according to the transmission signaling.

[0339] In one embodiment, the description information is used to define N codestreams having an alternative relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each of which is used to represent one of the N codestreams; each preselected identifier has the same encoding identifier; each preselected identifier corresponds to one or more adaptation sets, each adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; or each preselected identifier corresponds to one or more representations, each representation represents a media track or a media item in the codestream represented by each preselected identifier.

[0340] In one embodiment, the processing unit 802 is specifically configured to: determine a code stream to be presented from N replaceable code streams according to the replaceable relationship indicated by the relationship indication information; and decode and present the code stream to be presented; wherein the immersive media includes any one or more of the following: volumetric media, volumetric video media, multi-view video media, subtitle media, and audio media.

[0341] In an embodiment of the present application, an immersive media decoding terminal may obtain a media file of the immersive media, wherein the immersive media includes N interchangeable streams and includes relationship indication information, where the relationship indication information is used to indicate the interchangeable relationship between the N streams, where N is an integer greater than 1. The media file is decoded based on the relationship indication information to present the immersive media. The relationship indication information can clearly indicate the interchangeable relationship between the N streams of the immersive media, thereby guiding accurate decoding and presentation of the immersive media based on the interchangeable relationship, thereby improving the presentation quality of the immersive media.

[0342] See Figure 8b , Figure 8b This is a structural diagram of an immersive media data processing device provided in an embodiment of the present application. The immersive media data processing device can be set in the computer device provided in an embodiment of the present application. The computer device can be the service device mentioned in the above method embodiment. Figure 8b The data processing device for immersive media shown may be a computer program (including program code) running on a computer device, and the data processing device for immersive media may be used to execute Figure 6a Some or all of the steps in the method embodiment shown. Figure 8b The data processing device of the immersive media may include the following units:

[0343] The encoding unit 811 is configured to encode the immersive media to obtain N replaceable code streams;

[0344] A processing unit 812 is configured to generate relationship indication information according to the replaceable relationship between the N code streams, where the relationship indication information is used to indicate the replaceable relationship between the N code streams;

[0345] The processing unit 812 is further configured to encapsulate the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0346] In an embodiment of the present application, the encoding end of the immersive media can encode the immersive media to obtain N code streams of the immersive media, and the N code streams have an interchangeable relationship. According to the interchangeable relationship, relationship indication information for indicating the interchangeable relationship can be generated, and the relationship indication information and the N code streams can be encapsulated to obtain a media file of the immersive media. It can be seen that in the encoding process of the immersive media, relationship indication information can be added to the media file to indicate the interchangeable relationship between different code streams. In this way, the purpose of indicating the interchangeable relationship at the code stream level is achieved through the relationship indication information. Based on the relationship indication information, the decoding of the immersive media can be guided to be accurately presented, thereby improving the presentation effect of the immersive media.

[0347] Next, the consumer device and service device provided in the embodiments of the present application are described.

[0348] Furthermore, the present invention also provides a schematic diagram of the structure of a computer device. Figure 9 The computer device may include a processor 901, an input device 902, an output device 903, and a memory 904. The processor 901, input device 902, output device 903, and memory 904 are connected via a bus. The memory 904 is used to store a computer program, which includes program instructions. The processor 901 is used to execute the program instructions stored in the memory 904.

[0349] In one embodiment, the computer device may be the aforementioned consumer device; in this embodiment, the processor 901 executes the following operations by running the executable program code in the memory 904:

[0350] Obtaining a media file of immersive media; the immersive media includes N replaceable code streams; the media file includes relationship indication information, the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1;

[0351] The media file is decoded according to the relationship indication information to present the immersive media.

[0352] In one embodiment, the immersive media is time-sequential immersive media; N code streams are encapsulated as M media tracks in the media file; M is an integer and M is greater than or equal to N; and the relationship indication information is set in the media track.

[0353] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into one media track M among the M media tracks. i Relationship indication information is set in the media track M i middle.

[0354] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into multiple media tracks in M ​​media tracks; the relationship indication information is set in media track M i Medium; Media Track M i Refers to any one of the multiple media tracks into which the codestream i is encapsulated.

[0355] In one embodiment, the relationship indication information is also used to indicate the media track M i The relationship between other media tracks corresponding to the code stream i; wherein, other media tracks refer to the multiple media tracks to which the code stream i is encapsulated except the media track M i Media tracks other than M; the association relationship is used to indicate the media track M i Belongs to the same codestream i as other media tracks.

[0356] In one embodiment, any two code streams among N code streams are represented as code stream i and code stream j, where i and j are positive integers and both are less than or equal to N; code stream i is encapsulated into multiple media tracks among M media tracks; code stream j is encapsulated into multiple media tracks among M media tracks; if the media track into which code stream i is encapsulated and the media track into which code stream j is encapsulated both contain media track M ij , then the relationship indication information is also used to indicate the media track M ij Shared ownership relationship; shared ownership relationship is used to represent the media track M ij It is the media track shared by streams i and j.

[0357] In one embodiment, the immersive media is non-sequential immersive media; N code streams are encapsulated as P media items in the media file; P is an integer and P is greater than or equal to N; and the relationship indication information is set in the media item.

[0358] In one embodiment, any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; code stream i is encapsulated into multiple media items in P media items; relationship indication information is set in media item P. i In; Media Project P i It refers to any one of the multiple media items into which the code stream i is encapsulated.

[0359] In one embodiment, the relationship indication information is also used to indicate the media item P i The relationship between the other media items corresponding to the code stream i; wherein the other media items refer to the multiple media items into which the code stream i is encapsulated except the media item P i Media items other than P; the association relationship is used to represent the media item P iBelong to the same stream i as other media items.

[0360] In one embodiment, the relationship indication information is also used to indicate the media item P i The association relationship between other media items corresponding to stream i;

[0361] Among them, other media items refer to the multiple media items into which the code stream i is encapsulated except the media item P i Media items other than P; the association relationship is used to represent the media item P i Belong to the same stream i as other media items.

[0362] In one embodiment, any two code streams among the N code streams are represented as code stream i and code stream j, where i and j are positive integers and both are less than or equal to N; code stream i is encapsulated into multiple media items among P media items; code stream j is encapsulated into multiple media items among P media items; if the media items into which code stream i is encapsulated and the media items into which code stream j is encapsulated both contain media item P, ij , then the relationship indication information is also used to indicate the media item P ij Shared ownership relationship; shared ownership relationship is used to represent the media item P ij It is a media item shared by streams i and j.

[0363] In one embodiment, if the immersive media is time-series immersive media, N streams are encapsulated as M media tracks in the media file; M is an integer and is greater than or equal to N; wherein, when multiple media tracks among the M media tracks need to be played together, the multiple media tracks that need to be played together belong to the same playback track group;

[0364] If the immersive media is non-sequential immersive media, N code streams are encapsulated as P media items in the media file; P is an integer and P is greater than or equal to N; among them, when there are multiple media items among the P media items that need to be played jointly, the multiple media items that need to be played jointly belong to the same playback entity group.

[0365] In one embodiment, N code streams with a replaceable relationship belong to the same replaceable group, and different code streams in the same replaceable group can be replaced with each other during presentation; the relationship indication information includes a replaceable information data box;

[0366] If the alternative information data box is set in the current media track, the alternative information data box contains information about the alternative group to which the code stream corresponding to the current media track belongs;

[0367] If the replaceable information data box is set in the current media item, the replaceable information data box contains information about the replaceable group to which the corresponding code stream of the current media item belongs;

[0368] The current media track refers to the media track being decoded; the current media item refers to the media item being decoded.

[0369] In one embodiment, the replaceable information data box includes a replaceable group identification flag field and a replaceable group identification field;

[0370] If the alternative information data box is set in the current media track, the alternative group identifier flag field is used to indicate whether the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs. When the value of the alternative group identifier flag field is a first preset value, it means that the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs. When the value of the alternative group identifier flag field is a second preset value, it means that the alternative information data box in the current media track does not indicate the alternative group identifier to which the codestream corresponding to the current media track belongs. The alternative group identifier field is used to indicate the alternative group identifier to which the codestream corresponding to the current media track belongs.

[0371] If the replaceable information data box is set in the current media item, the replaceable group identification flag field is used to indicate whether the replaceable information data box in the current media item indicates the replaceable group identifier to which the corresponding code stream of the current media item belongs; when the value of the replaceable group identification flag field is a first preset value, it means that the replaceable information data box in the current media item indicates the replaceable group identifier to which the corresponding code stream of the current media item belongs; when the value of the replaceable group identification flag field is a second preset value, it means that the replaceable information data box in the current media item does not indicate the replaceable group identifier to which the corresponding code stream of the current media item belongs; the replaceable group identification field is used to indicate the replaceable group identifier to which the corresponding code stream of the current media item belongs.

[0372] In one embodiment, the relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item; the alternative information data box includes a multiple alternative code stream flag field and a code stream quantity field;

[0373] If the alternative information data box is set in the current media track, the multiple alternative codestream flag field is used to indicate whether the current media track belongs to multiple codestreams. When the value of the multiple alternative codestream flag field is a first preset value, it indicates that the current media track belongs to only one codestream. When the value of the multiple alternative codestream flag field is a second preset value, it indicates that the current media track belongs to multiple codestreams. The codestream number field is used to indicate the number of codestreams belonging to the current media track.

[0374] If the alternative information data box is set in the current media item, the multiple alternative codestream flag field is used to indicate whether the current media item belongs to multiple codestreams. When the value of the multiple alternative codestream flag field is a first preset value, it means that the current media item belongs to only one codestream. When the value of the multiple alternative codestream flag field is a second preset value, it means that the current media item belongs to multiple codestreams. The codestream number field is used to indicate the number of codestreams belonging to the current media item.

[0375] In one embodiment, the relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item;

[0376] If the current media track contains only one alternative information data box, it indicates that the current media track belongs to only one codestream; if the current media track contains multiple alternative information data boxes, it indicates that the current media track belongs to multiple codestreams, and the number of alternative information data boxes in the current media track should be equal to the number of codestreams to which the current media track belongs; or

[0377] If the current media item contains only one replaceable information data box, it indicates that the current media item belongs to only one code stream; if the current media item contains multiple replaceable information data boxes, it indicates that the current media item belongs to multiple code streams; the number of replaceable information data boxes in the current media item should be equal to the number of code streams to which the current media item belongs.

[0378] In one embodiment, the relationship indication information is further used to indicate the association relationship between the current media track and other media tracks belonging to the same code stream as the current media track, or to indicate the association relationship between the current media item and other media items belonging to the same code stream as the current media item.

[0379] The replaceable information data box includes a component reference type field, which is used to indicate the association between the current media track and other media tracks belonging to the same codestream as the current media track, or to indicate the association between the current media item and other media items belonging to the same codestream as the current media item.

[0380] When the value of the component reference type field is the first preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track reference, and the replaceable information data box also includes a track reference type field, which is used to indicate the type of the track reference;

[0381] When the value of the component reference type field is the second preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track group, and the replaceable information data box further includes a track group type field and a track group identifier field. The track group type field is used to indicate the type of track group to which the current media track belongs; and the track identifier field is used to indicate the identifier of the track group to which the current media track belongs.

[0382] When the value of the component reference type field is a third preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an item reference, and the replaceable information data box further includes an item reference type field, which is used to indicate the type of the item reference;

[0383] When the value of the component reference type field is the fourth preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an entity group, and the replaceable information data box also includes an entity group type field and an entity group identification field. The entity group type field is used to indicate the type of the entity group to which the current media item belongs; the entity group identification field is used to indicate the identifier of the entity group to which the current media item belongs.

[0384] In one embodiment, the replaceable information data box further includes a multi-component flag field;

[0385] If the replaceable information data box is set in the current media track, the multi-component flag field is used to indicate whether the code stream belonging to the current media track is encapsulated into multiple media tracks. When the value of the multi-component flag field is a first preset value, it indicates that the code stream belonging to the current media track is encapsulated into one media track, and the current media track is the media track into which the code stream belonging to the current media track is encapsulated. When the value of the multi-component flag field is a second preset value, it indicates that the code stream belonging to the current media track is encapsulated into multiple media tracks, and the current media track is any one of the multiple media tracks into which the code stream belonging to the current media track is encapsulated.

[0386] If the replaceable information data box is set in the current media item, the multi-component flag field is used to indicate whether the code stream belonging to the current media item is encapsulated into multiple media items. When the value of the multi-component flag field is a first preset value, it indicates that the code stream belonging to the current media item is encapsulated into one media item, and the current media item is the media item into which the code stream belonging to the current media item is encapsulated. When the value of the multi-component flag field is a second preset value, it indicates that the code stream belonging to the current media item is encapsulated into multiple media items, and the current media item is any one of the multiple media items into which the code stream belonging to the current media item is encapsulated.

[0387] Among them, if the code stream belonging to the current media track / current media project is a point cloud code stream, and the point cloud code stream is encapsulated in a multi-track encapsulation manner, the value of the multi-component flag field is the second preset value.

[0388] In one embodiment, the immersive media is transmitted in a streaming manner. The processor 901 is specifically configured to: obtain transmission signaling of the immersive media, where the transmission signaling includes description information of the relationship indication information; and obtain a media file of the immersive media according to the transmission signaling.

[0389] In one embodiment, the description information is used to define N codestreams having an alternative relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each of which is used to represent one of the N codestreams; each preselected identifier has the same encoding identifier; each preselected identifier corresponds to one or more adaptation sets, each adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; or each preselected identifier corresponds to one or more representations, each representation represents a media track or a media item in the codestream represented by each preselected identifier.

[0390] In one embodiment, the processor 901 is specifically configured to: determine a code stream to be presented from N replaceable code streams according to an replaceable relationship indicated by the relationship indication information; and decode and present the code stream to be presented; wherein the immersive media includes any one or more of the following: volumetric media, volumetric video media, multi-view video media, subtitle media, and audio media.

[0391] In an embodiment of the present application, an immersive media decoding terminal may obtain a media file of the immersive media, wherein the immersive media includes N interchangeable streams and includes relationship indication information, where the relationship indication information is used to indicate the interchangeable relationship between the N streams, where N is an integer greater than 1. The media file is decoded based on the relationship indication information to present the immersive media. The relationship indication information can clearly indicate the interchangeable relationship between the N streams of the immersive media, thereby guiding accurate decoding and presentation of the immersive media based on the interchangeable relationship, thereby improving the presentation quality of the immersive media.

[0392] In another embodiment, the computer device may be the aforementioned service device; in this embodiment, the processor 901 executes the following operations by running the executable program code in the memory 904: encoding the immersive media to obtain N replaceable code streams; generating relationship indication information based on the replaceable relationship between the N code streams, the relationship indication information being used to indicate the replaceable relationship between the N code streams; and encapsulating the relationship indication information and the N code streams to obtain a media file of the immersive media.

[0393] In an embodiment of the present application, the encoding end of the immersive media can encode the immersive media to obtain N code streams of the immersive media, and the N code streams have an interchangeable relationship. According to the interchangeable relationship, relationship indication information for indicating the interchangeable relationship can be generated, and the relationship indication information and the N code streams can be encapsulated to obtain a media file of the immersive media. It can be seen that in the encoding process of the immersive media, relationship indication information can be added to the media file to indicate the interchangeable relationship between different code streams. In this way, the purpose of indicating the interchangeable relationship at the code stream level is achieved through the relationship indication information. Based on the relationship indication information, the decoding of the immersive media can be guided to be accurately presented, thereby improving the presentation effect of the immersive media.

[0394] In addition, it should be noted that the present invention also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above Figure 5 and Figure 6a The method in the corresponding embodiment will therefore not be described in detail here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0395] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, so that the computer device can perform the above-mentioned Figure 5 and Figure 6a The method in the corresponding embodiment will therefore not be described in detail here.

[0396] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0397] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method for immersive media, characterized in that: include: Acquire a media file of immersive media; the immersive media includes N replaceable code streams; the media file includes relationship indication information, the relationship indication information is used to indicate the replaceable relationship between the N code streams, where N is an integer greater than 1; Decoding the media file according to the relationship indication information to present the immersive media; Wherein, if the immersive media is transmitted in a streaming transmission manner, then obtaining the media file of the immersive media includes: obtaining transmission signaling of the immersive media, wherein the transmission signaling includes description information of the relationship indication information; and obtaining the media file of the immersive media according to the transmission signaling; The description information is used to define N codestreams with an alternative relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each preselected identifier is used to represent one of the N codestreams; each preselected identifier has the same coding identifier; each preselected identifier corresponds to one or more adaptation sets, and an adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; or each preselected identifier corresponds to one or more representations, and a representation represents a media track or a media item in the codestream represented by each preselected identifier.

2. The method according to claim 1, wherein The immersive media is a time-series immersive media; the N code streams are encapsulated into M media tracks in the media file; M is an integer and M is greater than or equal to N; The relationship indication information is set in the media track.

3. The method according to claim 2, wherein Any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; the code stream i is encapsulated into a media track M of the M media tracks. i middle; The relationship indication information is set in the media track M i middle.

4. The method according to claim 2, wherein Any one of the N code streams is denoted as code stream i, where i is a positive integer and is less than or equal to N; the code stream i is encapsulated into multiple media tracks of the M media tracks; The relationship indication information is set in the media track M i middle; The media track M i It refers to any one of the multiple media tracks into which the code stream i is encapsulated.

5. The method according to claim 4, wherein The relationship indication information is also used to indicate the media track M i The association relationship between other media tracks corresponding to the code stream i; The other media tracks refer to the multiple media tracks to which the code stream i is encapsulated except the media track M. i The association relationship is used to represent the media track M i Belongs to the same codestream i as the other media tracks.

6. The method according to claim 2, wherein Any two code streams among the N code streams are denoted as code stream i and code stream j, respectively, where i and j are positive integers and both are less than or equal to N; the code stream i is encapsulated into multiple media tracks among the M media tracks; the code stream j is encapsulated into multiple media tracks among the M media tracks; If the media track into which the code stream i is encapsulated and the media track into which the code stream j is encapsulated both contain the media track M ij , then the relationship indication information is also used to indicate the media track M ij The shared ownership relationship is used to represent the media track M ij It is the media track shared by the code stream i and the code stream j.

7. The method according to claim 1, wherein The immersive media is non-sequential immersive media; the N code streams are encapsulated into P media items in the media file; P is an integer and P is greater than or equal to N; The relationship indication information is set in the media item.

8. The method according to claim 7, wherein Any one of the N code streams is represented as code stream i, where i is a positive integer and i is less than or equal to N; the code stream i is encapsulated into a media item P among the P media items. i middle; The relationship indication information is set on the media item P i middle.

9. The method according to claim 7, wherein Any one of the N code streams is denoted as code stream i, where i is a positive integer and is less than or equal to N; the code stream i is encapsulated into multiple media items among the P media items; The relationship indication information is set on the media item P i The media item P i It refers to any one of the multiple media items into which the code stream i is encapsulated.

10. The method according to claim 9, wherein The relationship indication information is also used to indicate the media item P i The association relationship between other media items corresponding to the code stream i; The other media items refer to the multiple media items into which the code stream i is encapsulated except the media item P i The association relationship is used to represent the media item P i Belongs to the same code stream i as the other media items.

11. The method according to claim 7, wherein Any two code streams among the N code streams are denoted as code stream i and code stream j, respectively, where i and j are positive integers and both are less than or equal to N; the code stream i is encapsulated into multiple media items among the P media items; the code stream j is encapsulated into multiple media items among the P media items; If the media item into which the code stream i is encapsulated and the media item into which the code stream j is encapsulated both contain the media item P ij , then the relationship indication information is also used to indicate the media item P ij The shared ownership relationship is used to represent the media item P ij is a media item shared by the code stream i and the code stream j.

12. The method according to claim 1, wherein: If the immersive media is time-series immersive media, the N code streams are encapsulated into M media tracks in the media file; M is an integer and is greater than or equal to N; wherein, when multiple media tracks among the M media tracks need to be played together, the multiple media tracks that need to be played together belong to the same playback track group; If the immersive media is non-sequential immersive media, the N code streams are encapsulated into P media items in the media file; P is an integer and P is greater than or equal to N; wherein, when there are multiple media items among the P media items that need to be played jointly, the multiple media items that need to be played jointly belong to the same playback entity group.

13. The method according to claim 1, wherein The N code streams having a replaceable relationship belong to the same replaceable group, and different code streams in the same replaceable group can be replaced with each other when presented; the relationship indication information includes a replaceable information data box; If the alternative information data box is set in the current media track, the alternative information data box contains information about the alternative group to which the code stream corresponding to the current media track belongs; If the replaceable information data box is set in the current media item, the replaceable information data box contains information about the replaceable group to which the corresponding code stream of the current media item belongs; The current media track refers to the media track being decoded; and the current media item refers to the media item being decoded.

14. The method according to claim 13, wherein The replaceable information data box includes a replaceable group identification flag field and a replaceable group identification field; If the alternative information data box is set in the current media track, the alternative group identifier flag field is used to indicate whether the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs; when the value of the alternative group identifier flag field is a first preset value, it means that the alternative information data box in the current media track indicates the alternative group identifier to which the codestream corresponding to the current media track belongs; when the value of the alternative group identifier flag field is a second preset value, it means that the alternative information data box in the current media track does not indicate the alternative group identifier to which the codestream corresponding to the current media track belongs; The alternative group identifier field is used to indicate the alternative group identifier to which the codestream corresponding to the current media track belongs; If the alternative information data box is provided in the current media item, the alternative group identifier flag field is used to indicate whether the alternative information data box in the current media item indicates the alternative group identifier to which the code stream corresponding to the current media item belongs; when the value of the alternative group identifier flag field is a first preset value, it indicates that the alternative information data box in the current media item indicates the alternative group identifier to which the code stream corresponding to the current media item belongs; when the value of the alternative group identifier flag field is a second preset value, it indicates that the alternative information data box in the current media item does not indicate the alternative group identifier to which the code stream corresponding to the current media item belongs; The alternative group identifier field is used to indicate the alternative group identifier to which the corresponding code stream of the current media item belongs.

15. The method according to claim 13, wherein The relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item; the replaceable information data box includes a multiple replacement code stream flag field and a code stream quantity field; If the alternative information data box is set in the current media track, the multiple alternative codestream flag field is used to indicate whether the current media track belongs to multiple codestreams; When the value of the multiple alternative codestream flag field is a first preset value, it indicates that the current media track belongs to only one codestream; When the value of the multiple alternative codestream flag field is a second preset value, it indicates that the current media track belongs to multiple codestreams; The code stream quantity field is used to indicate the quantity of code streams to which the current media track belongs; If the alternative information data box is set in the current media item, the multiple alternative code stream flag field is used to indicate whether the current media item belongs to multiple code streams; When the value of the multiple alternative code stream flag field is a first preset value, it indicates that the current media item belongs to only one code stream; When the value of the multiple alternative code stream flag field is a second preset value, it indicates that the current media item belongs to multiple code streams; The code stream quantity field is used to indicate the quantity of code streams to which the current media item belongs.

16. The method according to claim 13, wherein The relationship indication information is further used to indicate the shared ownership relationship of the current media track or the shared ownership relationship of the current media item; If the current media track contains only one replaceable information data box, it indicates that the current media track belongs to only one codestream; if the current media track contains multiple replaceable information data boxes, it indicates that the current media track belongs to multiple codestreams, and the number of replaceable information data boxes in the current media track should be equal to the number of codestreams to which the current media track belongs; or If the current media item contains only one replaceable information data box, it indicates that the current media item belongs to only one code stream; if the current media item contains multiple replaceable information data boxes, it indicates that the current media item belongs to multiple code streams; the number of replaceable information data boxes in the current media item should be equal to the number of code streams to which the current media item belongs.

17. The method according to any one of claims 13 to 16, wherein: The relationship indication information is further used to indicate an association relationship between the current media track and other media tracks belonging to the same code stream as the current media track, or to indicate an association relationship between the current media item and other media items belonging to the same code stream as the current media item; The replaceable information data box includes a component reference type field, where the component reference type field is used to indicate the association between the current media track and other media tracks belonging to the same codestream as the current media track, or to indicate the association between the current media item and other media items belonging to the same codestream as the current media item. When the value of the component reference type field is a first preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track reference, and the replaceable information data box further includes a track reference type field, wherein the track reference type field is used to indicate the type of the track reference; When the value of the component reference type field is the second preset value, it indicates that the current media track is associated with other media tracks belonging to the same codestream as the current media track through a track group, and the replaceable information data box further includes a track group type field and a track group identifier field, wherein the track group type field is used to indicate the type of the track group to which the current media track belongs; and the track identifier field is used to indicate the identifier of the track group to which the current media track belongs; When the value of the component reference type field is a third preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an item reference, and the replaceable information data box further includes an item reference type field, wherein the item reference type field is used to indicate the type of the item reference; When the value of the component reference type field is a fourth preset value, it indicates that the current media item is associated with other media items belonging to the same code stream as the current media item through an entity group, and the replaceable information data box further includes an entity group type field and an entity group identification field, wherein the entity group type field is used to indicate the type of the entity group to which the current media item belongs; and the entity group identification field is used to indicate the identifier of the entity group to which the current media item belongs.

18. The method according to claim 13, wherein The replaceable information data box further includes a multi-component flag field; If the replaceable information data box is set in the current media track, the multi-component flag field is used to indicate whether the code stream to which the current media track belongs is encapsulated into multiple media tracks; When the value of the multi-component flag field is the first preset value, it indicates that the code stream to which the current media track belongs is encapsulated into one media track, and the current media track is the media track to which the code stream to which the current media track belongs is encapsulated; When the value of the multi-component flag field is a second preset value, it indicates that the code stream to which the current media track belongs is encapsulated into multiple media tracks, and the current media track is any one of the multiple media tracks to which the code stream to which the current media track belongs is encapsulated; If the replaceable information data box is set in the current media item, the multi-component flag field is used to indicate whether the code stream belonging to the current media item is encapsulated into multiple media items; when the value of the multi-component flag field is a first preset value, it indicates that the code stream belonging to the current media item is encapsulated into one media item, and the current media item is the media item into which the code stream belonging to the current media item is encapsulated; when the value of the multi-component flag field is a second preset value, it indicates that the code stream belonging to the current media item is encapsulated into multiple media items, and the current media item is any one of the multiple media items into which the code stream belonging to the current media item is encapsulated; Among them, if the code stream belonging to the current media track / the current media project is a point cloud code stream, and the point cloud code stream is encapsulated in a multi-track encapsulation manner, the value of the multi-component flag field is a second preset value.

19. The method according to claim 1, wherein The decoding the media file according to the relationship indication information to present the immersive media includes: Determining a code stream to be presented from the N replaceable code streams according to the replaceable relationship indicated by the relationship indication information; Decode and present the required code stream; The immersive media includes any one or more of the following: volumetric media, volumetric video media, multi-view video media, subtitle media, and audio media.

20. A data processing method for immersive media, characterized in that: include: Encoding the immersive media to obtain N replaceable code streams; generating relationship indication information according to the replaceable relationship between the N code streams, wherein the relationship indication information is used to indicate the replaceable relationship between the N code streams; encapsulating the relationship indication information and the N code streams to obtain a media file of the immersive media; The replaceable relationship between the N codestreams is further used to generate description information of the relationship indication information, and the description information of the relationship indication information is included in the transmission signaling of the immersive media; the description information is used to define the N codestreams with a replaceable relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each preselected identifier is used to represent one of the N codestreams; each preselected identifier has the same coding identifier; each preselected identifier corresponds to one or more adaptation sets, and an adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; Alternatively, each pre-selected identifier corresponds to one or more representations, and one representation represents a media track or a media item in the code stream represented by each pre-selected identifier.

21. An immersive media data processing device, characterized in that: include: an acquisition unit, configured to acquire a media file of immersive media; the immersive media comprising N replaceable code streams; the media file comprising relationship indication information, the relationship indication information being used to indicate an replaceable relationship between the N code streams, where N is an integer greater than 1; a processing unit, configured to decode the media file according to the relationship indication information to present the immersive media; Wherein, if the immersive media is transmitted in a streaming transmission manner, then obtaining the media file of the immersive media includes: obtaining transmission signaling of the immersive media, wherein the transmission signaling includes description information of the relationship indication information; and obtaining the media file of the immersive media according to the transmission signaling; The description information is used to define N codestreams with an interchangeable relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each of which is used to represent one of the N codestreams; each preselected identifier has the same coding identifier; each preselected identifier corresponds to one or more adaptation sets, each adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; or each preselected identifier corresponds to one or more representations, each representation represents a media track or a media item in the codestream represented by each preselected identifier.

22. An immersive media data processing device, characterized in that: include: An encoding unit, configured to encode the immersive media to obtain N replaceable code streams; a processing unit, configured to generate relationship indication information according to the replaceable relationship between the N code streams, wherein the relationship indication information is used to indicate the replaceable relationship between the N code streams; The processing unit is further configured to encapsulate the relationship indication information and the N code streams to obtain a media file of the immersive media; The replaceable relationship between the N codestreams is further used to generate description information of the relationship indication information, and the description information of the relationship indication information is included in the transmission signaling of the immersive media; the description information is used to define the N codestreams with a replaceable relationship indicated by the relationship indication information; the description information includes N preselected identifiers, each preselected identifier is used to represent one of the N codestreams; each preselected identifier has the same coding identifier; each preselected identifier corresponds to one or more adaptation sets, and an adaptation set represents a media track or a media item in the codestream represented by each preselected identifier; Alternatively, each pre-selected identifier corresponds to one or more representations, and one representation represents a media track or a media item in the code stream represented by each pre-selected identifier.

23. A computer device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the immersive media data processing method according to any one of claims 1 to 20 is executed.

24. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the immersive media data processing method according to any one of claims 1 to 20 is executed.

25. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the immersive media data processing method according to any one of claims 1 to 20.

Images