Data processing method of media data and related device

Abstract

The application provides a data processing method of media data and related equipment. The media data is composed of N media frames, the N media frames are encapsulated into M tracks to obtain a video file corresponding to each track, media frames encapsulated into the same track have the same quality, at least two tracks in the M tracks contain media frames with the same quality, and N and M are integers greater than 1. The data processing method comprises the following steps: obtaining a signaling description file of the media data, the signaling description file comprising combination indication information, the combination indication information being used for indicating video files allowed to be combined and decoded in the M video files; and combining and decoding each video file requested according to the signaling description file. According to the application, the combination restriction information of the video files allowed to be combined and decoded can be obtained in advance by the content playing equipment, so that the bandwidth is saved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for processing media data, a device for processing media data, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In existing immersive media transmission schemes, in scenarios where multiple time-domain media frames and multiple quality media frames coexist, constraints are imposed on the tracks of different time-domain levels within the signaling description file at the encapsulation level. Specifically, to avoid the combined decoding of media frames of different qualities within different time-domain levels, combined decoding of media frames of different time-domain levels belonging to the same track group is supported. However, this constraint provided by existing technology is only reflected at the file transmission level. Since the combination restrictions between media frames of different qualities within different time-domain levels cannot be known in advance, the content playback device must first request media frames of all time-domain levels and all qualities, inevitably resulting in bandwidth waste. Summary of the Invention

[0003] This application provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing media data. The content playback device can obtain the combination restriction information of video files that are allowed to be combined and decoded in advance, thereby saving bandwidth.

[0004] On one hand, embodiments of this application provide a data processing method for media data. The media data consists of N media frames, which are encapsulated into M tracks to obtain a video file corresponding to each track. Furthermore, media frames encapsulated into the same track have the same quality, and at least two of the M tracks contain media frames of the same quality. N and M are both integers greater than 1. The data processing method includes:

[0005] A signaling description file for acquiring media data, the signaling description file includes combination indication information, the combination indication information is used to indicate which of the M video files are allowed to be combined for decoding;

[0006] After requesting one or more video files according to the signaling description file, the requested video files are combined and decoded.

[0007] On one hand, embodiments of this application provide a method for processing media data, the method comprising:

[0008] N media frames of media data are encapsulated into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1.

[0009] The signaling description file for media data is generated based on the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined and decoded.

[0010] On one hand, embodiments of this application provide a data processing apparatus for media data. The media data consists of N media frames, which are encapsulated into M tracks to obtain a video file corresponding to each track. Furthermore, the media frames encapsulated into the same track have the same quality, and at least two of the M tracks contain media frames of the same quality. N and M are both integers greater than 1. The data processing apparatus includes:

[0011] The acquisition unit is used to acquire the signaling description file of the media data. The signaling description file includes combination indication information, which is used to indicate the video files among the M video files that are allowed to be combined and decoded.

[0012] The processing unit is used to combine and decode the requested video files after requesting one or more video files according to the signaling description file.

[0013] On one hand, embodiments of this application provide a data processing apparatus for media data, the data processing apparatus comprising:

[0014] The encapsulation unit is used to encapsulate N media frames of media data into M tracks to obtain a video file corresponding to each track. The media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1.

[0015] The processing unit is used to generate a signaling description file for media data according to the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined and decoded.

[0016] On one hand, embodiments of this application provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the aforementioned media data processing method.

[0017] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned media data processing method.

[0018] On one hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned media data processing method.

[0019] In this embodiment, the content production device encapsulates N media frames into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks contain media frames of the same quality. The content production device then generates combination indication information based on the video files allowed for combined decoding and adds it to a signaling description file. After receiving the signaling description file from the content production device, the content playback device can, based on the combination indication information included in the signaling description file, know in advance which of the M video files are allowed for combined decoding. The content playback device can then select a suitable video file for consumption and decoding according to the combination indication information. Therefore, the content playback device can obtain the combination restriction information of the video files allowed for combined decoding in advance, and then request the corresponding video file for decoding and consumption based on the combination restriction information, thereby achieving the goal of accurately requesting the corresponding media frames without needing to request all video files, thus saving transmission bandwidth. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1a An architectural diagram of a media data system provided in an exemplary embodiment of this application is shown;

[0022] Figure 1b A flowchart illustrating a video processing method provided in an exemplary embodiment of this application is shown;

[0023] Figure 2a A schematic diagram of 6DoF provided in an exemplary embodiment of this application is shown;

[0024] Figure 2b A schematic diagram of 3DoF provided in an exemplary embodiment of this application is shown;

[0025] Figure 2cA schematic diagram of 3DoF+ provided in an exemplary embodiment of this application is shown;

[0026] Figure 2d A flowchart illustrating a video encoding method provided in an exemplary embodiment of this application is shown;

[0027] Figure 2e This illustration shows a schematic diagram of an input image segmentation provided by an exemplary embodiment of this application;

[0028] Figure 3a This illustration shows a schematic diagram of a temporal hierarchy provided by an exemplary embodiment of this application;

[0029] Figure 3b This illustration shows another temporal hierarchy partitioning provided by an exemplary embodiment of this application;

[0030] Figure 4 A flowchart illustrating a data processing method provided in an exemplary embodiment of this application is shown;

[0031] Figure 5 A flowchart illustrating another data processing method provided by an exemplary embodiment of this application is shown;

[0032] Figure 6 This invention provides a schematic diagram of the structure of a data processing apparatus according to an exemplary embodiment of the present application.

[0033] Figure 7 This invention provides a schematic diagram of the structure of another data processing apparatus according to an exemplary embodiment of the present application.

[0034] Figure 8 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0036] This application relates to data processing technology for media data. Specifically, media data may include immersive media data or media content. Immersive media refers to media files that provide immersive media content, allowing users immersed in the media content to obtain sensory experiences such as sight and sound in the real world. Specifically, immersive media can be 3DoF (Three Degrees of Freedom) immersive media, 3DoF+ immersive media, or 6DoF (Six Degrees of Freedom) immersive media. Immersive media content includes video content represented in three-dimensional (3D) space in various forms, such as three-dimensional video content represented in a spherical form. Specifically, immersive media content can be VR (Virtual Reality) video content, panoramic video content, spherical video content, or 360-degree video content; therefore, immersive media can also be called VR video, panoramic video, spherical video, or 360-degree video.

[0037] In this embodiment, immersive media may include visual volumetric video-based coding media (V3C). V3C visual volumetric media refers to immersive media that captures visual content in three-dimensional space and provides a 3DoF+ or 6DoF viewing experience, encoded with traditional video, and includes a visual volumetric video type track in the file encapsulation. This includes multi-view video and video-coded point clouds (or point cloud media). Multi-view video, also known as multi-viewpoint video, specifically refers to video with depth information captured from multiple angles using multiple camera arrays. Multi-view / multi-viewpoint video, also called free-viewpoint / free-viewpoint video, is a type of immersive media that provides a six-degrees-of-freedom experience. Furthermore, point cloud media refers to media files whose content includes point cloud data. A point cloud is a set of randomly distributed discrete points in space that express the spatial structure and surface attributes of a three-dimensional object or scene. Each point in the point cloud has at least three-dimensional position information and, depending on the application scenario, may also have color, material, or other information. Typically, each point in the point cloud has the same number of additional attributes.

[0038] Immersive media can include video data and / or audio data, wherein the production and transmission process of immersive media can be as follows: Figure 1a As shown, in the process of creating and transmitting immersive media, such as Figure 1aThe illustration primarily involves content creation equipment and content playback equipment. The content creation equipment can refer to the computer equipment used by the provider of immersive media (e.g., the content creation terminal of point cloud media), and may also be referred to as an encoding device. The content playback device refers to the computer equipment used by the consumer of immersive media (e.g., the content viewer of point cloud media), and may also be referred to as a decoding device. In one embodiment, the content creation equipment can be a terminal (such as a PC, a smart mobile device, or a smartphone) or a server, while the content playback device can be a terminal, a smart mobile device, a VR device (such as a VR headset or VR glasses), or an AR device.

[0039] The production and transmission of immersive media generally includes one or more of the following stages (or processes): acquisition, encoding, file encapsulation, file transfer, file decapsulation, decoding, and final presentation. Below, we will combine... Figure 1b The production and transmission process when the immersive media is video data is explained, such as... Figure 1b As shown, the production and transmission process of video data specifically includes one or more of the following stages: video acquisition, video encoding, video file encapsulation, video file transmission, video file decapsulation, video decoding, and final video presentation. Similarly, when the immersive media is point cloud media, the production and transmission process for that point cloud media also includes one or more of the above processing stages.

[0040] In the multiple processing stages of immersive media, acquisition, encoding, file encapsulation, and file transfer are performed by the aforementioned content creation equipment, while file decapsulation, decoding, and final presentation are performed by the content playback equipment. Below, we will describe each processing stage of immersive media in detail, based on the processing procedures involved in the content creation equipment and the processing procedures involved in the content playback equipment.

[0041] I. Processing procedures involved in content production equipment:

[0042] (1) The process of acquiring immersive media.

[0043] Immersive media content is obtained by capturing sound and visual scenes from the real world using capture devices. In one embodiment, the capture device can refer to a hardware component located within a content creation device, such as a microphone, camera, and sensors on a terminal. In other embodiments, the capture device can be a hardware device independent of but connected to the content creation device, such as a camera connected to a server. The capture device can include, but is not limited to, audio devices, video devices, and sensing devices. Audio devices can include audio sensors, microphones, etc. Video devices can include ordinary cameras, stereo cameras, light field cameras, etc. Sensing devices can include laser devices, radar devices, etc. Multiple capture devices can be used, and these devices can be deployed in specific viewpoints in the real space to simultaneously capture audio and video content from different viewpoints within that space, with the captured audio and video content remaining synchronized in both time and space. For example, 3DoF immersive content is recorded by a group of cameras or a video device with multiple cameras and sensors, while 6DoF immersive media content is primarily created from point clouds, light fields, and other forms of content captured by a camera array.

[0044] 2) Production of immersive media:

[0045] The captured audio content is inherently suitable for immersive media encoding, therefore no further processing is required. However, the captured video content needs to undergo a series of production processes before it can be considered suitable for immersive media encoding. These processes may include:

[0046] ① Stitching: Since the captured immersive media video content is shot from different angles by the capture device, stitching refers to stitching these video contents shot from various angles into a complete video that can reflect a 360-degree visual panorama of the real space. In other words, the stitched video is a panoramic video represented in three-dimensional space.

[0047] ② Projection: Projection refers to the process of mapping a three-dimensional video formed by splicing onto a two-dimensional (2D) image. The 2D image formed by projection is called a projected image. Projection methods may include, but are not limited to: latitude and longitude projection and regular hexahedral projection.

[0048] It should be noted that since the capture device can only capture panoramic video, after such video is processed by the content production device and transmitted to the content consumption device for corresponding data processing, the user on the content consumption device can only view 360-degree video information through certain specific actions (such as head rotation). Performing non-specific actions (such as moving the head) will not produce corresponding video changes, resulting in a poor VR experience. Therefore, it is necessary to provide additional depth information that matches the panoramic video to give users a better immersion and VR experience. This involves various production technologies, including 6DoF production technology, 3DoF production technology, and 3DoF+ production technology.

[0049] Figure 2a This illustration shows a schematic diagram of 6DoF provided in an exemplary embodiment of this application. 6DoF is divided into window 6DoF, omnidirectional 6DoF, and 6DoF. Window 6DoF means that the user's rotational movement along the X and Y axes, and translation along the Z axis, are restricted; for example, the user cannot see the scene outside the window frame, and the user cannot pass through the window. Omnidirectional 6DoF means that the user's rotational movement along the X, Y, and Z axes is restricted; for example, the user cannot freely move through 3D 360-degree VR content within a restricted movement area. 6DoF means that the user can freely translate along the X, Y, and Z axes; for example, the user can freely walk within 3D 360-degree VR content. Similar to 6DoF are 3DoF and 3DoF+ production techniques. Figure 2b This illustration shows a schematic diagram of 3DoF provided in an exemplary embodiment of this application; as shown Figure 2b As shown, 3DoF refers to a user viewing media content by having their head rotate along the X, Y, and Z axes while the user is fixed at the center point in a three-dimensional space. Figure 2c A schematic diagram of 3DoF+ provided in an exemplary embodiment of this application is shown, such as... Figure 2c As shown, 3DoF+ refers to the ability of a user's head to move within a limited space based on 3DoF to view the images provided by immersive media when the virtual scene provided has a certain depth information.

[0050] (2) Encoding of immersive media.

[0051] Projected images can be encoded directly, or they can be encapsulated into regions before encoding. Modern mainstream immersive media coding technologies, taking the international immersive media coding standards HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding), as well as the Chinese national immersive media coding standard AVS (Audio Video Coding Standard), as examples, employ a hybrid coding framework, performing the following series of operations and processing on the input raw immersive media signal. Please refer to... Figure 2d , Figure 2d A flowchart illustrating an immersive media encoding method provided in an exemplary embodiment of this application is shown. Next, in conjunction with... Figure 2d A detailed introduction to the immersive media encoding process:

[0052] 1) Block partition structure: The input image (i.e., image frames in immersive media) is divided into several non-overlapping processing units based on the size of the processing units. A similar compression operation is performed on each processing unit. This processing unit is called a Coding Tree Unit (CTU) or Largest Coding Unit (LCU). The CTU can be further subdivided to obtain one or more basic coding units, called Coding Units (CUs). Each CU is the most basic element in a coding process. Figure 2e This illustration shows a schematic diagram of an input image partitioning method provided in an embodiment of this application. The following describes various encoding methods that may be used for each CU.

[0053] 2) Predictive Coding: This includes intra-frame prediction and inter-frame prediction. The original immersive media signal is predicted from the selected reconstructed immersive media signal to obtain the residual immersive media signal. The content production device needs to determine the most suitable predictive coding mode from among many possible modes for the current CU and inform the content playback device.

[0054] a. Intra-frame prediction: The predicted signal comes from a region within the same image that has already been encoded and reconstructed.

[0055] b. Inter-frame prediction: The predicted signal comes from other images that have already been encoded and are different from the current image (called reference images).

[0056] 3) Transform & Quantization: The residual immersive media signal undergoes transformation operations such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) to convert the signal to the transform domain, where the coefficients are called transform coefficients. In the transform domain, the signal undergoes further lossy quantization, losing some information to make the quantized signal more suitable for compression. Some immersive media coding standards may offer more than one transformation method to choose from. Therefore, the content production device needs to select one of the transformations for the current encoding CU and inform the content playback device. The fineness of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a wider range of values ​​will be quantized into the same output, which usually results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller range of values ​​will be quantized into the same output, which usually results in less distortion and a higher bitrate.

[0057] 4) Entropy Coding or Statistical Coding: The quantized transform domain signal is statistically compressed and encoded based on the frequency of each value, ultimately outputting a binary (0 or 1) compressed bitstream. Simultaneously, other information generated during encoding, such as the selected mode and motion vectors, also requires entropy coding to reduce the bit rate. Statistical coding is a lossless coding method that effectively reduces the bit rate required to represent the same signal. Common statistical coding methods include Variable Length Coding (VLC) or Content-Adaptive Binary Arithmetic Coding (CABAC).

[0058] 5) Loop Filtering: After the encoded image undergoes inverse quantization, inverse transform, and prediction compensation operations (the reverse of operations 2-4 above), a reconstructed decoded image is obtained. Compared to the original image, the reconstructed image differs in some information due to the influence of quantization, resulting in distortion. Filtering the reconstructed image, such as deblocking, Sample Adaptive Offset (SAO) filters, or Adaptive Loop Filters (ALF), can effectively reduce the distortion caused by quantization. Since these filtered reconstructed images will serve as a reference for subsequent encoded images to predict future signals, the above filtering operations are also called loop filtering, or filtering operations within the coding loop.

[0059] For example, such as Figure 2d As shown, Figure 2d The k-th CU (labeled as S) K Taking [x,y] as an example, where k is a positive integer greater than or equal to 1 and less than or equal to the number of CUs in the current input image, and S K [x, y] represents the pixel with coordinates [x, y] in the k-th CU, where x represents the x-coordinate of the pixel and y represents the mid-coordinate of the pixel. K The predicted signal is obtained after [x,y] undergoes a preferred processing method such as motion compensation or intra-frame prediction. S K [x,y] and Subtraction yields the residual signal U K [x,y], and then the residual signal U K The [x,y] region is transformed and quantized. The quantized output data goes to two different places: one is sent to an entropy encoder for entropy encoding, and the encoded bitstream is output to a buffer for storage, waiting to be transmitted; the other application is to perform inverse quantization and inverse transform to obtain the signal U. K [x,y]. The signal U K '[x,y] and The sum is used to obtain a new prediction signal S. K *[x,y], and set S K *[x,y] is stored in the buffer of the current image. S K *[x,y] is obtained by intra-frame image prediction, f(S*k[x,y]), S KAfter *[x,y] undergoes loop filtering to obtain S'k[x,y], which is then stored in the decoded image buffer for use in generating the reconstructed media. After motion-compensated prediction, S'k[x+mx,y+my] is obtained, where S'r[x+mx,y+my] represents the reference block, and mx and my represent the horizontal and vertical components of the motion vector, respectively. It should be noted that if 6DoF (Six Degrees of Freedom) technology is used (where the user can move relatively freely in the simulated scene), a specific encoding method (such as point cloud encoding) is required during the encoding process.

[0060] In technologies for transmitting video data in immersive media, streaming technologies are commonly used to handle the transmission of media resources between servers and clients. Common media streaming technologies include DASH (Dynamic Adaptive Streaming over HTTP), HLS (HTTP Live Streaming), and SMT (Smart Media Transport). Taking DASH as an example, DASH is an adaptive bitrate streaming technology that allows high-quality streaming media to be delivered over the Internet through traditional HTTP web servers. DASH breaks the content into a series of small HTTP-based file segments, each containing a short length of playable content. Since the total length of immersive media content can be several hours long (such as live movies or sports events), the media content is made into multiple bitrate alternative segments to provide multiple bitrate versions for selection. In other words, after obtaining high-quality immersive media, its content (such as video data) is processed into media data of different qualities. This allows playback devices to select and combine media data of appropriate quality for subsequent consumption. For example, the content of an immersive medium might be processed into two alternative segments at different bitrates. The resulting media data would then contain frames at both bitrates. Therefore, when a DASH client (i.e., a playback device) requests playback of immersive media, the client automatically selects which alternative to download and play based on current network conditions. The client will choose the highest bitrate segment that can be downloaded immediately for playback, thus avoiding playback stuttering or rebuffering. As a result, DASH clients can seamlessly adapt to changing network conditions and provide a high-quality playback experience with fewer stutters and rebuffering occurrences.

[0061] DASH utilizes existing HTTP web server infrastructure. It allows devices such as internet TVs, set-top boxes, desktop computers, smartphones, and tablets to consume multimedia content (such as video, television, and radio) delivered over the internet, and can adapt to changing internet reception conditions.

[0062] (3) Encapsulation of immersive media files.

[0063] After encoding immersive media, the encoded data stream needs to be encapsulated and transmitted to the user. Immersive media file encapsulation refers to storing the encoded and compressed video and audio data in a file container (or track) according to a specific encapsulation format (or container, or file container). Common encapsulation formats include AVI (Audio Video Interleaved) or ISOBMFF (ISO Based Media File Format). ISOBMFF is the media file encapsulation standard, and the most typical ISOBMFF file is MP4 (Moving Picture Experts Group 4) file. It can be understood that since immersive media content is produced as media data of different qualities, the content production equipment needs to encapsulate the media frames contained in the media data of different qualities to obtain the corresponding encapsulated files (or media file resources). The encapsulation file can be a media file or a media file of immersive media formed from media files or media segments. It records the metadata of the immersive media file resource using Media Presentation Description (MPD) according to the file format requirements of immersive media. This metadata is a general term for information related to the presentation of immersive media, and may include descriptions of the encapsulated media frames, descriptions of the viewport, and signaling information related to presentation, etc. In one embodiment, the encapsulation file uses samples as the encapsulation unit in the file encapsulation process, and an encapsulation file consists of multiple samples. That is, in the media file encapsulation process, a media frame is typically encapsulated as a sample to generate the encapsulation file.

[0064] In one embodiment, the metadata recorded using media presentation description information can also be called a signaling description file. Since the media frames encapsulated in different tracks differ in this embodiment, the video files obtained after encapsulating media frames in each track also differ. When a content playback device requests content consumption, it generally needs to combine media frames of the same quality for consumption. Therefore, in order to facilitate the combined consumption by the content playback device, this embodiment creatively proposes to add corresponding combination indication information to the signaling description file of the content production device, so that the content playback device can request one or more of the obtained video files based on the indication of the combination indication information, and perform combined decoding on the requested video files.

[0065] II. Processing procedures involved in content playback devices:

[0066] (1) Immersive media file decapsulation process.

[0067] After receiving the signaling description file (i.e., media presentation description information) sent by the content production device, the content playback device can request different video files and perform subsequent combination decoding based on the combination indication information contained in the signaling description file. The media file resources (i.e., the aforementioned video files) and media presentation description information obtained after media frames are encapsulated into different tracks are transmitted from the content production device to the content playback device via a transmission mechanism (such as DASH or SMT). The file decapsulation process on the content playback device side is the reverse of the file encapsulation process on the content production device side. The content playback device decapsulates the media file resources according to the media file format requirements to obtain the decoded bitstream of the immersive media.

[0068] (2) Immersive media decoding and presentation process.

[0069] The decoding process on the content playback device is the reverse of the encoding process on the content production device. The content playback device decodes the audio stream to reconstruct the audio content. Furthermore, the decoding process for immersive media streams on the content playback device includes the following: ① Decoding the immersive media stream to obtain a planar projected image. ② Reconstructing the projected image based on the media presentation description information to convert it into a 3D image. This reconstruction process refers to reprojecting the two-dimensional projected image into 3D space.

[0070] As can be seen from the above encoding process, at the content playback device end, for each CU, after obtaining the compressed bitstream, the content playback device first performs entropy decoding to obtain various mode information and quantized transform coefficients. Each coefficient undergoes inverse quantization and inverse transform to obtain the residual signal. On the other hand, based on the known encoding mode information, the prediction signal corresponding to the CU can be obtained. After adding the two, the reconstructed signal can be obtained. Finally, the reconstructed value of the decoded image needs to undergo a loop filtering operation to generate the final output signal. This decoding process involves decoding the encoded media frames obtained from different tracks and then performing the final rendering after decoding.

[0071] In video coding technology, a temporal layering technique is also involved. This technique divides different video frames into different temporal levels according to their decoding dependencies. Specifically, when using this temporal layering technique to divide the temporal levels, video frames classified as lower-level frames do not need to be referenced during decoding. Please see [link to relevant documentation]. Figure 3a , Figure 3a This illustration shows a schematic diagram of a temporal hierarchy provided by an exemplary embodiment of this application. For example... Figure 3a As shown, the arrows indicate dependencies during decoding. The arrow from frame I0 to frame B1 indicates that frame B1 requires reference to frame I0 for decoding; that is, the decoding of frame B1 depends on the decoding of frame I0. The relationships between other frames follow the same pattern. Figure 3a As indicated by the middle arrow, the inter-frame dependencies show that all video frames are divided into four temporal levels, L0 to L3, based on these dependencies. Video frames belonging to each temporal level do not depend on higher-level video frames during decoding. It should be noted that the terms "low" and "high" in the temporal levels mentioned in this embodiment are relative concepts. Figure 3a The four time-domain levels L0 to L3 are defined in the code. For the L0 time-domain level, L1 to L3 are all higher time-domain levels. For the L1 time-domain level, L3 is a higher time-domain level than L1, while L0 is a lower time-domain level than L1. Figure 3a As shown, video frame types mainly include I-frames (Intra Slice), B-frames, and P-frames. I-frames, also known as keyframes, are intra-frame compressed; decoding only requires reference to the I-frame itself. B-frames are bidirectional predictive coded frames, requiring reference to both preceding and subsequent frames during decoding. P-frames are forward predictive coded frames, meaning decoding requires reference to information from relevant preceding frames. Figure 3aThe Arabic numeral subscripts added to I-frames, B-frames, and P-frames are used to indicate their respective temporal levels. It can be understood that, based on the decoding characteristics of I-frames, P-frames, and B-frames, to ensure that video frames belonging to lower temporal levels do not depend on higher temporal levels during decoding after temporal level division, it can be understood that video frames in the lowest temporal level (such as the L0 temporal level mentioned above) will not depend on any other temporal level during decoding. That is, video frames in the lowest temporal level can be decoded and displayed independently. Therefore, I-frames must be included in the video frames classified as the lowest temporal level.

[0072] Because when dividing video frames into temporal levels, video frames belonging to lower temporal levels do not need to refer to video frames belonging to higher temporal levels during decoding, such as... Figure 3a As shown, assuming a video frame in immersive media includes four temporal levels from L0 to L3, and... Figure 3a The arrows in the diagram represent the dependencies between video frames during decoding. Specifically, the arrows from frame I0 to frame B1 indicate that frame B1, located in the L1 time domain, requires reference to frame I0 (located in the L0 time domain) during decoding; frame B1, located in the L1 time domain, requires reference to frame P0 (located in the L0 time domain); the first B2 frame in the L2 time domain requires reference to frame I0 (located in the L0 time domain) and frame B1 (located in the L1 time domain); the second B2 frame in the L2 time domain requires reference to frame B1 (located in the L1 time domain) and frame P0 (located in the L0 time domain); and so on. The first B3 frame in the L3 time domain needs to refer to the first B2 frame in the L2 time domain and the I0 frame in the L0 time domain when decoding. The second B3 frame in the L3 time domain needs to refer to the first B2 frame in the L2 time domain and the B1 frame in the L1 time domain when decoding. The third B3 frame in the L3 time domain needs to refer to the B1 frame in the L1 time domain and the second B2 frame in the L2 time domain when decoding. The fourth B3 frame in the L3 time domain needs to refer to the second B2 frame in the L2 time domain and the P0 frame in the L0 time domain when decoding.

[0073] The process described in this application—creating immersive media content into media data of different qualities, encapsulating the media frames contained in the media data of different qualities into multiple tracks to generate corresponding encapsulation files, and adding corresponding combination indication information to the signaling description file generated for the media data encapsulation process—mainly applies to immersive media as point cloud media. However, it is equally applicable to ordinary video media data. The following mainly describes the process of creating, encapsulating, transmitting, and consuming point cloud media. Additionally, it should be noted that after encapsulating the media data into different tracks, a video file (representation) corresponding to each track is obtained. This video file can also be referred to as a transport stream or media resource.

[0074] Similar to the temporal layering of video frames in ordinary videos, temporal layering techniques also exist for point cloud media, as mentioned above. In this embodiment, for point cloud media, different temporal layers can be defined for the media frames included in the point cloud media, and these different temporal layers can be encapsulated into different tracks. Please refer to... Figure 3b , Figure 3b This illustration shows another temporal hierarchy partitioning provided by an exemplary embodiment of this application. (See diagram below.) Figure 3bAs shown, in point cloud media, assume the media consists of 12 media frames (Frame1, Frame2, Frame3, ..., Frame12). These 12 media frames can then be divided into temporal levels according to the time dimension. For example, the six media frames with a time interval of T0 (Frame1, Frame3, Frame5, Frame7, Frame9, Frame11) can be assigned to the L0 (Temporal Level 0) temporal level; the three media frames with a time interval of T1 (Frame2, Frame6, Frame10) can be assigned to the L1 (Temporal Level 1) temporal level; and the three media frames with a time interval of T1 (Frame4, Frame8, Frame12) can be assigned to the L2 (Temporal Level 2) temporal level. Here, T0 represents the representation time difference between any two adjacent media frames in the L0 temporal level. For example, T0 could represent the representation time difference between Frame1 and Frame3, or between Frame3 and Frame5, and so on. Additionally, T1 represents the representation time difference between any two adjacent media frames in the L1 time domain. For example, T1 can represent the representation time difference between Frame 2 and Frame 6, or between Frame 6 and Frame 10, and so on. Similarly, T1 can also represent the representation time difference between any two adjacent media frames in the L2 time domain. For instance, T1 can represent the representation time difference between Frame 4 and Frame 8, or between Frame 8 and Frame 12, and so on.

[0075] For point cloud media, a point cloud compression (PCC) technology is also involved. Point cloud compression technology can be further divided into G-PCC (Geometry-based Point Cloud Compression) and V-PCC (Video-based Point Cloud Compression).

[0076] In one embodiment, since only combinations of media frames of suitable quality (i.e., of equal quality) can be consumed by the user, but due to the compression and encoding methods of point cloud media content, media data containing media frames of different qualities will be obtained. Therefore, in order to enable the content playback device to know in advance the combination limitations between media frames of different qualities at different temporal levels, the content production device can add combination indication information for different video files in the corresponding signaling description file generated when encapsulating media frames of different qualities. This avoids the content playback device requesting all temporal levels and all qualities of media frames in advance, thereby achieving the purpose of accurately requesting and consuming appropriate combinations of media frames, and effectively saving bandwidth for the content playback device. The specific implementation steps of the content production device in defining and indicating how to select a specific quality combination of media frames when combining and consuming media frames at different temporal levels are as follows:

[0077] 1. The content production equipment, based on the structure of the container file corresponding to the media data obtained from immersive media production, and the temporal hierarchy and quality information in the video files obtained after encapsulating the media frames in each track, organizes the container file into a video stream (i.e., one or more video files (representations)) and generates a corresponding signaling description file. The signaling description file then instructs the generation of a combination of multiple video files based on the temporal hierarchy and quality information in the video files.

[0078] 2. The content creation device sends the signaling description file to the content playback device.

[0079] 3. After obtaining the signaling description file, the content production device requests a suitable video file and decodes and consumes it based on its own needs, network conditions, or decoding capabilities, combined with the temporal hierarchy information, quality information, and combination indication information in the signaling description file.

[0080] To support the above steps, embodiments of this application add corresponding combined indication information to the signaling description information by adding several descriptive fields at the system layer. This extension includes field extensions at the signaling description file level to support the relevant specific implementation steps of this application. Please refer to... Figure 4 , Figure 4 A flowchart illustrating a data processing method for media data provided in an exemplary embodiment of this application is shown. This method can be executed by a content playback device (also known as a decoding device) in a media data system. It is understood that the content playback device includes a terminal device (or client) where a user is consuming immersive media content. The method includes the following steps S401-S402:

[0081] S401: Signaling description file for acquiring media data. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined for decoding.

[0082] In this embodiment, the media data consists of N media frames, which are encapsulated into M tracks. Each track contains a corresponding video file, and media frames encapsulated into the same track have the same quality. At least two tracks among the M tracks contain media frames of the same quality. Both N and M are integers greater than 1. Specifically, the N media frames are multiple media frames of different qualities produced from immersive media. For example, different quality media frames can be obtained by using G-PCC compression on point cloud media. Specifically, two different bitrates can be used to compress the point cloud media, resulting in media data containing media frames of two different qualities. That is, the quality of the media frames obtained by using G-PCC compression on the point cloud media can be quality 1, quality 2, or quality 3, etc. When encapsulating media frames of different qualities, to avoid encapsulating media frames of different qualities in the same track, which could lead to inconsistent quality in the resulting video file and subsequent decoding errors, content production equipment typically encapsulates media frames of the same quality into multiple different tracks, while media frames of different qualities are generally encapsulated into different tracks. For example, the content production equipment can encapsulate multiple media frames of quality 1 into two different tracks, and multiple media frames of quality 2 into two different tracks. In one embodiment, when encapsulating media data, the content production equipment also encapsulates media frames belonging to different time domain levels into different tracks based on their temporal domain level.

[0083] In one embodiment, the content production device, based on the encapsulation process of media frames of different qualities, obtains video files corresponding to each track and generates corresponding signaling description files. Therefore, the signaling description files of the media data acquired by the content playback device are those generated by the content production device after encapsulating the media data and received via the DASH transmission protocol. The signaling description files include combination indication information, which indicates which video files among the M encapsulated video files are allowed to be combined for decoding. In this way, after acquiring the signaling description files, the content playback device can, according to the combination indication information in the signaling description files and based on its own needs, network conditions, or decoding capabilities, request one or more video files from the M video files for decoding and consumption.

[0084] In one possible implementation, the N media frames are divided into multiple temporal levels, with each temporal level containing one or more media frames. A track encapsulates the corresponding media frames to obtain a corresponding video file (representation), which contains media frames from one or more temporal levels. The combination instruction information includes temporal level combination descriptors, which define the allowed combination methods between video files from different temporal levels.

[0085] Suppose there are N media frames, represented as f1, f2, f3, ..., f12. f1 through f6 can be media frames of quality 1; f7 through f12 can be media frames with the same content as f1 through f6 but of different quality (quality 2). Then, the temporal hierarchy corresponding to f1, f3, and f5 can be L1; the temporal hierarchy corresponding to f2, f4, and f6 can be L2; the temporal hierarchy corresponding to f7, f9, and f11 can be L3; and the temporal hierarchy corresponding to f8, f10, and f12 can be L4. Suppose L1 is encapsulated in track 1, L2 in track 3, L3 in track 2, and L4 in track 4. Then, the video file obtained after encapsulating media frames in track 1 can be represented as representation1, the video file obtained after encapsulating media frames in track 2 can be represented as representation2, the video file obtained after encapsulating media frames in track 3 can be represented as representation3, and the video file obtained after encapsulating media frames in track 4 can be represented as representation4.

[0086] In this embodiment, the signaling description file can be a description file (MPD signaling) used to describe media segment information. Taking the extension of the existing DASH signaling description file as an example, a method for defining combined indication information for video files is defined, whereby the combined indication information includes a temporal-level combined descriptor. The temporal-level combined descriptor will be described below.

[0087] (1) Define a new temporal hierarchical combination descriptor:

[0088] A SupplementalProperty element with the @schemeIdUri attribute value of "urn:mpeg:mpegI:gpcc:2020:temporalLevelCombination" represents a temporal level combination descriptor. This temporal level combination descriptor is used to define the allowed combinations between representations (video files) of different temporal levels.

[0089] It is understandable that media segment information can be described using MPD signaling in a DASH signaling description file, and a DASH signaling description file can contain one or more Adaptation Sets. Therefore, an MPD signaling can contain one or more temporal hierarchical combined descriptors.

[0090] In one possible implementation, one track corresponds to one temporal-level combination descriptor, which contains a combination identifier element. The value of the combination identifier element indicates the video file that is allowed for combination decoding. The syntax and semantics of this temporal-level combination descriptor are shown in Table 1.

[0091] Table 1. Syntax and semantics of temporal hierarchical combination descriptors

[0092]

[0093] As shown in Table 1, the combination identifier element included in the temporal level combination descriptor can be an element from an existing descriptor. For example, an existing temporal level combination descriptor may include the element "TemporalLevelCombination". Therefore, the combination identifier element mentioned in this embodiment can be the element "TemporalLevelCombination@id" from an existing descriptor. Furthermore, the value (id) of the combination identifier element is used to indicate the video file that is allowed to be combined and decoded. In one possible implementation, the value of the combination identifier element can be equal to the identifier of the video file. For example, TemporalLevelCombination@id = 1 indicates that the video file allowed to be combined and decoded is representation1; similarly, TemporalLevelCombination@id = 2 indicates that the video file allowed to be combined and decoded is representation2, and so on.

[0094] (2) Expanding existing descriptors:

[0095] In another possible implementation, the combination identifier element included in the time-domain hierarchical combination descriptor can also be an element from a newly added time-domain hierarchical combination descriptor. The relevant definitions of existing combination identifier elements are as follows:

[0096] A SupplementalProperty element with a @schemeIdUri attribute equal to "urn:mpeg:mpegI:gpcc:2020:temporallevelIds" is referred to as a GPCCTemporalLevelId descriptor.

[0097] A GPCCTemporalLevelId descriptor is used to identify the different temporal levels present in a representation of a G-PCC content.

[0098] At most one GPCCTemporalLevelId descriptor shall be present at the Representation level for the G-PCC media when the G-PCC media is stored in multiple temporal level tracks.

[0099] At most, a single GPCCTemporalLevelId descriptor may be present at the Representation level for the G-PCC media when the G-PCC component media samples are divided into multiple temporal levels and all temporal level samples are stored in a single temporal level track.

[0100] The GPCCTemporalLevelId descriptor shall not be present at the Representation level when the G-PCC media samples are not divided based on temporal levels.

[0101] The @value attribute of the GPCCTemporalLevelId descriptor shall not be present.

[0102] Therefore, for the combined identifier element being an element in an existing temporal hierarchical combined descriptor, the syntax and semantics of the temporal hierarchical combined descriptor can be shown in Table 2:

[0103] Table 2. Syntax and semantics of temporal hierarchical combination descriptors

[0104]

[0105]

[0106] As shown in Table 2, the combination identifier element included in the temporal level combination descriptor can be the newly added element "GPCCTemporalLevelId@combinationId" in the temporal level combination descriptor. It is understood that the position of the newly added element in the temporal level combination descriptor can be customized. This embodiment does not specifically limit the position of the newly added element; for example, the position of the newly added element can be determined as the beginning of the text, the end of the text, or any position in the text, etc. Similarly, the value of the combination identifier element is used to indicate the video file that is allowed to be combined and decoded. In one possible implementation, the value of the combination identifier element can be equal to the identifier of the video file. For example, TemporalLevelCombination@id = 1 indicates that the video file allowed to be combined and decoded is representation1; and TemporalLevelCombination@id = 2 indicates that the video file allowed to be combined and decoded is representation2, and so on.

[0107] In one possible implementation, the temporal-level combined descriptor is encapsulated within the adaptation set of the signaling description file; or, the temporal-level combined descriptor is encapsulated within the representation level of the signaling description file; or, the temporal-level combined descriptor is encapsulated within the preselection level of the signaling description file. That is, the temporal-level combined descriptor can be used to describe media resources at the Representation, AdaptationSet, or Preselection levels. An AdaptationSet can include one or more representations, and a Preselection can also include one or more Preselections. Therefore, when the temporal-level combined descriptor is used to describe media resources at the AdaptationSet or Preselection level, it means that all representations within the AdaptationSet or Preselection level correspond to the same TemporalLevelCombination@id; or, all representations within the AdaptationSet or Preselection level correspond to the same GPCCTemporalLevelId@combinationId.

[0108] It should be noted that, in addition to defining the time-domain level combined descriptor mentioned above (i.e., including the combined identifier element as an element in an existing descriptor and the element in an extended time-domain level combined descriptor), the combined indication information may optionally be implemented in the following three ways in this application embodiment to generate the combined indication information.

[0109] (3) Relevant constraints are included in the standard in text form:

[0110] In one possible implementation, when media data is stored in multiple temporal-level tracks, and each temporal-level track has a replaceable track, each temporal-level track generates a corresponding replaceable track according to the target constraint rules. Therefore, the M tracks can include multiple temporal-level tracks and their corresponding replaceable tracks. A track encapsulates the corresponding media frames to obtain a corresponding video file. The media frames in the video files corresponding to multiple temporal-level tracks have the same quality, but the quality of the media frames in the video files corresponding to the multiple temporal-level tracks and the quality of the media frames in the video files corresponding to the corresponding replaceable tracks differ.

[0111] For example, suppose media data is stored in temporal tracks 1 and 2, and track1 has a replacement track 1', and track2 has a replacement track 2'. The replacement tracks 1' and 2' are generated according to target constraint rules. This means that if the media frame quality in the video file corresponding to the temporal track 1 is high quality, and the media frame quality in the video file corresponding to the replacement track 1' is low quality; then, if the media frame quality in the video file corresponding to the temporal track 2 is high quality, the media frame quality in the video file corresponding to the replacement track 2' is also low quality.

[0112] It should be noted that the terms "high quality" and "low quality" are relative concepts. In one possible implementation, the quality of a media frame can be determined based on its resolution. For example, a resolution threshold can be set by the user. If the resolution of media frame f1 is greater than the resolution threshold, then media frame f1 is a high-quality media frame; if the resolution of media frame f2 is less than or equal to the resolution threshold, then media frame f2 is a low-quality media frame.

[0113] In another possible implementation, for point cloud media, the quality of a media frame can be determined based on the components of the point cloud (e.g., attribute components and geometric components). Attribute components can include, but are not limited to, color (Red, Green, Blue) attributes; geometric components can include, but are not limited to, position (x, y, z) information. For example, if media frame f1 has attribute 1 as its attribute component and media frame f2 has attribute 2 as its attribute component, then media frames f1 and f2 are two media frames of different qualities. Similarly, if media frame f3 has geometry 1 as its geometric component and media frame f4 has geometry 2 as its geometric component, then media frames f3 and f4 are two media frames of different qualities. Likewise, if media frame f5 has geometry 1 as its geometric component and media frame f6 has attribute 1 as its attribute component, then media frames f5 and f6 are also two media frames of different qualities.

[0114] Understandably, the media frames in the video file corresponding to the temporal track 1 are identical to those in the video file corresponding to the replaceable track 1', except for their quality. Similarly, the media frames in the video file corresponding to the temporal track 2 are identical to those in the video file corresponding to the replaceable track 2', except for their quality.

[0115] In one possible implementation, the combination indication information includes a quality identifier for each video file; the value of the quality identifier indicates the quality of the media frames contained in a video file. Video files containing quality identifiers with the same value are allowed to be combined for decoding. For example, the quality identifier can be represented as @quality Ranking; then @quality Ranking = 1 can represent quality 1, @quality Ranking = 2 can represent quality 2, @quality Ranking = 3 can represent quality 3, and so on. If representation1 includes a quality identifier @quality Ranking = 1, and representation2 includes a quality identifier @quality Ranking = 1, then representation1 and representation2 are allowed to be combined for decoding; if representation1 includes a quality identifier @quality Ranking = 1, and representation2 includes a quality identifier @quality Ranking = 2, then representation1 and representation2 are not allowed to be combined for decoding.

[0116] (4) Use associationId (i.e., association identifier) ​​to associate different representations:

[0117] In one possible implementation, time-domain hierarchical tracks and replaceable tracks each correspond to a replaceable level, and tracks with the same replaceable level form a track group. The combination indication information includes an association identifier and an association type identifier. The value of the association identifier corresponding to the video file of the i-th track is determined by the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track. The value of the association type identifier indicates the relationship type between the corresponding video files; where the value of the association type identifier corresponding to the video file of the i-th track is the type of the track group to which the i-th track belongs, i is a positive integer and i ≤ M.

[0118] For example, suppose media data is stored using time-domain hierarchical tracks 'track1' and 'track2', and track1 has a replaceable track 'track1', and track2 has a replaceable track 'track2'. Furthermore, the replaceability level for time-domain hierarchical tracks 'track1' and 'track2' is level 1, and the replaceability level for replaceable tracks 'track1' and 'track2' is level 2. Then, time-domain hierarchical tracks 'track1' and 'track2' can belong to track group 1, and replaceable tracks 'track1' and 'track2' can belong to track group 2. The video file corresponding to time-domain hierarchical track 'track1' can be represented as 'representation1', the video file corresponding to time-domain hierarchical track 'track2' can be represented as 'representation2', the video file corresponding to replaceable track 'track1' can be represented as 'representation3', and the video file corresponding to replaceable track 'track2' can be represented as 'representation4'. For example, the associationId of representation1 corresponding to time-domain track 1 is the file identifier of representation2, i.e., associationId = 2 in representation1; and the associationId of representation2 corresponding to time-domain track 2 is the file identifier of representation1, i.e., associationId = 1 in representation2. Similarly, the associationId of representation3 corresponding to replaceable track 1' is the file identifier of representation4, i.e., associationId = 4 in representation3; and the associationId of representation4 corresponding to replaceable track 2' is the file identifier of representation3, i.e., associationId = 3 in representation4. In this way, video files containing media frames of the same quality can be associated using associationId, i.e., representation1 and representation2 are associated, and representation3 and representation4 are associated.

[0119] It is understandable that media frames in the video file corresponding to track i, and media frames in the video file indicated by the association identifier corresponding to track i, are allowed to be combined and decoded. Specifically, as mentioned above, different representations can be associated through associationId. For example, if representation1 and representation2 are associated, then media frames in representation1 and media frames in representation2 are allowed to be combined and decoded; similarly, if representation3 and representation4 are associated, then media frames in representation3 and media frames in representation4 are allowed to be combined and decoded.

[0120] (5) Use the Preselection tool to define a new Preselection (preselection file):

[0121] In one possible implementation, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to tracks with the same replaceable level constitute a preselection file. The signaling description file also includes a preselection descriptor, which contains the file identifier of each video file in a preselection file.

[0122] For example, suppose media data is stored in temporal-level tracks 'track1' and 'track2', and track1 has a replaceable track 'track1', and track2 has a replaceable track 'track2'. Furthermore, the replaceability level for temporal-level tracks 'track1' and 'track2' is level 1, and the replaceability level for replaceable tracks 'track1' and 'track2' is level 2. Also, the video file corresponding to temporal-level track 'track1' can be 'representation1', the video file corresponding to temporal-level track 'track2' can be 'representation2', the video file corresponding to replaceable track 'track1' can be 'representation3', and the video file corresponding to replaceable track 'track2' can be 'representation4'. Then, 'representation1' and 'representation2' can constitute a preselection file 'Preselection1', and the preselection descriptor corresponding to 'Preselection1' can include 1 and 2 (or 'Preselection1' and 'Preselection2'). Similarly, representation3 and representation4 can also form a preselection file Preselection2, and the preselection descriptors corresponding to Preselection2 can include 3 and 4 (or Preselection3 and Preselection4).

[0123] Understandably, media frames in the preselected files corresponding to the file identifiers recorded in the preselected descriptors are allowed to be decoded in combination. For example, media frames included in representation1 and representation2 are allowed to be decoded in combination; media frames included in representation3 and representation4 are allowed to be decoded in combination.

[0124] In this embodiment, the signaling description file can also be a Smart Media Transmission signaling file (SMT signaling file). When one or more media frames are encapsulated into a track, a corresponding resource file is obtained. The combination indication information includes a time-domain level combination resource descriptor; one resource file corresponds to one time-domain level combination resource descriptor, and the value of a time-domain level combination resource descriptor is the file identifier of the corresponding resource file. It is understood that media frames in resource files corresponding to time-domain level combination resource descriptors with the same value are allowed to be combined and decoded.

[0125] Specifically, the syntax and semantics of the temporal-domain hierarchical composite resource descriptor are shown in Table 3:

[0126] Table 3. Syntax and Semantics of Temporal Hierarchical Composite Resource Descriptors

[0127]

[0128] As shown in Table 3 above, `descriptor_tag` is the identifier of the temporal layer combination resource descriptor, used to identify the type of descriptor; `descriptor_length` indicates the length of the identifier of the temporal layer combination resource descriptor, in bytes. `combination_id`: Temporal layer combination identifier. When the content playback device needs to combine multiple media frames corresponding to different temporal layers, it can only select one or more representations from the set of video files containing the same `combination_id` for request.

[0129] S402: After requesting one or more video files according to the signaling description file, combine and decode the requested video files.

[0130] Specifically, the content playback device can request one or more video files from M video files based on the value of the combination identifier element in the temporal-level combination descriptor included in the signaling description file, and perform combination decoding on the requested video files. The video files being decoded have the same value in the combination identifier element in their corresponding temporal-level combination descriptors. The signaling description file may include, but is not limited to, MPD signaling files and SMT signaling files.

[0131] For example, when the combination indication information is a time-domain level combination descriptor, the signaling description file received by the content playback device may include the following information:

[0132] Representation1: {@temporalLevelId=1; @qualityRanking=0; @combinationId=100}

[0133] Representation2: {@temporalLevelId=1; @qualityRanking=1; @combinationId=200}

[0134] Representation3: {@temporalLevelId=2; @qualityRanking=0; @combinationId=100}

[0135] Representation4: {@temporalLevelId=2; @qualityRanking=1; @combinationId=200}

[0136] As described above, in this signaling description file, the `@temporalLevelId` in each video file indicates the temporal level information of the media frame. For example, if `@temporalLevelId = 1`, the temporal level of the media frame in the corresponding video file can be L1; if `@temporalLevelId = 2`, the temporal level of the media frame in the corresponding video file can be L2. It is understood that each representation can include media frames from one or more temporal levels.

[0137] The @qualityRanking property in each video file is used to indicate the quality information of the media frame. For example, if @qualityRanking = 0, the quality level of the media frame in the video file can be 0. If @qualityRanking = 1, the quality level of the media frame in the video file can be 1.

[0138] Of course, the @combinationId in each video file refers to the combination identifier element of the temporal-level combination descriptor. Media frames in video files with the same @combinationId are allowed to be combined for decoding. For example, if Representation1 has a @combinationId of 100, and Representation3 also has a @combinationId of 100 in the same signaling descriptor file, then Representation1 and Representation3 are allowed to be combined for decoding. Similarly, if Representation2 has a @combinationId of 200, and Representation4 also has a @combinationId of 200 in the same signaling descriptor file, then Representation2 and Representation4 are allowed to be combined for decoding.

[0139] In one possible implementation, the content playback device can request one or more video files according to the signaling description file based on its own needs, network conditions, or decoding capabilities, and then perform combined decoding on the requested video files. As analyzed above, after obtaining the signaling description file, the content playback device determines, based on the combination indication information (combination identifier element of the temporal-level combination descriptor) in the signaling description file, that the video files allowed for combined decoding can be: Representation1+Representation3; or Representation2+Representation4. In one embodiment, if the network conditions of the content playback device are good, or its decoding capability is strong, it can decode higher quality media frames (assuming quality 1 is higher than quality 0), then the content playback device can request Representation2+Representation4 for combined decoding; if the network conditions of the content playback device are poor, or its decoding capability is weak, it is allowed to decode lower quality media frames, then the content playback device can request Representation1+Representation3 for combined decoding. In this way, after obtaining the signaling description file, the content playback device can know in advance which video files are allowed to be combined and decoded according to the combination indication information in the signaling description file, thereby achieving the purpose of accurately requesting the appropriate media frames, saving transmission bandwidth, and improving data processing efficiency.

[0140] When the encapsulated tracks and corresponding alternative tracks are constrained in text form in the standard, in the first case, the signaling description file received by the content playback device may include the following information:

[0141] Representation1: {@temporalLevelId=1; @qualityRanking=0}

[0142] Representation2: {@temporalLevelId=1; @qualityRanking=1}

[0143] Representation3: {@temporalLevelId=2; @qualityRanking=0}

[0144] Representation4: {@temporalLevelId=2; @qualityRanking=1}

[0145] In this case, since the tracks corresponding to Representation1 and Representation3 are interchangeable, the content playback device can determine the video file to be combined for decoding based on the quality identifier (@qualityRanking). That is, the content playback device can combine and decode Representation1 and Representation3, whose corresponding quality identifiers are both 0, or it can combine and decode Representation2 and Representation4, whose corresponding quality identifiers are both 1.

[0146] In the second case, the signaling description file received by the content playback device may include the following information:

[0147] Representation1: {@temporalLevelId=1; @qualityRanking=0; @associationId=3; @associationType='gtsg'}

[0148] Representation2: {@temporalLevelId=1; @qualityRanking=1; @associationId=4; @associationType='xxx'}

[0149] Representation3: {@temporalLevelId=2; @qualityRanking=0; @associationId=1; @associationType='gtsg'}

[0150] Representation4: {@temporalLevelId=2; @qualityRanking=1; @associationId=2; @associationType='xxx'}

[0151] Therefore, the content playback device can determine the video file to be combined and decoded based on the quality identifier (@qualityRanking). That is, the content playback device can determine the value of @associationId contained in Representation1 to combine and decode Representation1 and Representation3, etc.

[0152] In the third case, the pre-selected descriptor received by the content playback device can directly record the identifiers of Representation1 and Representation3, or the identifiers of Representation2 and Representation4, so that the content playback device can determine, based on the pre-selected descriptor, to combine and decode Representation1 and Representation3, or to combine and decode Representation2 and Representation4.

[0153] In this embodiment, the content production device encapsulates N media frames into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks contain media frames of the same quality. The content production device then generates combination indication information based on the video files allowed for combined decoding and adds it to a signaling description file. After receiving the signaling description file from the content production device, the content playback device can, based on the combination indication information included in the signaling description file, know in advance which of the M video files are allowed for combined decoding. The content playback device can then select a suitable video file for consumption and decoding according to the combination indication information. Therefore, the content playback device can obtain the combination restriction information of the video files allowed for combined decoding in advance, and then request the corresponding video file for decoding and consumption based on the combination restriction information, thereby achieving the goal of accurately requesting the corresponding media frames without needing to request all video files, thus saving transmission bandwidth.

[0154] Please see Figure 5 , Figure 5 A flowchart illustrating another media data processing method provided by an exemplary embodiment of this application is shown. This method can be executed by the aforementioned content creation device (also known as an encoding device). Specifically, the content creation device can be a server or a terminal device. The server can be a standalone server, a server cluster or distributed system composed of multiple servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, and artificial intelligence platforms. Figure 5 As shown, the method includes the following steps S501-S502:

[0155] S501: Encapsulate N media frames of media data into M tracks to obtain a video file corresponding to each track. The media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1.

[0156] S502: Generate a signaling description file for media data according to the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files is allowed to be combined and decoded.

[0157] In this embodiment, N media frames can be composed of multiple media frames of different qualities. For point cloud media, the difference in quality between the corresponding media frames can be manifested in the difference in their components. Specifically, the components of a point cloud media media frame can include attribute components and geometric components. Therefore, for any two media frames (assuming Frame1 and Frame2), the difference in quality between Frame1 and Frame2 can include, but is not limited to: ① The attribute components of Frame1 and Frame2 are different, for example, the attribute component of Frame1 is attribute 1, and the attribute component of Frame2 is attribute 2; ② The geometric components of Frame1 and Frame2 are different, for example, the geometric component of Frame1 is geometry 1, and the geometric component of Frame2 is geometry 2; ③ The geometric components and attribute components of Frame1 are all different from those of Frame2, for example, the geometric component of Frame1 is geometry 1 and the attribute component is attribute 1, and the geometric component of Frame2 is geometry 2 and the attribute component is attribute 2.

[0158] When encapsulating media frames of different qualities, N media frames are divided into multiple temporal levels, with each temporal level containing one or more media frames. Encapsulating a track with corresponding media frames yields a corresponding video file, which contains media frames from one or more temporal levels. When the content production device generates a signaling description file for the media data based on the encapsulation process, it adds temporal level combination descriptors to the signaling description file. These temporal level combination descriptors serve as combination indication information within the signaling description file, defining the allowed combination methods between video files from different temporal levels. Specifically, when adding temporal level combination descriptors to the signaling description file, the content production device adds a corresponding temporal level combination descriptor to the signaling description file for each video file. These temporal level combination descriptors contain combination identifier elements; the values ​​of the combination identifier elements in the temporal level combination descriptors added to video files that are allowed to undergo combination decoding are identical.

[0159] For this temporal-level combined descriptor, the content creation device can add a temporal-level combined descriptor to the signaling description file and use the elements in the newly added temporal-level combined descriptor as combined identifier elements; or, use the elements of an existing descriptor in the signaling description file as combined identifier elements. In addition, the content creation device can encapsulate the temporal-level combined descriptor in the adaptive level of the signaling description file; or, encapsulate the temporal-level combined descriptor in the representation level of the signaling description file; or, encapsulate the temporal-level combined descriptor in the preceding level of the signaling description file.

[0160] In another implementation, when the content production device encapsulates N media frames into M tracks, it can also generate a replaceable track for any temporal level track according to the target constraint rules, provided that the media data is stored in multiple temporal level tracks and a replaceable track exists for each temporal level track. The M tracks include multiple temporal level tracks and corresponding replaceable tracks. This allows media frames of the same quality to be encapsulated into multiple temporal level tracks or into replaceable tracks, resulting in a corresponding video file for each track after encapsulating the corresponding media frames. The media frames in the video files corresponding to multiple temporal level tracks have the same quality, while the quality of the media frames in the video files corresponding to multiple temporal level tracks and the quality of the media frames in the video files corresponding to the corresponding replaceable tracks differ. Therefore, when the content production device generates a signaling description file for the media data according to the encapsulation process, it can also add a corresponding quality identifier to the signaling description file for a video file. The value of the quality identifier indicates the quality of the corresponding media frame in the video file. In this context, time-domain level tracks and replaceable tracks each correspond to a replaceable level, and tracks with the same replaceable level form a track group. Therefore, when the content production device generates a signaling description file for media data according to the encapsulation process, it can add a corresponding association identifier and association type identifier to the video file corresponding to the i-th track in the signaling description file. The value of the association identifier is determined based on the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track. The value of the association type identifier is determined based on the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the type of the track group to which the i-th track belongs, where i is a positive integer and i≤M.

[0161] In one embodiment, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to tracks with the same replaceable level constitute a pre-selection file. Therefore, when the content production device generates the signaling description file for media data according to the encapsulation process, it can first add a corresponding pre-selection descriptor to the signaling description file for each video file. The value of the pre-selection descriptor is generated based on the file identifier of each video file in a pre-selection file. The signaling description file is a description file used to describe media segment information.

[0162] In another implementation, if the signaling description file is intelligent media transmission signaling, after one or more media frames are encapsulated into a track, a corresponding resource file is obtained. The combination indication information includes a time-domain hierarchical combined resource descriptor, and one resource file corresponds to one time-domain hierarchical combined resource descriptor. The content production device can also generate a value of a time-domain hierarchical combined resource descriptor based on the file identifier of the corresponding resource file.

[0163] In this embodiment, the content creation device encapsulates N media frames of different qualities into M tracks, obtaining a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two of the M tracks contain media frames of the same quality. Based on the media frame encapsulation process, the content creation device generates a signaling description file for immersive media. This signaling description file includes combination indication information, which indicates which of the M video files is allowed to be combined and decoded. Subsequently, after obtaining the signaling description file, the content playback device can request the corresponding video files for combined consumption according to the combination indication information in the signaling description file, thereby saving transmission bandwidth.

[0164] Please see Figure 6 , Figure 6 This illustration shows a schematic diagram of a data processing apparatus 600 provided in an exemplary embodiment of this application. The data processing apparatus 600 can be applied to… Figure 1a The corresponding content playback device in the media data system. The data processing device 600 can be a computer program (including program code) running on a computer device, for example, the data processing device 600 is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of this application. The media data consists of N media frames, which are encapsulated into M tracks to obtain a video file corresponding to each track; and the media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality; N and M are both integers greater than 1. The data processing device 600 may include:

[0165] The acquisition unit 601 is used to acquire the signaling description file of the media data. The signaling description file includes combination indication information, which is used to indicate the video files among the M video files that are allowed to be combined and decoded.

[0166] The processing unit 602 is used to combine and decode the requested video files after requesting one or more video files according to the signaling description file.

[0167] In one possible implementation, N media frames are divided into multiple temporal levels, and each temporal level contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal levels.

[0168] The combination instruction information includes a temporal-level combination descriptor, which is used to define the allowed combination methods between video files at different temporal levels.

[0169] In one possible implementation, a video file corresponds to a temporal hierarchical composite descriptor, which contains a composite identifier element;

[0170] The value of the combined identifier element is used to indicate the video file that is allowed to be combined for decoding.

[0171] In one possible implementation, after the processing unit 602 requests one or more video files according to the signaling description file, it combines and decodes the requested video files to perform the following operations:

[0172] Based on the value of the combination identifier element in the temporal-level combination descriptor included in the signaling description file, request one or more video files from M video files, and perform combination decoding on each requested video file;

[0173] In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptor corresponding to the requested video file are the same.

[0174] In one possible implementation, the composite identifier element is an element in the newly added temporal-level composite descriptor; or,

[0175] The combined identifier element is an element from an existing descriptor.

[0176] In one possible implementation, the temporal-level combined descriptor is encapsulated within an adaptive hierarchy of the signaling description file; or...

[0177] The temporal-level combined descriptor is encapsulated within the representation level of the signaling description file; or,

[0178] The temporal-level combined descriptor is encapsulated in the preceding level of the signaling description file.

[0179] In one possible implementation, when media data is stored in multiple time-domain level tracks, and each time-domain level track has a replaceable track, each time-domain level track generates a corresponding replaceable track according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks;

[0180] A video file is obtained by encapsulating the corresponding media frames in a track.

[0181] The media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

[0182] In one possible implementation, the combined indication information includes a quality identifier for each video file; the value of the quality identifier is used to indicate the quality of the media frames contained in a video file corresponding to a track.

[0183] Video files containing the same quality identifier are allowed to be decoded together.

[0184] In one possible implementation, the time-domain level orbit and the replaceable orbit each correspond to a replaceable level, and orbits with the same replaceable level form an orbit group;

[0185] The combined indication information includes the association identifier and the association type identifier;

[0186] The value of the association identifier corresponding to the video file of the i-th track is determined by the file identifier of the video file corresponding to the track with the same replacement level as the i-th track. The value of the association type identifier is used to indicate the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M.

[0187] Among them, the video file corresponding to the i-th track and the video file indicated by the associated identifier corresponding to the i-th track are allowed to be combined and decoded.

[0188] In one possible implementation, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file.

[0189] The signaling description file also includes a preselection descriptor, which contains a file identifier for each video file in a preselection file;

[0190] Among them, the preselected files corresponding to the file identifiers recorded in the preselected descriptors are allowed to be combined and decoded.

[0191] In one possible implementation, the signaling description file is a description file used to describe media segment information.

[0192] In one possible implementation, the signaling description file is intelligent media transport signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained;

[0193] The combined indication information includes a time-domain level combined resource descriptor. One resource file corresponds to one time-domain level combined resource descriptor, and the value of a time-domain level combined resource descriptor is the file identifier of the corresponding resource file.

[0194] Among them, resource files corresponding to time-domain hierarchical combined resource descriptors with the same value are allowed to be combined for decoding.

[0195] In this embodiment, the content production device encapsulates N media frames into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks contain media frames of the same quality. The content production device then generates combination indication information based on the video files allowed for combined decoding and adds it to a signaling description file. After receiving the signaling description file from the content production device, the content playback device can, based on the combination indication information included in the signaling description file, know in advance which of the M video files are allowed for combined decoding. The content playback device can then select a suitable video file for consumption and decoding according to the combination indication information. Therefore, the content playback device can obtain the combination restriction information of the video files allowed for combined decoding in advance, and then request the corresponding video file for decoding and consumption based on the combination restriction information, thereby achieving the goal of accurately requesting the corresponding media frames without needing to request all video files, thus saving transmission bandwidth.

[0196] Please see Figure 7 , Figure 7 A schematic diagram of another data processing apparatus provided in an exemplary embodiment of this application is shown. This data processing apparatus 700 can be applied to… Figure 1a The corresponding content production equipment in the media data system. The data processing device 700 can be a computer program (including program code) running on a computer device; for example, the data processing device 700 is application software. This device can be used to execute the corresponding steps in the methods provided in the embodiments of this application. The data processing device 700 may include:

[0197] The encapsulation unit 701 is used to encapsulate N media frames of media data into M tracks to obtain a video file corresponding to each track. The media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1.

[0198] Processing unit 702 is used to generate a signaling description file for media data according to the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined and decoded.

[0199] In one possible implementation, N media frames are divided into multiple temporal levels, and each temporal level contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal levels.

[0200] Processing unit 702 generates a signaling description file for media data based on the encapsulation process, for performing the following operations:

[0201] Add a time-domain level combination descriptor to the signaling description file. The time-domain level combination descriptor is the combination indication information in the signaling description file. The time-domain level combination descriptor is used to define the allowed combination methods between video files of different time-domain levels.

[0202] In one possible implementation, processing unit 702 adds the time-domain hierarchical combination descriptor to the signaling description file for performing the following operations:

[0203] Add a corresponding temporal-level composite descriptor to a video file in the signaling description file. The temporal-level composite descriptor contains a composite identifier element.

[0204] In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptors added to the corresponding tracks of video files that are allowed to be combined and decoded are the same.

[0205] In one possible implementation, the processing unit 702 is further configured to perform the following operations:

[0206] Add a time-domain hierarchical composite descriptor to the signaling description file, and use the elements in the new time-domain hierarchical composite descriptor as composite identifier elements; or...

[0207] Use elements that already exist in the signaling description file as combined identifier elements.

[0208] In one possible implementation, the processing unit 702 is further configured to perform the following operations:

[0209] Encapsulate the temporal-level combined descriptor within the adaptive hierarchy of the signaling description file; or...

[0210] Encapsulate the temporal-level combined descriptor within the representation level of the signaling description file; or...

[0211] The temporal-level combined descriptor is encapsulated in the preceding level of the signaling description file.

[0212] In one possible implementation, processing unit 702 encapsulates N media frames of media data into M tracks for performing the following operations:

[0213] When media data is stored in multiple time-domain level tracks, and there are replaceable tracks for any time-domain level track, replaceable tracks for any time-domain level track are generated according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks;

[0214] Media frames of the same quality are encapsulated into multiple time-domain level tracks or into replaceable tracks, resulting in a corresponding video file for each track after encapsulating the corresponding media frames.

[0215] Among them, the media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

[0216] In one possible implementation, the processing unit 702 generates a signaling description file for the media data according to the encapsulation process, for performing the following operations:

[0217] Add a corresponding quality identifier to a video file in the signaling description file. The value of the quality identifier is used to indicate the quality of the corresponding media frame in the video file.

[0218] In one possible implementation, time-domain level orbitals and replaceable orbitals each correspond to a replaceable level, and orbitals with the same replaceable level form an orbital group.

[0219] Processing unit 702 generates a signaling description file for the media data according to the encapsulation process, and performs the following operations:

[0220] Add a corresponding association identifier and association type identifier to the video file corresponding to the i-th track in the signaling description file;

[0221] The value of the association identifier is determined based on the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track; the value of the association type identifier is determined based on the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M.

[0222] In one possible implementation, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file.

[0223] Processing unit 702 generates a signaling description file for the media data according to the encapsulation process, and performs the following operations:

[0224] Add a corresponding preselected descriptor to a video file in the signaling description file. The value of the preselected descriptor is generated based on the file identifier of each video file in a preselected file.

[0225] In one possible implementation, the signaling description file is a description file used to describe media segment information.

[0226] In one possible implementation, the signaling description file is intelligent media transmission signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained, and the combination indication information includes a time-domain level combined resource descriptor, with one resource file corresponding to one time-domain level combined resource descriptor;

[0227] Processing unit 702 is also configured to perform the following operations:

[0228] Based on the file identifier of the corresponding resource file, generate a value for a time-domain hierarchical combined resource descriptor.

[0229] In this embodiment, the content creation device encapsulates N media frames of different qualities into M tracks, obtaining a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two of the M tracks contain media frames of the same quality. Based on the media frame encapsulation process, the content creation device generates a signaling description file for immersive media. This signaling description file includes combination indication information, which indicates which of the M video files is allowed to be combined and decoded. Subsequently, after obtaining the signaling description file, the content playback device can request the corresponding video files for combined consumption according to the combination indication information in the signaling description file, thereby saving transmission bandwidth.

[0230] Please see Figure 8 , Figure 8This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. The computer device can be the content playback device described above. The computer device 800 is used to execute the steps performed by the computer device in the aforementioned method embodiments. The computer device 800 includes: one or more processors 810; one or more input devices 820; one or more output devices 830; and a memory 840. The processors 810, input devices 820, output devices 830, and memory 840 are connected via a bus 850. The memory 840 stores a computer program, which includes program instructions. The processor 810 is used to call the program instructions stored in the memory 840 to perform the following operations:

[0231] A signaling description file for acquiring media data, the signaling description file includes combination indication information, the combination indication information is used to indicate which of the M video files are allowed to be combined for decoding;

[0232] After requesting one or more video files according to the signaling description file, the requested video files are combined and decoded.

[0233] In one possible implementation, N media frames are divided into multiple temporal levels, and each temporal level contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal levels.

[0234] The combination instruction information includes a temporal-level combination descriptor, which is used to define the allowed combination methods between video files at different temporal levels.

[0235] In one possible implementation, a video file corresponds to a temporal hierarchical composite descriptor, which contains a composite identifier element;

[0236] The value of the combined identifier element is used to indicate the video file that is allowed to be combined for decoding.

[0237] In one possible implementation, after the processor 810 requests one or more video files according to the signaling description file, it combines and decodes the requested video files to perform the following operations:

[0238] Based on the value of the combination identifier element in the temporal-level combination descriptor included in the signaling description file, request one or more video files from M video files, and perform combination decoding on each requested video file;

[0239] In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptor corresponding to the requested video file are the same.

[0240] In one possible implementation, the composite identifier element is an element in the newly added temporal-level composite descriptor; or,

[0241] The combined identifier element is an element from an existing descriptor.

[0242] In one possible implementation, the temporal-level combined descriptor is encapsulated within an adaptive hierarchy of the signaling description file; or...

[0243] The temporal-level combined descriptor is encapsulated within the representation level of the signaling description file; or,

[0244] The temporal-level combined descriptor is encapsulated in the preceding level of the signaling description file.

[0245] In one possible implementation, when media data is stored in multiple time-domain level tracks, and each time-domain level track has a replaceable track, each time-domain level track generates a corresponding replaceable track according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks;

[0246] A video file is obtained by encapsulating the corresponding media frames in a track.

[0247] The media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

[0248] In one possible implementation, the combined indication information includes a quality identifier for each video file; the value of the quality identifier is used to indicate the quality of the media frames contained in a video file corresponding to a track.

[0249] Video files containing the same quality identifier are allowed to be decoded together.

[0250] In one possible implementation, the time-domain level orbit and the replaceable orbit each correspond to a replaceable level, and orbits with the same replaceable level form an orbit group;

[0251] The combined indication information includes the association identifier and the association type identifier;

[0252] The value of the association identifier corresponding to the video file of the i-th track is determined by the file identifier of the video file corresponding to the track with the same replacement level as the i-th track. The value of the association type identifier is used to indicate the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M.

[0253] Among them, the video file corresponding to the i-th track and the video file indicated by the associated identifier corresponding to the i-th track are allowed to be combined and decoded.

[0254] In one possible implementation, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file.

[0255] The signaling description file also includes a preselection descriptor, which contains a file identifier for each video file in a preselection file;

[0256] Among them, the preselected files corresponding to the file identifiers recorded in the preselected descriptors are allowed to be combined and decoded.

[0257] In one possible implementation, the signaling description file is a description file used to describe media segment information.

[0258] In one possible implementation, the signaling description file is intelligent media transport signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained;

[0259] The combined indication information includes a time-domain level combined resource descriptor. One resource file corresponds to one time-domain level combined resource descriptor, and the value of a time-domain level combined resource descriptor is the file identifier of the corresponding resource file.

[0260] Among them, resource files corresponding to time-domain hierarchical combined resource descriptors with the same value are allowed to be combined for decoding.

[0261] It is understood that the computer device can also be the aforementioned content creation device. When the computer device is a content creation device, the processor 810 is used to call program instructions stored in the memory 840, and is also used to perform the following operations:

[0262] N media frames of media data are encapsulated into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1.

[0263] The signaling description file for media data is generated based on the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined and decoded.

[0264] In one possible implementation, N media frames are divided into multiple temporal levels, and each temporal level contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal levels.

[0265] Processor 810 generates a signaling description file for media data based on the packaging process, for use in performing the following operations:

[0266] Add a time-domain level combination descriptor to the signaling description file. The time-domain level combination descriptor is the combination indication information in the signaling description file. The time-domain level combination descriptor is used to define the allowed combination methods between video files of different time-domain levels.

[0267] In one possible implementation, the processor 810 adds a time-domain hierarchical combination descriptor to the signaling description file for performing the following operations:

[0268] Add a corresponding temporal-level composite descriptor to a video file in the signaling description file. The temporal-level composite descriptor contains a composite identifier element.

[0269] In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptors added to the corresponding tracks of video files that are allowed to be combined and decoded are the same.

[0270] In one possible implementation, the processor 810 is also used to perform the following operations:

[0271] Add a time-domain hierarchical composite descriptor to the signaling description file, and use the elements in the new time-domain hierarchical composite descriptor as composite identifier elements; or...

[0272] Use elements that already exist in the signaling description file as combined identifier elements.

[0273] In one possible implementation, the processor 810 is also used to perform the following operations:

[0274] Encapsulate the temporal-level combined descriptor within the adaptive hierarchy of the signaling description file; or...

[0275] Encapsulate the temporal-level combined descriptor within the representation level of the signaling description file; or...

[0276] The temporal-level combined descriptor is encapsulated in the preceding level of the signaling description file.

[0277] In one possible implementation, processor 810 encapsulates N media frames of media data into M tracks for the following operations:

[0278] When media data is stored in multiple time-domain level tracks, and there are replaceable tracks for any time-domain level track, replaceable tracks for any time-domain level track are generated according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks;

[0279] Media frames of the same quality are encapsulated into multiple time-domain level tracks or into replaceable tracks, resulting in a corresponding video file for each track after encapsulating the corresponding media frames.

[0280] Among them, the media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

[0281] In one possible implementation, the processor 810 generates a signaling description file for the media data according to the encapsulation process, for performing the following operations:

[0282] Add a corresponding quality identifier to a video file in the signaling description file. The value of the quality identifier is used to indicate the quality of the corresponding media frame in the video file.

[0283] In one possible implementation, time-domain level orbitals and replaceable orbitals each correspond to a replaceable level, and orbitals with the same replaceable level form an orbital group.

[0284] Processor 810 generates a signaling description file for the media data according to the encapsulation process, for performing the following operations:

[0285] Add a corresponding association identifier and association type identifier to the video file corresponding to the i-th track in the signaling description file;

[0286] The value of the association identifier is determined based on the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track; the value of the association type identifier is determined based on the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M.

[0287] In one possible implementation, the temporal-level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file.

[0288] Processor 810 generates a signaling description file for the media data according to the encapsulation process, for performing the following operations:

[0289] Add a corresponding preselected descriptor to a video file in the signaling description file. The value of the preselected descriptor is generated based on the file identifier of each video file in a preselected file.

[0290] In one possible implementation, the signaling description file is a description file used to describe media segment information.

[0291] In one possible implementation, the signaling description file is intelligent media transmission signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained, and the combination indication information includes a time-domain level combined resource descriptor, with one resource file corresponding to one time-domain level combined resource descriptor;

[0292] Processor 810 is also used to perform the following operations:

[0293] Based on the file identifier of the corresponding resource file, generate a value for a time-domain hierarchical combined resource descriptor.

[0294] In this embodiment, the content production device encapsulates N media frames into M tracks to obtain a video file corresponding to each track. Media frames encapsulated into the same track have the same quality, and at least two tracks contain media frames of the same quality. The content production device then generates combination indication information based on the video files allowed for combined decoding and adds it to a signaling description file. After receiving the signaling description file from the content production device, the content playback device can, based on the combination indication information included in the signaling description file, know in advance which of the M video files are allowed for combined decoding. The content playback device can then select a suitable video file for consumption and decoding according to the combination indication information. Therefore, the content playback device can obtain the combination restriction information of the video files allowed for combined decoding in advance, and then request the corresponding video file for decoding and consumption based on the combination restriction information, thereby achieving the goal of accurately requesting the corresponding media frames without needing to request all video files, thus saving transmission bandwidth.

[0295] Furthermore, it should be noted that this application also provides a computer storage medium storing a computer program, which includes program instructions. When the processor executes these program instructions, it can perform the methods described in the preceding embodiments. Therefore, further details will not be provided here. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the program instructions can be deployed on a computer device, executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0296] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the preceding embodiments; therefore, further details will not be provided here.

[0297] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0298] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for processing media data, characterized in that, The media data consists of N media frames, which are encapsulated into M tracks to obtain a video file corresponding to each track; and the media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1; the method includes: Obtain the signaling description file of the media data, the signaling description file including combination indication information, the combination indication information being used to indicate the video files among the M video files that are allowed to be combined and decoded; After requesting one or more video files according to the signaling description file, the requested video files are combined and decoded. The N media frames are divided into multiple temporal levels. The combined indication information includes temporal level combined descriptors. The temporal level combined descriptors are encapsulated in the adaptive level of the signaling description file; or, the temporal level combined descriptors are encapsulated in the representation level of the signaling description file; or, the temporal level combined descriptors are encapsulated in the pre-level of the signaling description file.

2. The method as described in claim 1, characterized in that, A temporal hierarchy contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal hierarchy levels. The temporal-level combination descriptor is used to define the allowed combination methods between video files of different temporal levels.

3. The method as described in claim 2, characterized in that, One video file corresponds to one temporal hierarchical composite descriptor, and the temporal hierarchical composite descriptor contains a composite identifier element; The value of the combined identifier element is used to indicate the video file that is allowed to be combined for decoding.

4. The method as described in claim 3, characterized in that, After requesting one or more video files according to the signaling description file, the process of combining and decoding the requested video files includes: Based on the value of the combination identifier element in the temporal-level combination descriptor included in the signaling description file, request one or more video files from the M video files, and perform combination decoding on each requested video file; In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptor corresponding to the requested video file are the same.

5. The method as described in claim 3, characterized in that, The combined identifier element is an element in the newly added temporal-domain hierarchical combined descriptor; or, The combined identifier element is an element from an existing descriptor.

6. The method as described in claim 1, characterized in that, When the media data is stored in multiple time-domain level tracks, and there is a replaceable track for each time-domain level track, the corresponding replaceable track is generated for each time-domain level track according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks; A video file is obtained by encapsulating the corresponding media frames in a track. The media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

7. The method as described in claim 6, characterized in that, The combined indication information includes a quality identifier for each video file; the value of the quality identifier is used to indicate the quality of the media frames contained in a video file corresponding to a track. Video files containing the same quality identifier are allowed to be decoded together.

8. The method as described in claim 6, characterized in that, The time-domain level orbit and the replaceable orbit each correspond to a replaceable level, and orbits with the same replaceable level form an orbit group; The combined indication information includes an association identifier and an association type identifier; The value of the association identifier corresponding to the video file of the i-th track is determined by the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track. The value of the association type identifier is used to indicate the relationship type between the corresponding video files. The value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M. The video file corresponding to the i-th track and the video file indicated by the associated identifier corresponding to the i-th track are allowed to be decoded together.

9. The method as described in claim 6, characterized in that, The time-domain level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file; The signaling description file also includes a preselected descriptor, which contains a file identifier for each video file in a preselected file; Among them, the preselected files corresponding to the file identifiers recorded in the preselected descriptors are allowed to be combined and decoded.

10. The method according to any one of claims 1 to 9, characterized in that, The signaling description file is a description file used to describe media segment information.

11. The method as described in claim 1, characterized in that, The signaling description file is intelligent media transmission signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained; The combined indication information includes a time-domain hierarchical combined resource descriptor. One resource file corresponds to one time-domain hierarchical combined resource descriptor, and the value of one time-domain hierarchical combined resource descriptor is the file identifier of the corresponding resource file. Among them, resource files corresponding to time-domain hierarchical combined resource descriptors with the same value are allowed to be combined for decoding.

12. A method for processing media data, characterized in that, include: N media frames of media data are encapsulated into M tracks to obtain a video file corresponding to each track. The media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1. The signaling description file of the media data is generated according to the encapsulation process. The signaling description file includes combination indication information, which is used to indicate the video files that are allowed to be combined and decoded among the M video files. The N media frames are divided into multiple temporal levels. The combination indication information includes temporal level combination descriptors. The temporal level combination descriptors are encapsulated in the adaptive level of the signaling description file; or, the temporal level combination descriptors are encapsulated in the representation level of the signaling description file; or, the temporal level combination descriptors are encapsulated in the pre-level of the signaling description file.

13. The method as described in claim 12, characterized in that, A temporal hierarchy contains one or more media frames; a track encapsulates the corresponding media frames to obtain a corresponding video file, and a video file contains media frames from one or more temporal hierarchy levels. The step of generating the signaling description file for the media data according to the encapsulation process includes: The time-domain level combination descriptor is added to the signaling description file. The time-domain level combination descriptor is the combination indication information in the signaling description file. The time-domain level combination descriptor is used to define the allowed combination methods between video files of different time-domain levels.

14. The method as described in claim 13, characterized in that, Adding the time-domain hierarchical combination descriptor to the signaling description file includes: A corresponding temporal-level composite descriptor is added to a video file in the signaling description file, and the temporal-level composite descriptor contains a composite identifier element; In particular, the values ​​of the combination identifier elements in the temporal-level combination descriptors added to the corresponding tracks of video files that are allowed to be combined and decoded are the same.

15. The method as described in claim 14, characterized in that, The method further includes: A new time-domain hierarchical combined descriptor is added to the signaling description file, and the elements in the new time-domain hierarchical combined descriptor are used as the combined identifier element; or... The elements of the descriptors already existing in the signaling description file are used as the combined identifier elements.

16. The method as described in claim 12, characterized in that, The process of encapsulating N media frames of media data into M tracks includes: When the media data is stored in multiple time-domain level tracks, and there are replaceable tracks for any time-domain level track, replaceable tracks for any time-domain level track are generated according to the target constraint rules; the M tracks include multiple time-domain level tracks and corresponding replaceable tracks; Media frames of the same quality are encapsulated into the multiple time-domain level tracks or into the replaceable tracks, and a corresponding video file is obtained for each track after encapsulating the corresponding media frames. Among them, the media frames in the video files corresponding to multiple time-domain level tracks have the same quality, while the media frames in the video files corresponding to multiple time-domain level tracks and the media frames in the video files corresponding to the corresponding alternative tracks have different qualities.

17. The method as described in claim 16, characterized in that, The step of generating the signaling description file for the media data according to the encapsulation process includes: A corresponding quality identifier is added to a video file in the signaling description file. The value of the quality identifier is used to indicate the quality of the corresponding media frame in the video file.

18. The method as described in claim 16, characterized in that, The time-domain level orbit and the replaceable orbit each correspond to a replaceable level, and orbits with the same replaceable level form an orbit group; The step of generating the signaling description file for the media data according to the encapsulation process includes: Add a corresponding association identifier and association type identifier to the video file corresponding to the i-th track in the signaling description file; The value of the association identifier is determined based on the file identifier of the video file corresponding to the track with the same replaceable level as the i-th track; the value of the association type identifier is determined based on the relationship type between the corresponding video files, and the value of the association type identifier corresponding to the video file of the i-th track is the track group type to which the i-th track belongs, where i is a positive integer and i≤M.

19. The method as described in claim 16, characterized in that, The time-domain level track and the replaceable track each correspond to a replaceable level, and the video files corresponding to the tracks with the same replaceable level constitute a pre-selection file; The step of generating the signaling description file for the media data according to the encapsulation process includes: A corresponding preselected descriptor is added to a video file in the signaling description file. The value of the preselected descriptor is generated based on the file identifier of each video file in a preselected file.

20. The method according to any one of claims 12 to 19, characterized in that, The signaling description file is a description file used to describe media segment information.

21. The method as described in claim 12, characterized in that, The signaling description file is intelligent media transmission signaling; when one or more media frames are encapsulated into a track, a corresponding resource file is obtained, and the combination indication information includes a time-domain level combination resource descriptor, with one resource file corresponding to one time-domain level combination resource descriptor; The method further includes: The value of the time-domain hierarchical combined resource descriptor is generated based on the file identifier of the corresponding resource file.

22. A data processing apparatus for media data, characterized in that, The media data consists of N media frames, which are encapsulated into M tracks to obtain a video file corresponding to each track; and the media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1; the device includes: An acquisition unit is configured to acquire a signaling description file of the media data, the signaling description file including combination indication information, the combination indication information being used to indicate which of the M video files are allowed to be combined and decoded; The processing unit is configured to request one or more video files according to the signaling description file, and then combine and decode the requested video files.

23. A data processing apparatus for media data, characterized in that, The device includes: The encapsulation unit is used to encapsulate N media frames of media data into M tracks to obtain a video file corresponding to each track. The media frames encapsulated into the same track have the same quality, and at least two tracks among the M tracks contain media frames of the same quality. N and M are both integers greater than 1. The processing unit is configured to generate a signaling description file for the media data according to the encapsulation process. The signaling description file includes combination indication information, which is used to indicate which of the M video files are allowed to be combined and decoded.

24. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements a data processing method for media data as described in any one of claims 1-11 or 12-21.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform a data processing method for media data as described in any one of claims 1-11 or 12-21.

26. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and execute the data processing method for media data as described in any one of claims 1-11 or 12-21.

Images