A method for processing media data and related equipment

Abstract

An embodiment of the present application provides a method for processing media data and related devices, wherein the method includes: obtaining a description data box of the j-th track among the M tracks, the description data box including time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain levels, the time domain track indication information includes the time domain level information of the j-th track, wherein j is a positive integer and j≤M; decoding the media data according to the time domain track indication information, which can improve the flexibility of multi-track encapsulation of different time domain levels.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for processing media data, a device for processing media data, and a computer device. Background Art

[0002] In existing video coding technologies, the technology of temporal hierarchical division is supported, and in the system layer encapsulation technology, different temporal hierarchies within a track are also indicated. However, in the current system layer encapsulation technology, only the scenario of encapsulating a video bitstream using one track is considered, which shows that the current method of encapsulating different temporal hierarchies has the problem of low flexibility. Summary of the invention

[0003] The embodiments of the present application provide a method for processing media data and related devices, which can improve the flexibility when encapsulating different time domain levels.

[0004] On the one hand, an embodiment of the present application provides a method for processing media data, wherein the media data includes multiple media frames, the multiple media frames are divided into N time domain levels, and the multiple media frames are encapsulated into M tracks, where M and N are both positive integers; including:

[0005] Obtain a description data box of a j-th track among the M tracks, the description data box including time domain track indication information, the time domain track indication information is used to indicate a track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain level information of the j-th track, where j is a positive integer and j≤M;

[0006] The media data is decoded according to the time-domain track indication information.

[0007] On the one hand, an embodiment of the present application provides a method for processing media data, including:

[0008] Determine the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtain media frames of N time domain levels; wherein N is a positive integer;

[0009] The media frames of the N time domain levels are respectively encapsulated into M tracks, and a description data box of the j-th track is generated according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M.

[0010] On the one hand, an embodiment of the present application provides a media data processing device, wherein the media data includes multiple media frames, the multiple media frames are divided into N time domain levels, and the multiple media frames are encapsulated into M tracks, where M and N are both positive integers; the device includes:

[0011] an acquiring unit, configured to acquire a description data box of a j-th track among the M tracks, the description data box comprising time domain track indication information, the time domain track indication information being used to indicate a track encapsulation method of the N time domain levels, the time domain track indication information comprising time domain level information of the j-th track, wherein j is a positive integer and j≤M;

[0012] A processing unit is used to decode the media data according to the time domain track indication information.

[0013] On the one hand, an embodiment of the present application provides another media data processing device, including:

[0014] A determination unit, configured to determine the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtain media frames of N time domain levels; wherein N is a positive integer;

[0015] A processing unit is used to encapsulate the media frames of the N time domain layers into M tracks respectively, and generate a description data box of the j-th track according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain layers, and the time domain track indication information includes time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M.

[0016] In one aspect, an embodiment of the present application provides a computer device, including:

[0017] a processor adapted to implement one or more instructions; and,

[0018] The memory stores one or more instructions, and the one or more instructions are suitable for the processor to load and execute the above-mentioned related media data processing method.

[0019] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, wherein the computer program includes a first program instruction and a second program instruction, wherein the first program instruction, when executed by a processor, causes the processor to execute the method as described in the first aspect, and the second program instruction, when executed by the processor, causes the processor to execute the method as described in the second aspect.

[0020] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a video stream generated by a computer program. When the computer program is executed by a computer, the computer executes the method as described in the first aspect to decode the video stream; or executes the method as described in the second aspect to encapsulate the video stream.

[0021] On the one hand, an embodiment of the present application provides a method for storing or sending a video stream, wherein the video stream is encapsulated according to the method described in the second aspect, or is decoded based on the method described in the first aspect.

[0022] In the embodiment of the present application, the content generation device is supported to encapsulate media frames of multiple time domain levels into different tracks, and in the encapsulation process, the time domain track indication information is recorded in the description data box of each track, so as to realize the association of tracks of different time domain levels, the time domain level information in the indication track, the reference track, and the strategy information for the combination of multiple time domain level tracks. Then, the consumer-side device can select the appropriate time domain level according to the description data box in the track, and combine the samples of different tracks for decoding and presentation, thereby ensuring the flexibility of multi-track encapsulation and maximizing the savings of decoding computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of video processing provided by an exemplary embodiment of the present application is shown;

[0025] Figure 2 A schematic flow chart of a method for processing media data provided by an exemplary embodiment of the present application is shown;

[0026] Figure 3 A schematic diagram of a coding unit provided by an exemplary embodiment of the present application is shown;

[0027] Figure 4 A schematic diagram showing the division of time domain levels provided by an exemplary embodiment of the present application is shown;

[0028] Figure 5 A schematic flow chart of a method for processing media data provided by an exemplary embodiment of the present application is shown;

[0029] Figure 6a A schematic diagram of a multi-time domain multi-track packaging method provided by an exemplary embodiment of the present application is shown;

[0030] Figure 6b A schematic diagram of a multi-time domain multi-track packaging method provided by an exemplary embodiment of the present application is shown;

[0031] Figure 7 A schematic flow chart of a method for processing media data provided by an exemplary embodiment of the present application is shown;

[0032] Figure 8 A schematic diagram showing the structure of a media data processing device provided by an exemplary embodiment of the present application is shown;

[0033] Fig. 9 A schematic diagram showing the structure of a media data processing device provided by an exemplary embodiment of the present application is shown;

[0034] Fig.10 A schematic structural diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The embodiments of the present application relate to a technology for processing media data. Among them, media data (or multimedia data) refers to composite data formed by data of media such as text, graphics, images, sounds, animations, moving images, etc. that are interrelated in content, and the media data mentioned in the embodiments of the present application mainly includes audio data composed of sounds, and video data composed of images and sounds, etc., and in the embodiments of the present application, the data processing process of the media data is mainly described in detail by taking the media data as video data as an example, and when the media data is audio data, refer to the embodiments of the present invention. The processing process of media data involved in the embodiments of the present application mainly includes media data acquisition, media data encoding, media data file encapsulation, media data file transmission, media data decoding and final data presentation, and when the media data is video data, then the complete processing process for the video data can be as follows. Figure 1 As shown, it may specifically include: video acquisition, video encoding, video file encapsulation, video transmission, video file decapsulation, video decoding and final video presentation.

[0037] Video capture is used to convert analog video into digital video and save it in the format of digital video files. That is to say, video capture can convert video signals into binary digital information, where the binary information converted from the video signal is a binary data stream, which can also be called the code stream or bit stream (Bitstream) of the video signal. Video encoding is the conversion of files in the original video format into files in another video format through compression technology. The generation of video media content mentioned in the embodiments of the present application includes real scenes captured by cameras and screen content scenes generated by computers. From the perspective of the method of obtaining video signals, video signals can be divided into two methods: those captured by cameras and those generated by computers. Due to different statistical characteristics, the corresponding compression encoding methods may also be different. Modern mainstream video coding technology, taking the international video coding standard HEVC (High Efficiency Video Coding, international video coding standard HEVC / H.265), VVC (versatile video coding, international video coding standard VVC / H.266), and China's national video coding standard AVS (Audio Video Coding Standard, China's national video coding standard AVS), or AVS3 (the third-generation video coding standard launched by the AVS standard group) as examples, adopts a hybrid coding framework and performs the following series of operations and processing on the input original video signal, which can be specifically as follows: Figure 2 As shown:

[0038] ① Block partition structure: The input image (such as a video frame in video data) is divided into several non-overlapping processing units according to their size, and each processing unit will perform similar compression operations. This processing unit is called CTU (Coding Tree Unit) or LCU (Largest Coding Unit). Among them, the coding tree unit is generally divided downward from the largest coding unit, and the CTU can be further divided into more refined divisions to obtain one or more basic coding units, which are called CU (Coding Unit). Each CU is the most basic element in a coding process. The following describes various coding methods that may be used for each CU, among which the relationship between LCU (or CTU) and CU can be as follows: Figure 3 shown.

[0039] ② Predictive Coding: It includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by the selected reconstructed video signal to obtain the residual video signal. The encoder needs to choose the most suitable one among many possible predictive coding modes for the current CU and inform the decoder.

[0040] a. Intra (picture) Prediction: The predicted signal comes from the area that has been encoded and reconstructed in the same image.

[0041] b. Inter-frame prediction (Inter (picture) Prediction: Inter-frame prediction): The predicted signal comes from other images that have already been encoded and are different from the current image (called reference images).

[0042] ③ Transform coding and quantization (Transform & Quantization): The residual video signal is transformed into the transform domain through DFT (Discrete Fourier Transform), DCT (Discrete Cosine Transform, a subset of DFT) and other transformation operations. The signal is called transform coefficient. The signal in the transform domain is further subjected to lossy quantization operations, which loses certain information, making the quantized signal conducive to compression expression.

[0043] In some video coding standards, there may be more than one transform to choose from. Therefore, the encoder also needs to select one of the transforms for the current coding CU and inform the decoder. The degree of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which usually results in greater distortion and a lower bit rate. On the contrary, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which usually results in less distortion and a higher bit rate.

[0044] ④ Entropy Coding or Statistical Coding: The quantized transform domain signal will be statistically compressed and encoded according to the frequency of each value, and finally output as a binary (0 or 1) compressed bit stream. At the same time, the encoding generates other information, such as the selected mode, motion vector, etc., which also needs to be entropy encoded to reduce the bit rate.

[0045] Statistical coding is a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC).

[0046] ⑤ Loop Filtering: The encoded image can be reconstructed into a decoded image after inverse quantization, inverse transformation and prediction compensation (the reverse operation of ② to ④ above). Compared with the original image, the reconstructed image has some information different from the original image due to the influence of quantization, resulting in distortion. Filtering operations such as deblocking, SAO (Sample Adaptive Offset) or ALF (Adaptive Loop Filter) on the reconstructed image can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images and used to predict future signals, the above filtering operation is also called loop filtering, and filtering operations within the encoding loop.

[0047] Figure 2 The basic flow of a video encoder is shown in Figure 2 The kth CU (marked as S k [x, y]) is used as an example, where k is a positive integer greater than or equal to 1 and less than or equal to the number of CUs in the current input image, S k [x, y] represents the pixel with coordinates [x, y] in the kth CU, where x represents the horizontal coordinate of the pixel and y represents the median coordinate of the pixel. k [x, y] is obtained through a better processing such as motion compensation or intra-frame prediction to obtain a prediction signal S k [x, y] and Subtract the residual signal U k [x, y], and then the residual signal U k [x, y] is transformed and quantized. The quantized output data has two different destinations: one is to send it to an entropy encoder for entropy encoding. The encoded bit stream is output to a buffer for storage and waiting to be transmitted; the other application is to obtain the signal U' after inverse quantization and inverse transformation. k [x, y]. Transform the signal U' k [x, y] and Add together to get the new prediction signal S* k [x, y], and S* k [x, y] is sent to the current image buffer for storage. S* k [x, y] is obtained through intra-image prediction f(S* k [x, y]), S* k [x, y] is obtained after loop filtering S' k [x, y], and S' k [x, y] is sent to the decoded image buffer for storage to generate the reconstructed video. S' k [x, y] is obtained after motion-compensated prediction S' r [x+m x , y+m y ], S' r [x+m x , y+m y ] represents the reference block, m x and m y Represent the horizontal and vertical components of the motion vector respectively.

[0048] After encoding the media data, the encoded data stream needs to be encapsulated and transmitted to the user. Video file encapsulation refers to storing the encoded and compressed video and audio in a file in a certain format according to the encapsulation format (or container, or file container). Common encapsulation formats include AVI format (Audio Video Interleaved) or ISOBMFF (ISO Based Media File Format, a media file format based on the ISO (International Standard Organization) standard). ISOBMFF is the encapsulation standard for media files. The most typical ISOBMFF file is the MP4 (Moving Picture Experts Group 4) file. The main improvement points of the embodiment of the present application are also for the ISOBMFF data box. In one embodiment, the audio code stream and the video code stream are encapsulated in a file container according to the file format of ISOBMFF to form an encapsulated file. In the encapsulated file, the sample is used as the encapsulation unit in the file encapsulation process, and a encapsulated file consists of multiple samples. That is to say, in the process of encapsulating media files, a media frame is usually encapsulated as a sample to generate an encapsulation file. When the media data is video media, the media frame is a video frame, and when the media data is audio media, the media frame is an audio frame. That is, the encapsulation file for video media includes multiple video frames, and the encapsulation file for audio media includes multiple audio frames. In the embodiment of the present application, the media data is video media, and a sample in the encapsulation file is a video frame in the video media for example.

[0049] The encapsulated file will be transmitted to the user terminal through video, and the user terminal can present the final video content in the user terminal after performing reverse operations such as decapsulation and decoding. The encapsulated file can be sent to the user terminal through a transmission protocol, and the transmission protocol can be, for example, DASH (dynamic adaptive streaming over HTTP, which is an adaptive bitrate streaming technology based on HTTP). Using DASH for transmission allows high-quality streaming media to be transmitted over the Internet through traditional HTTP network servers. In DASH, MPD (media presentation description, media presentation description signaling in DASH) is used to describe media segment information, and in DASH, a combination of one or more media components, such as a video file of a certain resolution, can be regarded as a Representation, and the multiple Representations contained can be regarded as an Adaptation Set (a set of video streams), and a DASH can contain one or more Adaptation Sets.

[0050] It can be understood that the file decapsulation process of the user terminal is the opposite of the above-mentioned file encapsulation process. The user terminal can decapsulate the encapsulated file according to the file format requirements at the time of encapsulation to obtain audio code streams and video code streams. The decoding process of the user terminal is also the opposite of the encoding process. The user terminal can decode the audio code stream and restore the audio content. According to the above encoding process, it can be seen that at the decoding end, for each CU, after the decoder obtains the compressed code stream, it first performs entropy decoding to obtain various mode information and quantized transform coefficients. Each coefficient is dequantized and inversely transformed to obtain a residual signal. On the other hand, based on the known encoding mode information, the prediction signal corresponding to the CU can be obtained. After adding the two, the reconstructed signal can be obtained. Finally, the reconstructed value of the decoded image needs to undergo a loop filtering operation to generate the final output signal.

[0051] In video coding technology, there is also a time domain stratification technology, which can divide different video frames into different time domain levels according to the dependencies during decoding. Specifically, the time domain stratification technology is used to divide the time domain levels. The video frames divided into low levels do not need to refer to higher level video frames during decoding, such as Figure 4 As shown, the arrows represent the dependency relationship during decoding. The arrow from the I0 frame to the B1 frame indicates that the B1 frame needs to refer to the I0 frame for decoding, that is, the decoding of the B1 frame must rely on the decoding of the I0 frame, and the relationship between the remaining frames is similar. Figure 4It can be seen from the inter-frame dependency relationship indicated by the arrow in the middle that all video frames are divided into four time domain levels, L0 to L3, according to the inter-frame dependency relationship. Video frames belonging to each time domain level do not depend on video frames of higher levels during decoding. It should be noted that the low and high time domain levels mentioned in the embodiments of the present application are relative concepts, such as Figure 4 Among the four time domain layers L0 to L3 determined in the specification, for the L0 time domain layer, L1 to L3 are all high time domain layers, and for the L1 time domain layer, the L3 time domain layer is the high time domain layer of L1, and the L0 time domain layer is the low time domain layer of L1.

[0052] like Figure 4 As shown in the figure, the types of video frames mainly include I frame (Intra Slice), B frame and P frame. Among them, I frame is also called key frame, which belongs to intra-frame compression. When decoding, only the information of I frame itself needs to be referred. B frame is bidirectional prediction coding frame. When decoding, it needs to refer to both the previous existing frame and the next frame to be decoded. P frame is forward prediction coding frame, that is, when decoding, P frame needs to refer to the information of previous related frames to be decoded. Figure 4 The Arabic numeral subscripts added to I frames, B frames and P frames in the figure are used to indicate the corresponding time domain levels in which they are located. It can be understood that based on the characteristics of the three types of video frames, I frames, P frames and B frames, when decoding, in order to make the video frames belonging to the low time domain level in the video frames belonging to each time domain level after the time domain level division, they will not rely on the high time domain level when decoding. Then it can be understood that the video frames in the lowest time domain level (such as the L0 time domain level mentioned above) will not rely on the video frames belonging to any other time domain level when decoding, that is, the video frames belonging to the lowest time domain level can be independently decoded and displayed, so the video frames divided into the lowest time domain level must include I frames.

[0053] When dividing the video frames into time domain levels, the video frames belonging to the low time domain level do not need to refer to the video frames of the high time domain level when decoding, such as Figure 4 As shown, it is assumed that a video frame in the video data includes four time domain levels L0 to L3, and Figure 4The arrows in the figure are used to indicate the dependency of each video frame during decoding. That is, the arrow from the I0 frame to the B1 frame indicates that the B1 frame at the L1 time domain level needs to refer to the I0 frame at the L0 time domain level during decoding, the B1 frame at the L1 time domain level needs to refer to the P0 frame at the L0 time domain level during decoding, the first B2 frame at the L2 time domain level needs to refer to the I0 frame at the L0 time domain level and the B1 frame at the L1 time domain level during decoding, the second B2 frame at the L2 time domain level needs to refer to the B1 frame at the L1 time domain level and the P0 frame at the L0 time domain level during decoding, and the first B2 frame at the L2 time domain level needs to refer to the B1 frame at the L1 time domain level and the P0 frame at the L0 time domain level during decoding. The first B3 frame at the 3 time domain layer needs to refer to the first B2 frame at the L2 time domain layer and the I0 frame at the L0 time domain layer when decoding. The second B3 frame at the L3 time domain layer needs to refer to the first B2 frame at the L2 time domain layer and the B1 frame at the L1 time domain layer when decoding. The third B3 frame at the L3 time domain layer needs to refer to the B1 frame at the L1 time domain layer and the second B2 frame at the L2 time domain layer when decoding. The fourth B3 frame at the L3 time domain layer needs to refer to the second B2 frame at the L2 time domain layer and the P0 frame at the L0 time domain layer when decoding.

[0054] The existing AVS3 video coding technology can support the temporal layer division technology (or temporal stratification technology). At the same time, in the AVS3 system layer encapsulation technology, different temporal layers in the track are also indicated. When the video frame is encapsulated by using the existing temporal layer support technology, the temporal layer number field (temporal_layer_num) in the encapsulation file can be used to indicate the number of temporal layers in the video bitstream corresponding to the track encapsulating each video frame. In addition, the temporal layer identification field (temporal_layer_id) will be used to indicate the temporal layer to which each video frame belongs in the video bitstream corresponding to the track encapsulating the target video frame. Among them, the track refers to a series of samples with time attributes that are encapsulated in accordance with the ISO base media file format (ISOBMFF), such as a video track, which is obtained by encapsulating the bitstream generated by the video encoder after encoding each frame according to the ISOBMFF specification. The existing AVS3 decoder configuration information (ie, description data box) records the decoder configuration information for the AVS3 encoding method. The decoding configuration information can be represented by configuration information 1, and the configuration information 1 is as follows:

[0055] class Avs3DecoderConfigurationRecord{ / / AVS3 decoder configuration record

[0056] unsigned int(8)configurationVersion; / / 8-bit unsigned integer configuration version field

[0057] unsigned int(8)profile_id; / / profile identifier

[0058] unsigned int(8)level_id; / / level identifier

[0059] bit(6)reserved = '111111'b; / / Reserved field. Generally, a field requires an integer number of bytes, so the reserved bits are needed to fill the gap.

[0060] unsigned int(2)chroma_format; / / chroma format

[0061] bit(5)reserved='11111'b; / /

[0062] unsigned int(3)encoding_precision; / / encoding precision

[0063] bit(4)reserved = '1111'b; / /

[0064] unsigned int(4)frame_rate_code; / / encoding frame rate

[0065] bit(6)reserved='111111'b; / /

[0066] unsigned int(2)library_indication; / / library indication

[0067] bit(4)reserved='11111'b; / /

[0068] unsigned int(3)temporal_layer_num; / / temporal layer number field

[0069] }

[0070] From the above, it can be seen that although the technology of time domain layer division is supported in AVS3 video encoding technology, and different time domain layers in the track are also indicated in AVS3 system layer encapsulation technology. However, in the current system layer encapsulation technology, only the scenario of encapsulating the video bitstream with one track is considered. If the user encapsulates a video bitstream containing different time domain layers into multiple video file tracks, it is difficult for the existing technology to provide sufficient information to support the user to selectively reconstruct the video bitstream through different file tracks and consume it.

[0071] Based on this, the media data processing method proposed in the embodiment of the present application can support multi-track file encapsulation for the time domain hierarchical division technology in AVS3 encoding. When the media data is video data, the specific implementation steps are as follows:

[0072] 1. In the video encoding process, the content generation device can determine the temporal hierarchy for different video frames according to the inter-frame dependency between the video frames of the video data;

[0073] 2. According to the temporal level of the video frame, the video bitstream is encapsulated into multiple different tracks, and the specific temporal level information contained in each file track is indicated, including the temporal level ID, the frame rate and bit rate information corresponding to the temporal level, etc.;

[0074] 3. For the track containing the lowest temporal level, it is marked as the reference track, and the other high temporal level tracks are indexed to the reference track through 'tlrf'. At the same time, in the file encapsulation of the reference track, the strategy information of combining multiple temporal levels is given;

[0075] 4. On the user consumption side, the user's consumption side device can select one or more tracks corresponding to the required time domain level according to its own device capabilities and the policy information of the combination of multiple time domain levels, decapsulate these different tracks and reconstruct them into a bit stream for decoding. Ultimately, the purpose of flexibly selecting file tracks and saving decoding computing resources is achieved.

[0076] To support the above steps, the embodiment of the present application adds several descriptive fields at the system layer, taking the form of extending the existing ISOBMFF data box as an example, and defines relevant fields to support the multi-track file encapsulation technology of AVS3 time domain hierarchical division. Figure 5 , the media data processing method proposed in the embodiment of the present application is described in detail, wherein the media data processing method can be executed by any consumer-side device that consumes media content. It can be understood that the consumer-side device includes a terminal device (or server) where a user who consumes media content is located, such as Figure 5 As shown, the method may specifically include:

[0077] S501, obtaining a description data box of a j-th track among M tracks, where the description data box includes time domain track indication information, where the time domain track indication information is used to indicate a track encapsulation method of N time domain levels, and the time domain track indication information includes time domain level information of the j-th track, where j is a positive integer and j≤M.

[0078] First, the description data box obtained by the consumer-side device is generated based on the encoding and encapsulation of the media data by the content generating device. When encapsulating the media data, the content generating device can encapsulate the media frames belonging to different time domain levels into multiple different tracks based on the time domain level to which each media frame belongs. The media data includes multiple media frames, and the multiple media frames are divided into N time domain levels. Then, the multiple media frames can be encapsulated into M tracks, where M and N are both positive integers. It should be noted that the track encapsulating the multiple media frames can be divided into a reference track and a non-reference track according to the characteristics of the video frames encapsulated in the track during decoding, wherein the reference track means that the media frames encapsulated in the track can be decoded independently, that is, the media frames encapsulated in the reference track will not refer to the media frames in any other tracks when decoding. It can be understood that when the media data is video data, the media frames encapsulated in the reference track must include I frames. Based on the above, the video frames classified into the plantar time domain level must also include I frames, that is to say, the time domain levels encapsulated in the reference track must include the lowest time domain levels.

[0079] That is to say, if the media data is video data, when the content generation device needs to send the video data to the user side for consumption and display, it can first determine the time domain level for different video frames according to the inter-frame dependency of each video frame during video encoding, and then encapsulate the video bitstream into multiple different tracks according to the time domain level, and indicate the specific time domain level information in each file track through the description data box. Then, when consuming on the user side, the time domain level can be selected based on the decoding capability of its own terminal device, thereby saving decoding computing resources. In one embodiment, if the media frames included in a piece of media data belong to N time domain levels respectively, then after the content generation device encapsulates the media frames belonging to the N time domain levels into one or more tracks, it will generate a corresponding description data box in each track, so that the consumption side device (such as the terminal device on the user side) can determine the encapsulation method of the video frames belonging to the N time domain levels by the content generation device based on the record of the description data box, and further select the video frames of the appropriate time domain level for decoding and display.

[0080] In the embodiment of the present application, the description data box is a multi-track file encapsulation technology that supports VAS3 time domain hierarchical division by adding time domain track information to the existing ISOBMFF data box. It can be understood that adding time domain track information to the ISOBMFF data box includes one or more related fields added to the ISOBMFF data box. In the embodiment of the present application, taking the description data box of the jth track among the M tracks as an example, the related fields extended in the description data box are described in detail. Among them, the related fields extended in the description data box can be shown in configuration information 2, and the configuration information 2 is as follows:

[0081]

[0082]

[0083]

[0084] Among them, the fields included in the time domain track indication information included in the description data box are respectively the multi-time domain track identification field (multi_temporal_track_flag), the total time domain layer number field (total_temporal_layer_num), the time domain layer number field (time domain layer number field), the time domain layer level identification field (temporal_layer_id[i]), the frame rate field (frame_rate_code[i]), the low bit rate field (temporal_bit_rate_lower[i]), the high bit rate field (temporal_bit_rate_upper[i]), the base track identification field (base_track_flag), the track identification field (track_ID[i]), the priority decoding presentation field (is_output_track_flag[i]), the alternative track field (is_alternative_track_flag[i]), and the alternative track identification field (alternate_track_ID). Among them, the temporal layer number field (temporal_layer_id[i]), frame rate field (frame_rate_code[i]), low bit rate field (temporal_bit_rate_lower[i]) and high bit rate field (temporal_bit_rate_upper[i]) in the above-mentioned fields are used to indicate the specific temporal layer information in the corresponding track (such as the j-th track mentioned above).

[0085] The following describes the semantics and syntax involved in configuration information 2 in detail:

[0086] (1) The multi-temporal track identification field (multi_temporal_track_flag) is used to indicate the track encapsulation mode of N time domain layers of media data, and the track encapsulation mode includes: a multi-track encapsulation mode and a single track encapsulation mode. When the multi-temporal track identification field is a first value, the multi-temporal track identification field is used to indicate that multiple media frames belonging to the N time domain layers are encapsulated into multiple different tracks, and when the multi-temporal track identification field is a second value, the multi-temporal track identification field is used to indicate that multiple media frames belonging to the N time domain layers are encapsulated into a single track. Specifically, the first value may be 1, and the second value may be 0.

[0087] (2) The temporal layer number field (temporal_layer_num) is used to indicate the number of temporal layers contained in the current track (i.e., the j-th track mentioned above). As shown in configuration information 2, when the value of the temporal layer number field is greater than 1, that is, the i-th track encapsulates multiple temporal layers, or when the value of the multi-temporal track identification field is 1, that is, the N temporal layers of the media data are encapsulated into multiple different tracks, the consumer-side device can further read the values ​​of the relevant fields from the description data box during decoding, and then decode and display according to the values ​​of the relevant fields, wherein the description data box is a data box of the 'tlin' type.

[0088] (3) The description data box in the j-th track includes the specific temporal layer information of the j-th track. The temporal layer information includes a temporal layer identification field (temporal_layer_id[i]). The temporal layer field is used to indicate the ID (Identity document, a unique identity) of a single temporal layer, wherein one temporal layer among the N temporal layers corresponds to one temporal_layer_id, that is, the temporal layer identification field can be used to indicate the layer identification of the i-th temporal layer among the N temporal layers, wherein i and N are both positive integers, and i∈[0, N-1]. Combined with configuration information 2, it can be known that in the description data box of the j-th track, the layer identification of any i-th temporal layer will be recorded in temporal_layer_id[i].

[0089] In addition, the time domain layer information in the j-th track also includes a frame rate field (frame_rate_code[i]) and a bit rate. The frame rate field is used to indicate the frame rate when the media frame belonging to the i-th time domain layer is accumulated (that is, the time domain layer is equal to temporal_layer_id[i]). The bit rate information is used to indicate the bit rate when the media frame belonging to the i-th time domain layer is accumulated (that is, the time domain layer is equal to temporal_layer_id[i]). The bit rate information includes a low bit rate field (temporal_bit_rate_lower[i]), which is used to indicate the lower 18 bits of the bit rate when the media frame belonging to the i-th time domain layer is accumulated. In addition, the bit rate information also includes a high bit rate field (temporal_bit_rate_upper[i]), which is used to indicate the upper 12 bits of the bit rate when the media frame belonging to the i-th time domain layer is accumulated. Here, accumulation means assuming that temporal_layer_id[i]=3, then when temporal_layer_id[i]=3, its corresponding frame rate (and bit rate) is the frame rate (and bit rate) of all frames whose temporal_layer_id[i] is less than 3.

[0090] As shown in the above-mentioned configuration information 2, if the value of the multi-time domain track identification field in the description data box is 0, that is, indicating that the multimedia frames belonging to N time domain levels in the media data are encapsulated into a single track, the consumer-side device can determine the corresponding frame rate and bit rate accumulated to each time domain level by reading the values ​​in the time domain level identification field, frame rate field, low bit rate field, and high bit rate field, so that the consumer-side device can select video frames belonging to some or all time domain levels for decoding and display based on its own decoding performance.

[0091] (4) When the value of the multi-time domain track identification field is 1, it indicates that the multimedia frames belonging to N time domain levels in the media data are encapsulated into multiple different tracks. When the multimedia frames belonging to N time domain levels are encapsulated into multiple different tracks, before the consumer-side device reads the values ​​in the frame rate field, the low bit rate field, and the high bit rate field to determine the accumulated frame rate and bit rate corresponding to each time domain level, the consumer-side device needs to read the values ​​of some other fields. The fields that the consumer-side device needs to read include some or all of the fields mentioned in ① to ④ below:

[0092] ① The time domain track indication information includes the base track identification field (base_track_flag).

[0093] The reference track identification field is used to indicate whether the jth track is a reference track; when the reference track identification field is a first value, the reference track identification field is used to indicate that the jth track is a reference track, and when the reference track identification field is a second value, the reference track identification field is used to indicate that the jth track is a non-reference track; wherein the media frames encapsulated in the reference track are decoded independently, wherein the first value may be 1, and the second value may be 0. It can be seen from the above configuration information 2 that when the content generation device performs multi-track encapsulation of video frames belonging to N time domain levels, the frame rate and bit rate accumulated to the video frames belonging to each time domain level are recorded in the reference track, that is, only the frame rate field, low bit rate field, and high bit rate field in the description data box of the reference track have values, while in the description data box of the non-reference track, the frame rate field, low bit rate field, and high bit rate field are empty.

[0094] In one embodiment, when the consumer-side device reads the value of the reference track identification field as 1 from the description data box of the j-th track, it means that the j-th track is the reference track. Furthermore, the consumer-side device can also read the values ​​of the frame rate field, the low bit rate field and the low bit rate field from the description data box of the j-th track, so as to determine the frame rate and bit rate corresponding to each time domain level when it is accumulated.

[0095] ②Total temporal layer number field (total_temporal_layer_num).

[0096] The total number of temporal layers field is used to indicate the total number of temporal layers corresponding to all tracks contained in the current file, that is, to indicate the total number of temporal layers encapsulated in M ​​tracks. When multiple temporal layers are encapsulated in a multi-track manner, if the jth track is a reference track, then when the content generation device determines the value of the temporal layer identification field (temporal_layer_id[i]) in the description data box of the reference track, it will record the identification of each temporal layer in the description data box of the reference track based on the total number of temporal layers field.

[0097] ③Index type identification field ('tlrf').

[0098] The index type identification field is used to define the index relationship between the reference track (or reference time domain layer track) and the non-reference track (or high time domain layer track) when multiple time domain layers use multi-track encapsulation, where the reference time domain layer track is the track containing the lowest time domain layer ID. There is only one reference time domain layer track in a file, and the remaining tracks containing each time domain layer are all high time domain layer tracks.

[0099] The high-level temporal track should be indexed to the reference temporal track on which its decoding depends through the track index data box (TrackReferenceBox). The corresponding track index type data box (TrackReferenceTypeBoxes) should be added to the TrackReferenceBox of the high-level temporal track, where the current reference track (or reference temporal track) is indicated by track_IDs in the TrackReferenceTypeBoxes data box, where the index between the non-reference track and the reference track is identified by the corresponding index type identification field (reference_type) in TrackReferenceTypeBoxes, and the type field is defined as:

[0100] 'tlrf': The track being indexed is the base temporal level track.

[0101] That is to say, if the consumer-side device reads the value of the reference track identification field as 0 from the description data box of the j-th track, it means that the j-th track is a non-reference track. If the j-th track is a non-reference track, the j-th track also includes a track index data box, and the track index data box includes a track index type data box; the track index type data box includes a track identification field and an index type identification field; the track identification field is used to store the identification of the reference track, and the index type identification field is used to indicate that the indexed track is a reference track. It can be understood that when the j-th track is a non-reference track, since the values ​​of the frame rate field, the low bit rate field and the high bit rate field are all empty in the description data box of the non-reference track, when the j-th track is a non-reference track, the consumer-side device will not be able to read the values ​​of the frame rate field, the low bit rate field and the high bit rate field through the description data box of the j-th track, and thus cannot determine the frame rate and bit rate accumulated to each time domain level. Then, in this case, the consumer-side device can index from the non-reference track (i.e., the j-th track) to the reference track through the index type identification field in the track index type data box included in the track index data box in the j-th track, and read the values ​​of the above-mentioned frame rate field, low bit rate field and high bit rate field in the reference track.

[0102] ④ The time domain track indication information includes track combination strategy information, which includes a track identification field (track_ID[i]), a priority decoding presentation field (is_output_track_flag[i]), an alternative track field (is_alternative_track_flag[i]), and an alternative track identification field (alternate_track_ID).

[0103] The track identification field is used to indicate the identification (ID) of the track containing part of the time domain layer, and one track in the M tracks corresponds to one track_ID.

[0104] The priority decoding presentation field is used to indicate whether the j-th track (i.e., the current track) is a track for priority decoding presentation; when the priority decoding presentation field is a first value, the priority decoding presentation field is used to indicate that the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a second value, the priority decoding presentation field is used to indicate that the j-th track is not a track for priority decoding presentation; wherein, the first value may be 1, and the second value may be 0.

[0105] The alternative track field is used to indicate whether the j-th track (i.e., the current track) is a substitute track for one of the M tracks; when the alternative track field is a first value, the alternative track field is used to indicate that the j-th track is a substitute track for one of the M tracks; when the alternative track field is a second value, the alternative track field is used to indicate that the j-th track is not a substitute track; wherein, the first value may be 1, and the second value may be 0.

[0106] The replacement track identifier field is used to indicate the identifier of a track replaced by the j-th track (ie, the current track).

[0107] Based on the above description of the syntax and semantics in the configuration information 2, when the multiple media frames included in the media data belong to N different time domain levels, the content generating device can be divided into the following two cases when encapsulating the multiple media frames included in the media data:

[0108] In the first case, media frames belonging to N time domain layers can be encapsulated into a single track. Then, after obtaining the encapsulation file for the media data, the consumer-side device can determine the layer identifier of the i-th time domain layer in the N time domain layers, as well as the corresponding frame rate and bit rate by reading the layer identifier field, frame rate field, low bit rate field and high bit rate field from the description data box of the single track encapsulating the media frames belonging to the N time domain layers, so that the consumer-side device can select some or all of the media frames of the time domain layers for decoding and display in combination with its own decoding capability, and then execute step S502.

[0109] In the second case, the media frames belonging to N time domain layers can be encapsulated into multiple different tracks. When the content generation device encapsulates the media frames in a multi-track encapsulation manner, it will record the strategy of combining the time domain layers encapsulated in each track, as well as the frame rate and bit rate of the video frames corresponding to each time domain layer accumulated in the reference track, and index other tracks into the reference track through the index type identification field. Then, the consumer-side device can select part or all of the media frames for decoding and display through the relevant information recorded in the reference track and in combination with its own decoding capability, that is, turn to execute step S502.

[0110] S502: Decode and display the media data according to the time domain track indication information.

[0111] After the consumer-side device obtains the description data box in the j-th track, it will decode and display the media data according to the time domain track indication information in the description data box. Specifically, the consumer-side device can retain the time domain layers that match the decoding performance among the N time domain layers based on the time domain track indication information and the decoding performance of the decoding device, and decode and display the media frames of the retained time domain layers. When the consumer-side device retains the time domain layer that matches the decoding performance in N time domain layers according to the time domain track indication information and the decoding performance of the decoding device, in one implementation, since the time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and a bit rate information, and the code stream information includes a low bit rate field and a high bit rate field, then the consumer-side device can read the value of the multi-time domain track identification field in the time domain track indication information, and when the read multi-time domain track identification field is a second value, it indicates that the media frames of the N time domain layers are encapsulated into a single track, and reads the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information; thereby, the consumer-side device can retain the time domain layer that matches the decoding performance in the N time domain layers according to the value of the time domain layer identification field, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device (i.e., the above-mentioned consumer-side device).

[0112] If the media data is video data, after the content generation device encapsulates the video data (or video content) belonging to N time domain levels into a single track, the process of decoding and consumption by the consumer-side device is as follows:

[0113] The content generation device encodes and encapsulates the video content A. Assuming that the video content A has three time domain levels L0 to L2, the video frames of any of the three time domain levels are encapsulated into one track, and when accumulated to each time domain level, the corresponding frame rate and bit rate are as follows:

[0114] L0: 20fps, bitrate = 1mbps;

[0115] L1:30fps, bitrate=1.5mbps;

[0116] L2: 60fps, bitrate = 3mbps.

[0117] The content generation device sends video file A to user 1 and user 2 respectively according to the request of the consumption side device. User 1 and user 2 receive the corresponding file A respectively, and decode and consume it according to the frame rate and bit rate information corresponding to each time domain layer in the track. Specifically:

[0118] The decoding device of the consumer-side device where user 1 is located has better performance, and chooses to retain all video frames of L0 to L2 for decoding and presentation to obtain the best viewing effect. However, the decoding device of the consumer-side device where user 2 is located has poorer performance, and chooses to retain all video frames of L0, and may discard video frames of L1 to L2, and only decodes and presents the retained video frames belonging to L0.

[0119] When the consumer-side device retains the time domain layers that match the decoding performance in N time domain layers according to the time domain track indication information and the decoding performance of the decoding device, in another implementation manner, if the consumer-side device can read the value of the multi-time domain track identification field in the time domain track indication information, when the multi-time domain track identification field read by the consumer-side device is a first value, indicating that the media frames of the N time domain layers are encapsulated into multiple different tracks, when the time domain layers in each track do not overlap, the consumer-side device can read the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information from the reference track; the media frames encapsulated in the reference track are decoded independently, and then according to the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information read from the reference track, and the decoding performance of the decoding device, some or all of the time domain layers in the track that match the decoding performance are retained.

[0120] If the media data is video data, the content generation device encapsulates the video data (or video content) belonging to N time domain levels into multiple different tracks, and the time domains of the tracks do not overlap. The process of decoding and consuming by the consumer side device can be as follows: Figure 6a As shown, the details are as follows:

[0121] The content generation device encodes and encapsulates video content A. Assume that video content A has three time domain levels L0 to L2, and the video frames belonging to these three time domain levels are encapsulated into three different tracks, where rack1 is the reference track, and track2 and track3 are indexed to track1 with the 'tlrf' type. In the reference track, the corresponding frame rate and bit rate when indicating the accumulation to each time domain level are as follows:

[0122] L0: 20fps, bitrate = 1mbps;

[0123] L1:30fps, bitrate=1.5mbps;

[0124] L2: 60fps, bitrate = 3mbps.

[0125] The content generation device sends video file A to user 1 and user 2 respectively according to the request of the consumption side device. User 1 and user 2 receive the corresponding file A respectively, and decode and consume it according to the frame rate and bit rate information corresponding to each time domain layer in the track. Specifically:

[0126] The decoding device of the consumer-side device where user 1 is located has better performance, and chooses to retain all video frames of track1 to track3 for decoding and presentation to obtain the best viewing effect. However, the decoding device of the consumer-side device where user 2 is located has poorer performance, and chooses to retain all video frames of track1, and may discard video frames of track2 and track3, and only decodes and presents the retained video frames belonging to track1.

[0127] In another implementation, if the multi-time domain track identification field read by the consumer-side device is the first value and indicates that the media frames of N time domain layers are encapsulated into multiple different tracks, when the time domain layers in each track overlap, the consumer-side device can read the values ​​of each field in the track combination strategy information from the reference track, and retain part or all of the time domain layers in the track that match the decoding performance according to the values ​​of each field in the track combination strategy information and the decoding performance of the decoding device, wherein the time domain track indication information also includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field. Figure 6b As shown, if the media data is video data, the content generation device encapsulates video frames of the video data (or video content) belonging to N time domain levels into multiple different tracks, and the time domain levels of each track overlap, then the process of decoding and consuming by the consumer-side device can be specifically as follows:

[0128] The content generation device encodes and encapsulates video content A. Assume that video content A has three time domain levels L0 to L2, and the video frames belonging to these three time domain levels are encapsulated into three different tracks, where rack1 is the reference track, track2 and track3 are indexed to track1 with the 'tlrf' type, and Track2 and track3 each contain a portion of L1 and L2 video frames, and do not overlap each other. The decoding of track2 and track3 depends on track1, but there is no dependency between track2 and track3. In the reference track, the information indicating each track when combined:

[0129] track1: is_output_track_flag=1; is_alternative_track_flag=0:

[0130] track2: is_output_track_flag=1; is_alternative_track_flag=0:

[0131] track3: is_output_track_flag=0; is_alternative_track_flag=1: alternative_track=2.

[0132] The content generation device sends video file A to user 1 and user 2 respectively according to the request of the consumption side device. User 1 and user 2 receive the corresponding file A respectively, and decode and consume it according to the frame rate and bit rate information corresponding to each time domain layer in the track. Specifically:

[0133] The decoding device of the consumer-side device where user 1 is located has better performance, and chooses to retain all video frames of track1 to track3 for decoding and presentation to obtain the best viewing effect. The decoding device of the consumer-side device where user 2 is located has poorer performance, and chooses to retain all video frames of track1 and track2, and may discard the video frames of track3, and decode the video frames in track1 for presentation.

[0134] In one embodiment, when the number of retained media frames is one or more, the consumer-side device can reorder (i.e., reconstruct) the one or more retained media frames according to the decoding time of each media frame in the one or more retained media frames when decoding and displaying the retained media frames at the time domain level, and then decode and display the reordered one or more media frames. That is, when combining video frames of different tracks, the consumer-side device arranges all video frames in the selected multiple tracks according to the decoding time corresponding to each video frame during encapsulation, and then decodes them after reconstruction.

[0135] In the embodiment of the present application, the content generation device is supported to encapsulate media frames of multiple time domain levels into different tracks, and in the encapsulation process, the time domain track indication information is recorded in the description data box of each track, so as to realize the association of tracks of different time domain levels, the time domain level information in the indication track, the reference track, and the strategy information for the combination of multiple time domain level tracks. Then, the consumer-side device can select the appropriate time domain level according to the description data box in the track, and combine the samples of different tracks for decoding and presentation, thereby ensuring the flexibility of multi-track encapsulation and maximizing the savings of decoding computing resources.

[0136] Next, we will combine Figure 7 , describes the process of encapsulating media data by a content generation device. The content generation device can be a server or a terminal device. The server can be an independent server, a server cluster or a distributed system composed of multiple servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, and artificial intelligence platforms. See Figure 7 , is a schematic flow chart of a method for processing media data provided by an embodiment of the present application, such as Figure 7 As shown, the method may include:

[0137] S701, determining the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtaining media frames of N time domain levels; wherein N is a positive integer.

[0138] S702, respectively encapsulate media frames of N time domain levels into M tracks, and generate a description data box of the j-th track according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M.

[0139] In step S701 and step S702, the inter-frame dependency relationship of each media frame in the media data can be as follows: Figure 4As shown, the time domain level of each media frame can be determined based on the inter-frame dependency. The time domain track indication information includes a multi-time domain track identification field, which is used to indicate the track encapsulation method of N time domain levels. Then, when the content generation device generates the description data box of the jth track according to the encapsulation process of the media frame, if the content generation device encapsulates multiple media frames of N time domain levels into multiple different tracks, the value of the generated multi-time domain track identification field is a first value; and if the content generation device encapsulates multiple media frames of N time domain levels into a single track, the value of the generated multi-time domain track identification field is a second value. In addition, the time domain track indication information also includes a total time domain layer number field. Then, when the content generation device generates the description data box of the jth track according to the encapsulation process of the media frame, it will also generate the value of the total time domain layer number field according to the total number of time domain layers encapsulated by the M tracks.

[0140] Among them, the time domain layer information of the j-th track includes a time domain layer number field. Then, when the content generation device generates the description data box of the j-th track according to the encapsulation process of the media frame, it will generate the value of the time domain layer number field according to the number of time domain layers encapsulated by the j-th track. In addition, the time domain layer information of the j-th track includes a time domain layer identification field, a frame rate field and bit rate information. Then, when the content generation device generates the description data box of the j-th track according to the encapsulation process of the media frame, it can store the layer identification of the i-th time domain layer in the N time domain layers in the time domain layer identification field, where i is an integer and i∈[0, N-1], store the frame rate accumulated to the media frame belonging to the i-th time domain layer in the frame rate field, and store the bit rate accumulated to the media frame belonging to the i-th time domain layer as the bit rate information. In one embodiment, the bit rate information includes a low bit rate field and a high bit rate field; then, when the content generating device uses the bit rate accumulated for the media frame belonging to the i-th time domain layer as the bit rate information, the lower 18 bits of the bit rate for the media frame belonging to the i-th time domain layer may be stored in the low bit rate field, and the higher 12 bits of the bit rate for the media frame belonging to the i-th time domain layer may be stored in the high bit rate field.

[0141] The time domain track indication information also includes a reference track identification field. When the content generating device generates a description data box for the j-th track according to the encapsulation process of the media frame, if the j-th track is a reference track, the value of the generated reference track identification field is a first value, and if the j-th track is a non-reference track, the value of the generated reference track identification field is a second value; wherein the media frames encapsulated in the reference track are decoded independently. In addition, the time domain track indication information also includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; then, when the content generation device generates a description data box of the j-th track according to the encapsulation process of the media frame, the identification of the track containing part of the time domain layer can be stored in the track identification field, and when the j-th track is a track for priority decoding presentation, the value of the priority decoding presentation field is generated as a first value, and if the j-th track is not a track for priority decoding presentation, the value of the priority decoding presentation field is generated as a second value; and, if the j-th track is a substitute track for a track, the value of the alternative track field is generated as a first value, and the identification of a track replaced by the j-th track is stored in the alternative track identification field; if the j-th track is not a substitute track, the value of the alternative track field is generated as a second value, wherein the first value may be 1 and the second value may be 0.

[0142] In one embodiment, if the j-th track is a non-reference track, the content generation device will generate a track index data box for the j-th track, the track index data box includes a track index type data box; the track index type data box includes a track identification field and an index type identification field. In addition, the content generation device may store the identification of the reference track in the track identification field, and index the j-th track to the reference track according to the index type identification field.

[0143] In an embodiment of the present application, the content generating device can determine the time domain level of each media frame through the inter-frame dependency between the media frames included in the media data, and then can encapsulate the media frames of N time domain levels into M tracks respectively, and generate a description data box of the jth track based on the encapsulation process of the media frames, and in the description data box, set corresponding values ​​for each field included in the time domain track indication information, so as to associate tracks of different time domain levels through the values ​​of each field, indicate the time domain level information in the track, mark the reference track, indicate the strategy of combining multiple time domain level tracks with each other, thereby realizing the media frame encapsulation process of the content generating device instructing the consumer-side device, so that the consumer-side device can select the appropriate time domain level for decoding and presentation according to the values ​​of each field in the description data box in the track, thereby ensuring the flexibility of multi-track encapsulation and maximizing the saving of decoding computing resources.

[0144] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0145] See also Figure 8 , Figure 8 A schematic diagram of the structure of a media data processing device provided by an exemplary embodiment of the present application is shown; the media data processing device may be a computer program (including program code) running on the above-mentioned consumer-side device, for example, the media data processing device may be an application software in the consumer-side device. Figure 8 As shown, the media data processing device may include: an acquisition unit 801 and a processing unit 802.

[0146] In an exemplary embodiment, the media data processing device can be used to perform Figure 5 The corresponding steps in the method shown; the media data includes multiple media frames, the multiple media frames are divided into N time domain levels, and the multiple media frames are encapsulated into M tracks, where M and N are both positive integers; then:

[0147] An acquiring unit 801 is configured to acquire a description data box of a j-th track among the M tracks, where the description data box includes time domain track indication information, where the time domain track indication information is used to indicate a track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain level information of the j-th track, where j is a positive integer and j≤M;

[0148] The processing unit 802 is configured to decode the media data according to the time domain track indication information.

[0149] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, and the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers;

[0150] When the multi-time domain track identification field is a first value, the multi-time domain track identification field is used to indicate that the multiple media frames belonging to the N time domain levels are encapsulated into multiple different tracks;

[0151] When the multi-time domain track identification field is a second value, the multi-time domain track identification field is used to indicate that multiple media frames belonging to the N time domain levels are encapsulated into a single track.

[0152] In one embodiment, the time domain track indication information includes a total time domain layer number field; the total time domain layer number field is used to indicate the total number of time domain layers of the M track packages.

[0153] In one embodiment, the temporal layer information of the j-th track includes a temporal layer number field, and the temporal layer number field is used to indicate the number of temporal layers encapsulated by the j-th track.

[0154] In one embodiment, the temporal level information of the j-th track includes a temporal level identification field, a frame rate field and bit rate information;

[0155] The time domain layer identifier field is used to indicate the layer identifier of the i-th time domain layer among the N time domain layers, where i is an integer and i∈[0, N-1];

[0156] The frame rate field is used to indicate the frame rate accumulated to the media frame belonging to the i-th time domain level;

[0157] The bit rate information is used to indicate the bit rate accumulated to the media frame belonging to the i-th time domain layer.

[0158] In one embodiment, the bit rate information includes a low bit rate field and a high bit rate field;

[0159] The low bit rate field is used to indicate the lower 18 bits of the bit rate accumulated to the media frame belonging to the i-th time domain layer;

[0160] The high code rate field is used to indicate the high 12 bits of the code rate accumulated to the media frame belonging to the i-th time domain layer.

[0161] In one embodiment, the time domain track indication information includes a reference track identification field; the reference track identification field is used to indicate whether the j-th track is a reference track;

[0162] When the reference track identification field is a first value, the reference track identification field is used to indicate that the j-th track is a reference track; when the reference track identification field is a second value, the reference track identification field is used to indicate that the j-th track is a non-reference track;

[0163] The media frames encapsulated in the reference track are decoded independently.

[0164] In one embodiment, the time-domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field;

[0165] The track identification field is used to indicate the identification of the track including the partial time domain layer;

[0166] The priority decoding presentation field is used to indicate whether the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a first value, the priority decoding presentation field is used to indicate that the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a second value, the priority decoding presentation field is used to indicate that the j-th track is not a track for priority decoding presentation;

[0167] The Substitute Track field is used to indicate whether the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a first value, the Substitute Track field is used to indicate that the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a second value, the Substitute Track field is used to indicate that the j-th track is not a substitute track;

[0168] The replacement track identification field is used to indicate the identification of a track replaced by the j-th track.

[0169] In one embodiment, if the j-th track is a non-reference track, the j-th track further includes a track index data box, and the track index data box includes a track index type data box;

[0170] The track index type data box includes a track identification field and an index type identification field;

[0171] The track identification field is used to store the identification of the reference track, and the index type identification field is used to indicate that the indexed track is the reference track.

[0172] In one embodiment, the processing unit 802 is specifically configured to:

[0173] According to the time domain track indication information and the decoding performance of the decoding device, retaining the time domain layer matching the decoding performance among the N time domain layers;

[0174] The media frames of the preserved temporal layer are decoded and displayed.

[0175] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit stream information includes a low bit rate field and a high bit rate field; the processing unit 802 is specifically used to:

[0176] Reading a value of the multi-time domain track identification field in the time domain track indication information, and when the read multi-time domain track identification field is a second value, indicating that the media frames of the N time domain layers are encapsulated into a single track, and reading a value of the time domain layer identification field, a value of the frame rate field, and a value of a low bit rate field and a value of a high bit rate field in the bit rate information;

[0177] According to the value of the time domain layer identification field, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, the time domain layer among the N time domain layers that matches the decoding performance is retained.

[0178] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit stream information includes a low bit rate field and a high bit rate field; the processing unit 802 is specifically used to:

[0179] Reading the value of the multi-time domain track identification field in the time domain track indication information, when the read multi-time domain track identification field is a first value, indicating that the media frames of the N time domain layers are encapsulated into multiple different tracks, and when the time domain layers in each track do not overlap, reading the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information from the reference track; the media frames encapsulated in the reference track are independently decoded;

[0180] According to the value of the time domain layer identification field read from the reference track, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, retain part or all of the time domain layers in the track that match the decoding performance.

[0181] In one embodiment, the time-domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field;

[0182] The processing unit 802 is further configured to, when the read multi-time domain track identification field is a first value, indicate that the media frames of the N time domain levels are encapsulated into a plurality of different tracks, and that the time domain levels in each track overlap, and read the value of each field in the track combination strategy information from a reference track;

[0183] The processing unit 802 is further configured to retain, according to the values ​​of the fields in the track combination strategy information and the decoding performance of the decoding device, some or all of the time domain levels in the track that match the decoding performance.

[0184] In one embodiment, the processing unit 802 is specifically configured to:

[0185] According to the decoding time of each media frame in the retained one or more media frames, reorder the retained one or more media frames according to the decoding time;

[0186] The reordered one or more media frames are decoded and displayed.

[0187] In the embodiment of the present application, the content generation device is supported to encapsulate media frames of multiple time domain levels into different tracks, and in the encapsulation process, the time domain track indication information is recorded in the description data box of each track, so as to realize the association of tracks of different time domain levels, the time domain level information in the indication track, the reference track, and the strategy information for the combination of multiple time domain level tracks. Then, the processing unit 602 can select the appropriate time domain level according to the description data box in the track, and decode and present the samples of different tracks after combining them, thereby ensuring the flexibility of multi-track encapsulation and maximizing the saving of decoding computing resources.

[0188] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0189] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a media data processing device provided by an exemplary embodiment of the present application is shown; the media data processing device may be a computer program (including program code) running in the above-mentioned content generation device, for example, the media data processing device may be an application software in the content generation device. Fig. 9 As shown, the media data processing device may include: a determination unit 901 and a processing unit 902.

[0190] In an exemplary embodiment, the media data processing device can be used to perform Figure 7 The corresponding steps in the method shown; then:

[0191] The determining unit 901 is configured to determine the time domain level of each media frame according to the inter-frame dependency relationship of each media frame included in the media data, and obtain media frames of N time domain levels, wherein N is a positive integer;

[0192] The processing unit 902 is used to encapsulate the media frames of the N time domain layers into M tracks respectively, and generate a description data box of the j-th track according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain layers, and the time domain track indication information includes the time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M.

[0193] In an embodiment, the time domain track indication information includes a multi-time domain track identification field, and the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers; the processing unit 902 is specifically used to:

[0194] If the multiple media frames of the N time domain levels are encapsulated into multiple different tracks, the value of the multiple time domain track identification field is generated as a first value;

[0195] If the multiple media frames of the N time domain levels are encapsulated into a single track, the value of the multi-time domain track identification field is generated as a second value.

[0196] In one embodiment, the time domain track indication information includes a total time domain layer number field; the processing unit 902 is specifically configured to:

[0197] The value of the total time domain layer number field is generated according to the total number of time domain layers encapsulated by the M tracks.

[0198] In one embodiment, the time domain level information of the j-th track includes a time domain layer number field; the processing unit 902 is specifically configured to:

[0199] The value of the time domain layer number field is generated according to the number of time domain layers of the j-th track package.

[0200] In one embodiment, the time domain level information of the j-th track includes a time domain level identification field, a frame rate field and bit rate information; the processing unit 902 is specifically configured to:

[0201] The layer identifier of the i-th time domain layer among the N time domain layers is stored in the time domain layer identifier field, where i is an integer and i∈[0, N-1];

[0202] storing the frame rate accumulated when the media frame belongs to the i-th time domain level in the frame rate field;

[0203] The bit rate accumulated when the media frame belongs to the i-th time domain layer is obtained is used as the bit rate information.

[0204] In one embodiment, the bit rate information includes a low bit rate field and a high bit rate field; the processing unit 902 is specifically configured to:

[0205] The lower 18 bits of the bit rate of the media frame belonging to the i-th time domain level are stored in the low bit rate field;

[0206] The high 12 bits of the code rate of the media frame belonging to the i-th time domain layer are stored in the high code rate field.

[0207] In an embodiment, the time domain orbit indication information includes a reference orbit identification field; the processing unit 902 is specifically configured to:

[0208] If the j-th track is a reference track, the value of the reference track identification field is generated as a first value;

[0209] If the j-th track is a non-reference track, the value of the reference track identification field is generated as a second value;

[0210] The media frames encapsulated in the reference track are decoded independently.

[0211] In one embodiment, the time domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; the processing unit 902 is specifically used to:

[0212] storing an identification of a track comprising the partial temporal hierarchy in the track identification field;

[0213] If the j-th track is a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a first value; if the j-th track is not a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a second value;

[0214] If the j-th track is a replacement track for a track, the value of the replacement track field is generated as a first value, and the identifier of the track replaced by the j-th track is stored in the replacement track identifier field; if the j-th track is not a replacement track, the value of the replacement track field is generated as a second value.

[0215] In one embodiment, the processing unit 902 is further configured to generate a track index data box of the jth track if the jth track is a non-reference track, wherein the track index data box includes a track index type data box; the track index type data box includes a track identification field and an index type identification field;

[0216] The processing unit 902 is further configured to store the identifier of the reference track in the track identifier field, and index the j-th track to the reference track according to the index type identifier field.

[0217] In an embodiment of the present application, the processing unit 902 can determine the time domain level of each media frame through the inter-frame dependency between the media frames included in the media data, and then can encapsulate the media frames of N time domain levels into M tracks respectively, and generate a description data box of the jth track based on the encapsulation process of the media frames, and in the description data box, set corresponding values ​​for each field included in the time domain track indication information, so as to associate tracks of different time domain levels through the values ​​of each field, indicate the time domain level information in the track, mark the reference track, indicate the strategy of combining multiple time domain level tracks with each other, so as to instruct the consumer-side device, the media frame encapsulation process of the processing unit 902, so that the consumer-side device can select the appropriate time domain level for decoding and presentation according to the values ​​of each field in the description data box in the track, thereby ensuring the flexibility of multi-track encapsulation and maximizing the saving of decoding computing resources.

[0218] See also Fig.10 , is a schematic block diagram of a computer device provided by an embodiment of the present invention, the computer device may be the above-mentioned consumer-side device, or may also be the above-mentioned content generation device, wherein the computer device may be a server or a terminal device. Fig.10 The computer device in the embodiment shown may include: one or more processors 101; one or more input devices 102, one or more output devices 103 and a memory 104. The processor 101, the input device 102, the output device 103 and the memory 104 are connected via a bus 105. The memory 104 is used to store a computer program, the computer program includes program instructions, and the processor 101 is used to execute the program instructions stored in the memory 104.

[0219] The memory 104 may include a volatile memory, such as a random-access memory (RAM); the memory 104 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 104 may also include a combination of the above types of memory.

[0220] The processor 101 may be a central processing unit (CPU). The processor 101 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc. The processor 101 may also be a combination of the above structures.

[0221] In the embodiment of the present invention, the memory 104 is used to store a computer program, the computer program includes program instructions, and the processor 101 is used to execute the program instructions stored in the memory 104 to implement the above-mentioned Figure 5 The steps involved in the method for processing media data in the embodiment of the present invention, wherein the media data includes multiple media frames, the multiple media frames are divided into N time domain levels, and the multiple media frames are encapsulated into M tracks, wherein M and N are both positive integers.

[0222] In one embodiment, the processor 101 is configured to call the program instructions to execute:

[0223] Obtain a description data box of a j-th track among the M tracks, the description data box including time domain track indication information, the time domain track indication information is used to indicate a track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain level information of the j-th track, where j is a positive integer and j≤M;

[0224] The media data is decoded according to the time-domain track indication information.

[0225] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, and the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers;

[0226] When the multi-time domain track identification field is a first value, the multi-time domain track identification field is used to indicate that the multiple media frames belonging to the N time domain levels are encapsulated into multiple different tracks;

[0227] When the multi-time domain track identification field is a second value, the multi-time domain track identification field is used to indicate that multiple media frames belonging to the N time domain levels are encapsulated into a single track.

[0228] In one embodiment, the time domain track indication information includes a total time domain layer number field; the total time domain layer number field is used to indicate the total number of time domain layers of the M track packages.

[0229] In one embodiment, the temporal layer information of the j-th track includes a temporal layer number field, and the temporal layer number field is used to indicate the number of temporal layers encapsulated by the j-th track.

[0230] In one embodiment, the temporal level information of the j-th track includes a temporal level identification field, a frame rate field and bit rate information;

[0231] The time domain layer identifier field is used to indicate the layer identifier of the i-th time domain layer among the N time domain layers, where i is an integer and i∈[0, N-1];

[0232] The frame rate field is used to indicate the frame rate accumulated to the media frame belonging to the i-th time domain level;

[0233] The bit rate information is used to indicate the bit rate accumulated to the media frame belonging to the i-th time domain layer.

[0234] In one embodiment, the bit rate information includes a low bit rate field and a high bit rate field;

[0235] The low bit rate field is used to indicate the lower 18 bits of the bit rate accumulated to the media frame belonging to the i-th time domain layer;

[0236] The high code rate field is used to indicate the high 12 bits of the code rate accumulated to the media frame belonging to the i-th time domain layer.

[0237] In one embodiment, the time domain track indication information includes a reference track identification field; the reference track identification field is used to indicate whether the j-th track is a reference track;

[0238] When the reference track identification field is a first value, the reference track identification field is used to indicate that the j-th track is a reference track; when the reference track identification field is a second value, the reference track identification field is used to indicate that the j-th track is a non-reference track;

[0239] The media frames encapsulated in the reference track are decoded independently.

[0240] In one embodiment, the time-domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field;

[0241] The track identification field is used to indicate the identification of the track including the partial time domain layer;

[0242] The priority decoding presentation field is used to indicate whether the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a first value, the priority decoding presentation field is used to indicate that the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a second value, the priority decoding presentation field is used to indicate that the j-th track is not a track for priority decoding presentation;

[0243] The Substitute Track field is used to indicate whether the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a first value, the Substitute Track field is used to indicate that the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a second value, the Substitute Track field is used to indicate that the j-th track is not a substitute track;

[0244] The replacement track identification field is used to indicate the identification of a track replaced by the j-th track.

[0245] In one embodiment, if the j-th track is a non-reference track, the j-th track further includes a track index data box, and the track index data box includes a track index type data box;

[0246] The track index type data box includes a track identification field and an index type identification field;

[0247] The track identification field is used to store the identification of the reference track, and the index type identification field is used to indicate that the indexed track is the reference track.

[0248] In one embodiment, the processor 101 is configured to call the program instructions to execute:

[0249] According to the time domain track indication information and the decoding performance of the decoding device, retaining the time domain layer matching the decoding performance among the N time domain layers;

[0250] The media frames of the preserved temporal layer are decoded and displayed.

[0251] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit stream information includes a low bit rate field and a high bit rate field; the processor 101 is configured to call the program instruction to execute:

[0252] The retaining, according to the time domain track indication information and the decoding performance of the decoding device, a time domain layer matching the decoding performance among the N time domain layers comprises:

[0253] Reading a value of the multi-time domain track identification field in the time domain track indication information, and when the read multi-time domain track identification field is a second value, indicating that the media frames of the N time domain layers are encapsulated into a single track, and reading a value of the time domain layer identification field, a value of the frame rate field, and a value of a low bit rate field and a value of a high bit rate field in the bit rate information;

[0254] According to the value of the time domain layer identification field, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, the time domain layer among the N time domain layers that matches the decoding performance is retained.

[0255] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit stream information includes a low bit rate field and a high bit rate field; the processor 101 is configured to call the program instruction to execute:

[0256] Reading the value of the multi-time domain track identification field in the time domain track indication information, when the read multi-time domain track identification field is a first value, indicating that the media frames of the N time domain layers are encapsulated into multiple different tracks, and when the time domain layers in each track do not overlap, reading the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information from the reference track; the media frames encapsulated in the reference track are independently decoded;

[0257] According to the value of the time domain layer identification field read from the reference track, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, retain part or all of the time domain layers in the track that match the decoding performance.

[0258] In one embodiment, the time domain track indication information further includes track combination strategy information, the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; the processor 101 is configured to call the program instruction to execute:

[0259] When the read multi-time domain track identification field is a first value, indicating that the media frames of the N time domain levels are encapsulated into a plurality of different tracks, and there is overlap in the time domain levels in each track, the value of each field in the track combination strategy information is read from the reference track;

[0260] According to the values ​​of each field in the track combination strategy information and the decoding performance of the decoding device, some or all of the time domain levels in the track that match the decoding performance are retained.

[0261] In one embodiment, the processor 101 is configured to call the program instructions to execute:

[0262] According to the decoding time of each media frame in the retained one or more media frames, reorder the retained one or more media frames according to the decoding time;

[0263] The reordered one or more media frames are decoded and displayed.

[0264] In one embodiment, the memory 104 is used to store a computer program, the computer program includes program instructions, and the processor 101 is used to execute the program instructions stored in the memory 104, and can also be used to implement the above-mentioned Figure 7 The steps of the corresponding method in .

[0265] In one embodiment, the processor 101 is configured to call the program instructions to execute:

[0266] Determine the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtain media frames of N time domain levels; wherein N is a positive integer;

[0267] The media frames of the N time domain levels are respectively encapsulated into M tracks, and a description data box of the j-th track is generated according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation method of the N time domain levels, and the time domain track indication information includes time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M.

[0268] In one embodiment, the time domain track indication information includes a multi-time domain track identification field, and the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers; the processor 101 is configured to call the program instruction to execute:

[0269] If the multiple media frames of the N time domain levels are encapsulated into multiple different tracks, the value of the multiple time domain track identification field is generated as a first value;

[0270] If the multiple media frames of the N time domain levels are encapsulated into a single track, the value of the multi-time domain track identification field is generated as a second value.

[0271] In one embodiment, the time domain track indication information includes a total time domain layer number field; the processor 101 is configured to call the program instruction to execute:

[0272] The value of the total time domain layer number field is generated according to the total number of time domain layers encapsulated by the M tracks.

[0273] In one embodiment, the temporal layer information of the j-th track includes a temporal layer number field; the processor 101 is configured to call the program instruction to execute:

[0274] The value of the time domain layer number field is generated according to the number of time domain layers of the j-th track package.

[0275] In one embodiment, the temporal level information of the j-th track includes a temporal level identification field, a frame rate field and bit rate information; the processor 101 is configured to call the program instruction to execute:

[0276] The layer identifier of the i-th time domain layer among the N time domain layers is stored in the time domain layer identifier field, where i is an integer and i∈[0, N-1];

[0277] storing the frame rate accumulated when the media frame belongs to the i-th time domain level in the frame rate field;

[0278] The bit rate accumulated when the media frame belongs to the i-th time domain layer is obtained is used as the bit rate information.

[0279] In one embodiment, the bit rate information includes a low bit rate field and a high bit rate field; the processor 101 is configured to call the program instruction to execute:

[0280] The lower 18 bits of the bit rate of the media frame belonging to the i-th time domain level are stored in the low bit rate field;

[0281] The high 12 bits of the code rate of the media frame belonging to the i-th time domain layer are stored in the high code rate field.

[0282] In one embodiment, the time domain orbit indication information includes a reference orbit identification field; the processor 101 is configured to call the program instruction to execute:

[0283] If the j-th track is a reference track, the value of the reference track identification field is generated as a first value;

[0284] If the j-th track is a non-reference track, the value of the reference track identification field is generated as a second value;

[0285] The media frames encapsulated in the reference track are decoded independently.

[0286] In one embodiment, the time domain track indication information further includes track combination strategy information, the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; the processor 101 is configured to call the program instruction to execute:

[0287] storing an identification of a track comprising the partial temporal hierarchy in the track identification field;

[0288] If the j-th track is a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a first value; if the j-th track is not a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a second value;

[0289] If the j-th track is a replacement track for a track, the value of the replacement track field is generated as a first value, and the identifier of the track replaced by the j-th track is stored in the replacement track identifier field; if the j-th track is not a replacement track, the value of the replacement track field is generated as a second value.

[0290] In one embodiment, the processor 101 is configured to call the program instructions to execute:

[0291] If the j-th track is a non-reference track, a track index data box of the j-th track is generated, wherein the track index data box includes a track index type data box; the track index type data box includes a track identification field and an index type identification field;

[0292] The identifier of the reference track is stored in the track identifier field, and the j-th track is indexed to the reference track according to the index type identifier field.

[0293] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0294] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for processing media data, characterized in that: The media data includes a plurality of media frames, the plurality of media frames are divided into N time domain levels, and the plurality of media frames are encapsulated into M tracks, where M and N are both positive integers; the method includes: Acquire a description data box of a j-th track among the M tracks, the description data box including time domain track indication information, the time domain track indication information is used to indicate a track encapsulation mode of the N time domain levels, the time domain track indication information including time domain level information of the j-th track, wherein j is a positive integer and j≤M; the track encapsulation mode includes: a multi-track encapsulation mode and a single track encapsulation mode; The media data is decoded according to the time-domain track indication information.

2. The method according to claim 1, characterized in that The time domain track indication information includes a multi-time domain track identification field, where the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers; When the multi-time domain track identification field is a first value, the multi-time domain track identification field is used to indicate that the multiple media frames belonging to the N time domain levels are encapsulated into multiple different tracks; When the multi-time domain track identification field is a second value, the multi-time domain track identification field is used to indicate that multiple media frames belonging to the N time domain levels are encapsulated into a single track.

3. The method according to claim 1, characterized in that The time domain track indication information includes a total time domain layer number field; the total time domain layer number field is used to indicate the total number of time domain layers of the M track packages.

4. The method according to claim 1, characterized in that The time domain layer information of the j-th track includes a time domain layer number field, and the time domain layer number field is used to indicate the number of time domain layers encapsulated by the j-th track.

5. The method according to claim 1, characterized in that The time domain level information of the j-th track includes a time domain level identification field, a frame rate field and bit rate information; The time domain layer identifier field is used to indicate the layer identifier of the i-th time domain layer among the N time domain layers, where i is an integer and i∈[0, N-1]; The frame rate field is used to indicate the frame rate accumulated to the media frame belonging to the i-th time domain level; The bit rate information is used to indicate the bit rate accumulated to the media frame belonging to the i-th time domain layer.

6. The method according to claim 5, characterized in that The bit rate information includes a low bit rate field and a high bit rate field; The low bit rate field is used to indicate the lower 18 bits of the bit rate accumulated to the media frame belonging to the i-th time domain layer; The high code rate field is used to indicate the high 12 bits of the code rate accumulated to the media frame belonging to the i-th time domain layer.

7. The method according to claim 1, characterized in that The time domain track indication information includes a reference track identification field; the reference track identification field is used to indicate whether the j-th track is a reference track; When the reference track identification field is a first value, the reference track identification field is used to indicate that the j-th track is a reference track; when the reference track identification field is a second value, the reference track identification field is used to indicate that the j-th track is a non-reference track; The media frames encapsulated in the reference track are decoded independently.

8. The method according to claim 1, characterized in that The time domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; The track identification field is used to indicate the identification of the track including the partial time domain layer; The priority decoding presentation field is used to indicate whether the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a first value, the priority decoding presentation field is used to indicate that the j-th track is a track for priority decoding presentation; when the priority decoding presentation field is a second value, the priority decoding presentation field is used to indicate that the j-th track is not a track for priority decoding presentation; The Substitute Track field is used to indicate whether the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a first value, the Substitute Track field is used to indicate that the j-th track is a substitute track for one of the M tracks; when the Substitute Track field is a second value, the Substitute Track field is used to indicate that the j-th track is not a substitute track; The replacement track identification field is used to indicate the identification of a track replaced by the j-th track.

9. The method according to claim 1, characterized in that If the j-th track is a non-reference track, the j-th track further includes a track index data box, and the track index data box includes a track index type data box; The track index type data box includes a track identification field and an index type identification field; The track identification field is used to store the identification of the reference track, and the index type identification field is used to indicate that the indexed track is the reference track; The media frames encapsulated in the reference track are decoded independently.

10. The method according to any one of claims 1 to 9, characterized in that: The decoding of the media data according to the time domain track indication information includes: According to the time domain track indication information and the decoding performance of the decoding device, retaining the time domain layer matching the decoding performance among the N time domain layers; Decode the media frames of the preserved temporal layer.

11. The method according to claim 10, characterized in that The time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit rate information includes a low bit rate field and a high bit rate field; The retaining, according to the time domain track indication information and the decoding performance of the decoding device, a time domain layer matching the decoding performance among the N time domain layers comprises: Reading a value of the multi-time domain track identification field in the time domain track indication information, and when the read multi-time domain track identification field is a second value, indicating that the media frames of the N time domain layers are encapsulated into a single track, and reading a value of the time domain layer identification field, a value of the frame rate field, and a value of a low bit rate field and a value of a high bit rate field in the bit rate information; According to the value of the time domain layer identification field, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, the time domain layer among the N time domain layers that matches the decoding performance is retained.

12. The method according to claim 10, characterized in that The time domain track indication information includes a multi-time domain track identification field, a time domain layer identification field, a frame rate field and bit rate information, and the bit rate information includes a low bit rate field and a high bit rate field; The retaining, according to the time domain track indication information and the decoding performance of the decoding device, a time domain layer matching the decoding performance among the N time domain layers comprises: Reading the value of the multi-time domain track identification field in the time domain track indication information, when the read multi-time domain track identification field is a first value, indicating that the media frames of the N time domain layers are encapsulated into multiple different tracks, and when the time domain layers in each track do not overlap, reading the value of the time domain layer identification field, the value of the frame rate field, and the value of the low bit rate field and the value of the high bit rate field in the bit rate information from the reference track; the media frames encapsulated in the reference track are independently decoded; According to the value of the time domain layer identification field read from the reference track, the value of the frame rate field, the value of the low bit rate field and the value of the high bit rate field in the bit rate information, and the decoding performance of the decoding device, retain part or all of the time domain layers in the track that match the decoding performance.

13. The method according to claim 12, characterized in that The time domain track indication information also includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; the method also includes: When the read multi-time domain track identification field is a first value, indicating that the media frames of the N time domain levels are encapsulated into a plurality of different tracks, and there is overlap in the time domain levels in each track, the value of each field in the track combination strategy information is read from the reference track; According to the values ​​of each field in the track combination strategy information and the decoding performance of the decoding device, some or all of the time domain levels in the track that match the decoding performance are retained.

14. The method according to claim 10, characterized in that The number of the reserved media frames is one or more, and the decoding of the reserved media frames of the time domain layer includes: According to the decoding time of each media frame in the retained one or more media frames, reorder the retained one or more media frames according to the decoding time; Decode the reordered one or more media frames.

15. A method for processing media data, characterized in that: include: Determine the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtain media frames of N time domain levels; wherein N is a positive integer; The media frames of the N time domain levels are respectively encapsulated into M tracks, and a description data box of the j-th track is generated according to the encapsulation process of the media frames; the description data box includes time domain track indication information, and the time domain track indication information is used to indicate the track encapsulation method of the N time domain levels, and the time domain track indication information includes the time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M; the track encapsulation method includes: a multi-track encapsulation method and a single-track encapsulation method.

16. The method according to claim 15, characterized in that The time domain track indication information includes a multi-time domain track identification field, where the multi-time domain track identification field is used to indicate the track encapsulation method of the N time domain layers; The process of encapsulating the media frame to generate a description data box of the jth track includes: If the multiple media frames of the N time domain levels are encapsulated into multiple different tracks, the value of the multiple time domain track identification field is generated as a first value; If the multiple media frames of the N time domain levels are encapsulated into a single track, the value of the multi-time domain track identification field is generated as a second value.

17. The method according to claim 15, characterized in that The time domain track indication information includes a total time domain layer number field; The process of encapsulating the media frame to generate a description data box of the jth track includes: The value of the total time domain layer number field is generated according to the total number of time domain layers encapsulated by the M tracks.

18. The method according to claim 15, characterized in that The time domain level information of the j-th track includes a time domain layer number field; The process of encapsulating the media frame to generate a description data box of the jth track includes: The value of the time domain layer number field is generated according to the number of time domain layers of the j-th track package.

19. The method according to claim 15, characterized in that The time domain level information of the j-th track includes a time domain level identification field, a frame rate field and bit rate information; The process of encapsulating the media frame to generate a description data box of the jth track includes: The layer identifier of the i-th time domain layer among the N time domain layers is stored in the time domain layer identifier field, where i is an integer and i∈[0, N-1]; storing the frame rate accumulated when the media frame belongs to the i-th time domain level in the frame rate field; The bit rate accumulated when the media frame belongs to the i-th time domain layer is obtained is used as the bit rate information.

20. The method of claim 19, wherein: The bit rate information includes a low bit rate field and a high bit rate field; the bit rate accumulated to the media frame belonging to the i-th time domain layer, as the bit rate information, includes: The lower 18 bits of the bit rate of the media frame belonging to the i-th time domain level are stored in the low bit rate field; The high 12 bits of the code rate of the media frame belonging to the i-th time domain layer are stored in the high code rate field.

21. The method of claim 15, wherein: The time domain orbit indication information includes a reference orbit identification field; The process of encapsulating the media frame to generate a description data box of the jth track includes: If the j-th track is a reference track, the value of the reference track identification field is generated as a first value; If the j-th track is a non-reference track, the value of the reference track identification field is generated as a second value; The media frames encapsulated in the reference track are decoded independently.

22. The method of claim 15, wherein: The time domain track indication information further includes track combination strategy information, and the track combination strategy information includes a track identification field, a priority decoding presentation field, an alternative track field, and an alternative track identification field; The process of encapsulating the media frame to generate a description data box of the jth track includes: storing an identification of a track comprising the partial temporal hierarchy in the track identification field; If the j-th track is a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a first value; if the j-th track is not a track for priority decoding and presentation, the value of the priority decoding and presentation field is generated as a second value; If the j-th track is a replacement track for a track, the value of the replacement track field is generated as a first value, and the identifier of the track replaced by the j-th track is stored in the replacement track identifier field; if the j-th track is not a replacement track, the value of the replacement track field is generated as a second value.

23. The method of claim 15, wherein: The method further comprises: If the j-th track is a non-reference track, a track index data box of the j-th track is generated, wherein the track index data box includes a track index type data box; the track index type data box includes a track identification field and an index type identification field; The identifier of the reference track is stored in the track identifier field, and the j-th track is indexed to the reference track according to the index type identifier field; the media frames encapsulated in the reference track are independently decoded.

24. A media data processing device, characterized in that: The media data includes a plurality of media frames, the plurality of media frames are divided into N time domain levels, and the plurality of media frames are encapsulated into M tracks, where M and N are both positive integers; the apparatus includes: an acquiring unit, configured to acquire a description data box of a j-th track among the M tracks, the description data box comprising time domain track indication information, the time domain track indication information being used to indicate a track encapsulation mode of the N time domain levels, the time domain track indication information comprising time domain level information of the j-th track, wherein j is a positive integer and j≤M; the track encapsulation mode comprises: a multi-track encapsulation mode and a single track encapsulation mode; A processing unit is used to decode the media data according to the time domain track indication information.

25. A media data processing device, characterized in that: include: A determination unit, configured to determine the time domain level of each media frame according to the inter-frame dependency of each media frame included in the media data, and obtain media frames of N time domain levels; wherein N is a positive integer; A processing unit is used to encapsulate the media frames of the N time domain layers into M tracks respectively, and generate a description data box of the j-th track according to the encapsulation process of the media frames; the description data box includes time domain track indication information, the time domain track indication information is used to indicate the track encapsulation mode of the N time domain layers, the time domain track indication information includes the time domain layer information of the j-th track, wherein M and j are both positive integers, and j≤M; the track encapsulation mode includes: a multi-track encapsulation mode and a single track encapsulation mode.

26. A computer device, characterized in that: include: a processor adapted to implement one or more instructions; as well as, A memory storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the method for processing media data as described in any one of claims 1-23.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes a first program instruction and a second program instruction. When the first program instruction is executed by a processor, the processor executes the method according to any one of claims 1 to 14, and when the second program instruction is executed by the processor, the processor executes the method according to any one of claims 15 to 23.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a video stream generated by a computer program. When the computer program is executed by a computer, the computer executes the method according to any one of claims 1 to 14 to decode the video stream; or executes the method according to any one of claims 15 to 23 to encapsulate the video stream.

29. A method for storing or sending a video code stream, characterized in that: The video code stream is encapsulated according to the method described in any one of claims 15 to 23, or is decoded based on the method described in any one of claims 1 to 14.

Images