Media resource loading method and device

Abstract

The embodiment of the invention provides a media resource loading method and device, and the method comprises the steps: generating a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and packaging the first track file and the second track file into a target container to generate a media file, the rich media resource is a resource superposed on an interface corresponding to the multimedia resource; generating a resource list file according to the control logic of the first track file; and sending the media file and the resource list file to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource list file.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication, in particular to a media resource loading method and device. BACKGROUND

[0002] Traditional online video resource distribution and playing system mainly relies on the mode of processing and transmitting audio and video stream and additional resources separately, which brings many challenges. First, audio and video content is transmitted through mature standard streaming media protocols such as HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH), which are designed to efficiently transmit media streams so that the client can download, cache and decode the main video and audio tracks on demand to achieve a smooth playing experience.

[0003] However, for rich media resources such as bullet screen, subtitles, stickers, filters, interactive effects, etc., a more scattered and asynchronous management and loading method is adopted. Due to the diversity of special effect resources, from simple image overlay to complex real-time rendering, the synchronization of special effect resources with video playing becomes complex, and additional mechanisms are needed to ensure the coordinated work between them.

[0004] In view of the problem in the prior art that rich media resources are usually managed and loaded independently of the main stream, thereby causing the problem that they are usually not synchronized with multimedia resource playing, no effective solution has been proposed so far.

[0005] Therefore, it is necessary to improve the related art to overcome the defects in the related art. SUMMARY

[0006] The embodiments of the present application provide a media resource loading method and device to at least solve the problem in the prior art that rich media resources are usually managed and loaded independently of the main stream, thereby causing the problem that they are usually not synchronized with multimedia resource playing.

[0007] According to one embodiment of the present application, a media resource loading method is provided, applied to a service device, comprising: generating a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulating the first track file and the second track file into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; generating a resource manifest file according to a control logic of the first track file; and sending the media file and the resource manifest file to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file.

[0008] According to another embodiment of the present application, a media resource loading method is provided, applied to a client, comprising: receiving a media file and a resource manifest file sent by a service device, wherein a cloud platform generates a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulates the first track file and the second track file into a target container to generate a media file, generates a resource manifest file according to a control logic of the first track file, the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource, and the multimedia resource at least includes one of the following: a video resource, an audio resource; loading the multimedia resource according to the media file, and loading the rich media resource according to the media file and the resource manifest file.

[0009] According to another embodiment of the present application, a media resource loading device is provided, applied to a service device, comprising: a first generation module, configured to generate a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulate the first track file and the second track file into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; a second generation module, configured to generate a resource manifest file according to a control logic of the first track file; and a sending module, configured to send the media file and the resource manifest file to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file.

[0010] According to another embodiment of the present application, a media resource loading apparatus applied to a client is provided, comprising: a receiving module configured to receive a media file and a resource manifest file sent by a service device, wherein a cloud platform generates a first track file corresponding to a rich media resource and a second track file of a multimedia resource, encapsulates the first track file and the second track file into a target container to generate the media file, and generates the resource manifest file according to control logic of the first track file, the rich media resource being a resource superimposed on an interface corresponding to the multimedia resource, the multimedia resource comprising at least one of a video resource and an audio resource; and a loading module configured to load the multimedia resource according to the media file, and load the rich media resource according to the media file and the resource manifest file.

[0011] According to still another embodiment of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the steps in any of the method embodiments when running.

[0012] According to still another embodiment of the present application, an electronic device is further provided, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the method embodiments.

[0013] According to still another embodiment of the present application, a computer program product is further provided, comprising a computer program, and the computer program is executed by a processor to implement the steps in any of the method embodiments.

[0014] According to the present application, a first track file corresponding to a rich media resource and a second track file of a multimedia resource are generated, and the first track file and the second track file are encapsulated into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; a resource manifest file is generated according to control logic of the first track file; the media file and the resource manifest file are sent to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file. In the embodiments of the present application, multiple tracks are uniformly encapsulated in a container to form a single download / cache object, so that the number of requests and loading delay are reduced, and the delay of playing the rich media resource and the multimedia resource is reduced, and thus the problem that the rich media resource is usually managed and loaded independently of mainstream media, and thus is usually out of synchronization with the playing of the multimedia resource, can be solved. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a hardware structure block diagram of a service device for a media resource loading method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart (a) of a media resource loading method according to an embodiment of this application;

[0019] Figure 3 This is a flowchart (II) of a media resource loading method according to an embodiment of this application;

[0020] Figure 4 This is a structural block diagram (a) of a media resource loading device according to an embodiment of this application;

[0021] Figure 5 This is a structural block diagram (II) of a media resource loading device according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] The methods and embodiments provided in this application can be executed on a service device or a similar computing device. Taking running on a service device as an example, Figure 1 This is a hardware structure block diagram of a service device for a media resource loading method according to an embodiment of this application. For example... Figure 1 As shown, the service equipment may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MPU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The aforementioned service equipment may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The illustrated structure is only a schematic, which does not limit the structure of the service device. For example, the service device can further include more or less components than those shown, or have a different configuration of components than those shown. Figure 1 The illustrated structure is only a schematic, which does not limit the structure of the service device. For example, the service device can further include more or less components than those shown, or have a different configuration of components than those shown. Figure 1 The illustrated structure is only a schematic, which does not limit the structure of the service device. For example, the service device can further include more or less components than those shown, or have a different configuration of components than those shown.

[0025] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the media resource loading method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above method. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the service device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0026] The transmission device 106 is used to receive or send data via a network. The specific examples of the above network can include a wireless network provided by a communication provider of the service device. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0027] Figure 2 is a flowchart of a media resource loading method according to an embodiment of the present application, wherein the media resource includes rich media resources and multimedia resources, such as Figure 2 The flowchart includes the following steps:

[0028] In step S202, a first track file corresponding to the rich media resource and a second track file of the multimedia resource are generated, and the first track file and the second track file are encapsulated into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource.

[0029] It should be noted that the multimedia resource includes but is not limited to video and audio; the rich media resource includes but is not limited to: barrage, sticker, subtitle, and special effect.

[0030] Step S202 can ensure the separation and modularization of resources by abstracting and modeling multimedia resources and rich media resources into independent track files, facilitating subsequent unified packaging and flexible control. Then, the resources are combined into a media file through specific packaging techniques (such as using MP4, WebM, etc. container format). This unified packaging method enables different types of resources to be transmitted as a whole, reducing the number of client requests, simplifying resource management and synchronization issues, and improving transmission efficiency.

[0031] Step S204, generating a resource manifest file according to the control logic of the first track file;

[0032] The resource manifest file in step S204 (usually in JSON, XML or YAML format) is used to describe the meta information and control rules of all tracks packaged in the media file. It contains the ID, type, encoding format, priority, display conditions (such as device type, user identity, play scene), rendering order, and dependency relationship of the track, etc. The generation of the manifest file is based on the control logic of the rich media resource track file, providing detailed resource management guidelines for the client, so that the client can dynamically decide which resources to load, when to load, and how to render, thereby realizing intelligent scheduling and display of resources.

[0033] It should be noted that the resource manifest file can also include the control logic of the multimedia resources, which includes but is not limited to: one-key skipping, starting to display audio only when the video reaches a certain time point, shielding target objects, shielding target voiceprints.

[0034] Step S206, sending the media file and the resource manifest file to the client, so that the client loads the multimedia resources according to the media file, and loads the rich media resources according to the media file and the resource manifest file.

[0035] The media file and the resource manifest file are sent to the client, which can be efficiently distributed through a standard CDN (content distribution network), or the resources can be delivered through a self-developed distribution channel. The unified distribution mechanism ensures the consistency of the resources and control information received by the client.

[0036] At the client side, the client first parses the resource manifest file, and determines which tracks (multimedia resources and rich media resources) to load according to the description and control logic therein. Then, the client will unpackage and decode the corresponding resources from the media file according to the requirements of the tracks. Through a unified rendering interface, different types of track resources can be effectively superimposed on the video playback screen to realize the richness and diversity of visual content. In addition, the resource manifest file enables the client to flexibly adjust the loading and rendering strategy of resources according to device characteristics, user preferences and network conditions, thereby optimizing the playback experience.

[0037] Through the above steps, the first track file corresponding to the rich media resource and the second track file of the multimedia resource are generated, and the first track file and the second track file are encapsulated into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; a resource manifest file is generated according to the control logic of the first track file; the media file and the resource manifest file are sent to the client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file. The embodiments of the present application encapsulate multiple tracks into a container to form a single download / cache object, reduce the number of requests and loading delay, and thus can reduce the delay of rich media resource and multimedia resource playback. Therefore, the problem that rich media resources are usually managed and loaded independently of mainstream media, and thus are usually out of synchronization with multimedia resource playback, can be solved.

[0038] Optionally, the "generating the first track file corresponding to the rich media resource" in the above step S202 can be realized by the following way: determining attribute information of each sub-resource in the rich media resource; generating a track event of each sub-resource according to the attribute information of each sub-resource; and generating the first track file of the rich media resource according to the display time of each sub-resource and the track event of each sub-resource.

[0039] In the embodiments of the present application, detailed attribute information of each sub-resource (such as a single barrage, a single poster, a single subtitle, etc.) is extracted or defined, wherein the attribute information includes but is not limited to the type (such as image, text, interactive instruction, etc.), size, position, display time, duration, special effect parameter (such as scaling, rotation, transparency, etc.) of the resource. By determining the attribute information, a clear description framework is established for each sub-resource.

[0040] Based on the transformation of the attribute information of each sub-resource into one or a series of track events. Track events are data structures that describe the specific behavior of sub-resources on the time axis, containing key information such as display time, duration, position parameters, etc. For example, for a barrage resource, track events may include barrage text, display time, duration, speed, and display position. Event data will be used to build the track structure of the entire rich media resource, ensuring that the resource can be correctly presented when playing.

[0041] Finally, all the track events of the sub-resources are summarized into a structured first track file, which is a specific format (such as JSON, WebVTT, custom binary format, etc.) to be packaged into a multimedia container. The structure of the track file needs to include the display time, type identification, metadata, and other control information of each track event, so that the client can understand when to load, how to decode and render each sub-resource, and thus ensure the synchronization of rich media resources with the main video stream.

[0042] Optionally, processing the rich media resource into a standardized track file can be divided into the following steps:

[0043] Step 1: Prepare the input resource;

[0044] 1. Main video file: Ensure that there is a usable video file, such as in mp4 or webm format.

[0045] 2. Main audio file: Confirm the existence of the audio file, with an encoding format of AAC or Opus.

[0046] 3. Expression sticker image resource: Collect all expression sticker pictures that need to be processed, with formats such as png, webp, etc.

[0047] 4. Barrage text script: Prepare a JSON format file containing barrage content and its display timestamp.

[0048] Step 2: Use tool scripts for processing;

[0049] 1. Configure ffmpeg and Python environment: Ensure that the system is installed and configured with ffmpeg and a Python-supported environment.

[0050] 2. Write Python script: Create a Python script to read expression stickers and barrage scripts, then use ffmpeg to process these resources to meet the standard format of track files. The script will be responsible for matching expression stickers with their display time in the video and generating corresponding track descriptions.

[0051] Step 3: Create track description;

[0052] 1. Extracting Emoticon Map Information: The Python script extracts necessary information from the emoticon map resources, such as image file names, position data, display times, etc.

[0053] 2. Associating Timestamps: By reading the timestamps in the bullet screen script, the Python script associates these timestamps with the emoticon map information.

[0054] 3. Generating Track Description Data: The processed information is structured into a standardized format (e.g., JSON format) to build track descriptions, including:

[0055] track_id: Unique identifier of the track.

[0056] type: Resource type, such as image-overlay.

[0057] start_time: Time when the resource starts displaying.

[0058] duration: Duration of the resource display.

[0059] image_src: File name or path of the image resource.

[0060] position: Display position of the image on the video screen.

[0061] scale: Scaling ratio of the image.

[0062] For example:

[0063]

[0064] Step 4: Integrating Track Data;

[0065] 1. Serializing Track Events: All emoticon and bullet screen events are concatenated in chronological order to form a JSON sequence, which can describe multiple track events in one file.

[0066] 2. Output Track File: The generated track description sequence is written to a standard track file, such as overlay_track.json. This file contains the standardized description of all emoticons and bullet screens, so that the subsequent packaging process can recognize and process them.

[0067] Through the above steps, the originally scattered rich media resources, such as emoticons and bullet screens, are converted into structured, uniformly managed track file formats, achieving standardized management and efficient distribution of resources.

[0068] Optionally, the step of "packaging the first track file and the second track file into a target container to generate a media file" in the step S202 can be implemented by converting the first track file into a first data file in a target format, converting the second track file into a second data file in the target format, and writing the first data file and the second data file into target areas of the target container, respectively; and executing a target packaging command to package the target container with the first data file and the second data file written therein into the media file.

[0069] The first track file is a track description file of a rich media resource, such as a bullet screen, a subtitle, or a special effect track. The first track file is converted into a target format (such as a custom box structure conforming to the ISO Base Media File Format standard), so that the rich media resource can be correctly recognized and decoded by an existing media container (such as MP4 or WebM). The format conversion includes but is not limited to reconstructing a JSON, XML, or binary data first track file into a format of a specific area (such as moov, trak, or mdia) in a container file.

[0070] The second track file is a basic video and audio track file, which needs to be further processed to ensure compatibility with the track data of the rich media resource during the packaging process.

[0071] After the format conversion is completed, the first and second data files are written into designated areas or boxes of a target container (such as MP4), respectively. The designated areas depend on the provisions of the specific container format. For example, the first data file is written into a moof (Movie Fragment) box, and the second data file is written into a user extension box (udta or uuid box) of the MP4.

[0072] The packaging command is a media packaging tool (such as ffmpeg or Gpac MP4Box). The processed video, audio, and additional rich media resource track data are packaged into a single media file by the packaging command. This process includes but is not limited to writing all track data and metadata into corresponding blocks of the media container, ensuring the consistency of the time axis and the integrity of the decoding information, and finally generating a media file (such as unified_asset.mp4).

[0073] The format conversion in the embodiments of the present application ensures that the rich media resource can be supported by various devices and clients, avoiding resource loading failure or playback abnormality due to format incompatibility. By writing the first and second data files into different areas of the target container and then executing the packaging command, all resources can be efficiently integrated into a single media file, reducing the number of requests and the complexity of data processing of the client.

[0074] Optionally, the process of uniformly packaging the main video, main audio, and multiple track files into a parsable multi-track container includes the following steps:

[0075] Step 1: Prepare input data;

[0076] 1. Video / Audio Resources: Obtain video and audio files, such as videos in.mp4 or.webm format, and audio encoded in AAC or Opus.

[0077] Note that the main audio and main video are actually already in the form of tracks, and they are the initial and most basic components of the multi-track container.

[0078] 2. JSON format track files: Prepare JSON files that describe additional resources (such as overlays, bullet screens), such as overlay_track.json, ensuring that the files contain all necessary attribute information.

[0079] 3. Timeline information (time anchor points): Collect or generate time point information synchronized with the video content, used to locate when track data is displayed or applied.

[0080] Step 2: Track data conversion;

[0081] 1. Convert track to binary data segment: Use appropriate tools (such as ffmpeg combined with Python scripts) to convert JSON format track files to binary data to adapt to the storage format of multimedia containers.

[0082] Step 3: Multi-track container packaging;

[0083] 1. Select tool chain: Determine whether to use GPAC MP4Box or FFmpeg+bmffmux for packaging. GPAC MP4Box is suitable for quick and simple packaging, while FFmpeg+bmffmux provides more advanced customization capabilities and flexibility.

[0084] 2. Package main video and audio: Use the selected tool to package the main video and audio files as part of a multi-track container.

[0085] For example, when using ffmpeg, use the -map parameter to map video and audio streams to the output container.

[0086] Step 4: Write extended box;

[0087] Write user extension box: Write converted track data into the extension box of the MP4 container for storage, such as using udta or uuidbox.

[0088] Step 5: Map track types;

[0089] Define track types: Clearly define the type and purpose of each track in the container, for example:

[0090] Video: Track 1 (type: vide);

[0091] Audio: Track 2 (type: soun);

[0092] Overlay: Track 3 (type: meta / json);

[0093] Step 6: Execute the packaging command;

[0094] The packaging command is as follows:

[0095] {ffmpeg -i video.mp4 -i overlay_track.json,

[0096] -map 0:v -map 0:a -map 1,

[0097] -c copy -metadata:s:2 handler="overlay-json",

[0098] -movflags use_metadata_tags,

[0099] output_with_tracks.mp4}.

[0100] 1. Specify input files: Use the -i parameter to input the main video, main audio, and track file (e.g., overlay_track.json).

[0101] 2. Map streams: Use the -map parameter to specify which streams to extract from which inputs, for example -map 0:v -map 0:a -map 1 to map the video, audio, and third input (track file) to the output container.

[0102] 3. Copy encoding: Use -c copy to avoid re-encoding the video and audio, maintaining the original quality.

[0103] 4. Add metadata: Use -metadata:s:2 handler="overlay-json" to add metadata about the track processing method to the corresponding stream, where handler is used to identify the track type to the client, and s:2 indicates adding to the second stream (track).

[0104] 5. Use movflags: Use -movflags use_metadata_tags to ensure that metadata tags are used correctly.

[0105] 6. Output packaging file: After executing the command, a single packaging media file containing all tracks (video, audio, stickers, bullet screen, etc.) is generated, for example, named output_with_tracks.mp4.

[0106] Through the above steps, various types of resource data can be integrated into a unified multi-track container, facilitating parsing and efficient playback by the client, while ensuring synchronization and cross-platform consistency of resources.

[0107] Optionally, the embodiments of the present application provide a technical solution for dynamic updating of rich media resources (such as bullet screen, stickers, special effects, etc.), including: determining the update state of the rich media resource; in the case where the update state of the rich media resource indicates that there is an update of the rich media resource, generating a third track file according to the updated rich media resource; and packaging the third track file into a target container to generate an updated media file.

[0108] Monitoring and detecting whether the rich media resource in the multimedia content needs to be updated, the determination of the update state can be realized in various ways, including but not limited to version control, resource hash value checking, update timestamp comparison, etc. When it is detected that there is an update of the rich media resource, the new resource will be processed to generate or update the corresponding track file (i.e. the third track file). The generated third track file contains the attribute information of the updated sub-resource, track event and display time, etc., ensuring the latest and accuracy of the resource.

[0109] Finally, the updated third track file and other tracks (such as video, audio tracks) in the original media file are uniformly packaged to form an updated media file. The updated media file not only contains video and audio, but also contains the latest rich media resource track, realizing seamless upgrade of the overall resource.

[0110] Optionally, taking the new bullet screen track file as an example, the new track extension steps are as follows:

[0111] Step 1: New bullet screen track construction and format conversion;

[0112] 1. Time axis event modeling of new bullet screen style;

[0113] Using tools such as ffmpeg combined with Python scripts or dedicated media editing software, the newly designed bullet screen style is transformed into an event sequence on a timeline. Each event contains information such as the display time of the bullet screen, its text content, and its display style. The format can be WebVTT (a text format for subtitles and metadata) or a custom JSON-Event format.

[0114] JSON-Event is a flexible data exchange format that can describe various attributes of each bullet comment event. For example, here is a simple example of a JSON-Event:

[0115] {"time":32.5,"text":"This turning point is too abrupt!","style":"emphasized"}

[0116] {"time":45.0,"text":"Who knows...","style":"fade-in"}.

[0117] Each JSON object represents a bullet screen event, which includes the time of the event (in seconds), the text content of the bullet screen, and a specific display style (such as emphasis effect or fade-in effect).

[0118] Step 2: Track encapsulation and media file update;

[0119] 1. Package the track file into an existing MP4 container;

[0120] Use ffmpeg's encapsulation command to create a track file, then encapsulate it together with the existing media file (main video + audio track) to form an updated media file.

[0121] Here are some examples of ffmpeg wrapper commands:

[0122] {ffmpeg-i main.mp4-i new_danmaku_track.json,

[0123] -map 0-map 1,

[0124] -c copy-metadata:s:1handler_name="danmaku_beta",

[0125] -movflags use_metadata_tags,

[0126] output_with_danmaku.mp4}

[0127] -i main.mp4 and -i new_danmaku_track.json specify the main video file and the JSON description file of the new danmaku track as inputs, respectively.

[0128] The -map 0 and -map 1 parameters are used to tell ffmpeg which streams from the first and second inputs should be selected for output, here it means to keep all streams of main.mp4 and add new_danmaku_track.json as an extra track.

[0129] -c copy means to keep the video and audio encoding unchanged and directly copy the streams to the output file, avoiding the quality and performance loss caused by re-encoding.

[0130] -metadata:s:1 handler_name="danmaku_beta" is used to add metadata tags to the newly added danmaku track, indicating that it is a danmaku track and has a specific handler name for identification and control when the client parses.

[0131] -movflags use_metadata_tags ensures that ffmpeg uses metadata tags to correctly create the metadata part of the output file, which is particularly important for multi-track packaging, as it helps the client understand how to handle each track.

[0132] -output_with_danmaku.mp4 is the final packaged media file name, which contains the original video and audio information and the newly added danmaku track data.

[0133] Through the above steps, dynamic updating of rich media resources is achieved without the need for overall upgrade of the App, greatly improving the flexibility and efficiency of content updating, while ensuring the continuity and improvement of user experience.

[0134] Optionally, after encapsulating the third track file into the target container to generate an updated media file, the method further includes: generating an updated resource manifest file according to the meta information and control logic of the third track file; and sending the updated media file and the updated resource manifest file to a client, so that the client loads the multimedia resource according to the updated media file, and loads the updated rich media resource according to the updated media file and the updated resource manifest file.

[0135] The third track file contains newly added or updated rich media resource track data, including resource type, display time, duration, position, special effect parameters, priority, dependency relationship, and other meta information. The resource manifest file is responsible for describing the meta information and control logic of all tracks, so after updating the third track file, a resource manifest file with the latest resource state is generated, i.e., the updated resource manifest file.

[0136] The updated media file contains all multimedia resources and the newly added third track file, and the updated resource manifest file is a comprehensive description and control instruction of all available tracks. These two files need to be sent to the client together to ensure that the client can play according to the latest resource data and control logic.

[0137] After receiving the updated media file and resource manifest file, the client first parses the resource manifest file to understand which resource tracks are available and their control logic. Then, the client loads the corresponding track data according to the instructions in the manifest file. For the updated rich media resources, the client will load and render them according to the real-time calculated display time and control logic, ensuring precise synchronization with the video content.

[0138] The embodiments of the present application realize instant updating and dynamic control of resources by synchronously updating the third track file and the resource manifest file in the media container, providing a powerful and flexible content management capability for online video platforms, and significantly improving user experience and resource distribution efficiency.

[0139] Optionally, a new track is added to the resource control manifest by the following steps:

[0140] Step 1: Define the meta information and control information of the new track.

[0141] Determine the basic attributes of the new track and its control logic. Specifically, the following key fields are included:

[0142]

[0143] "dependencies":["track_main_video"]}.

[0144] Among them, id: "track_danmaku_beta", the unique identifier of the new track, is used to distinguish and reference different resource tracks.

[0145] type: "danmaku", identifies the type of resource carried by the track, which is a danmaku here.

[0146] codec: "json-event", describes the encoding format or data structure of the resource, where the bullet screen data is stored in the form of JSON-Event.

[0147] priority: 3, defines the priority of the track, the smaller the value, the higher the priority, which affects the loading order and display level of the resource.

[0148] conditions: set the conditions for enabling the track, such as only open to "beta" test users, and the system needs to enable the "enable_danmaku_v2" function flag.

[0149] render_layer: "overlay", determines the display layer of the resource in the playback picture, ensuring the correct superposition of the resource.

[0150] dependencies: ["track_main_video"], lists other tracks that must be loaded before the track is rendered, ensuring that the main video track is loaded and prepared before the bullet screen track.

[0151] Step 2: Integrate new track information into resource control list;

[0152] The list is usually a structured file, such as JSON format, used to describe the meta information and control logic of all tracks in the media container. The process of adding new track information in the list includes:

[0153] 1. Open the current resource control list file:

[0154] Read the current stored resource control list, which may be a JSON format file containing the description information of all known tracks.

[0155] 2. Add new track description to the list:

[0156] Add the newly defined track information to the appropriate position in the list (such as the tracks array), such as the complete description of "track_danmaku_beta" defined above. Ensure that the structure of the list is complete and the field format is correct, so that the client can successfully parse it.

[0157] 3. Save the updated resource control list:

[0158] Save the modified list file to ensure that the newly added track information is persisted and can be used for subsequent distribution and client reading.

[0159] Through the above steps, the resource control list is updated to include the newly added bullet screen track information, which provides a basis for further distribution and dynamic selection and loading of resources by the client. The client will be able to intelligently determine whether to load the "track_danmaku_beta" track and how to display it according to its control logic during playback based on the updated manifest file, thereby achieving the goal of adding playback functions or content without the need for App upgrades.

[0160] Optionally, generating the resource manifest file according to the control logic of the first track file includes: obtaining meta information of the first track file and writing the meta information into a first field in the resource manifest file; and determining the control logic according to a resource type of the rich media resource and writing the control logic into a second field in the resource manifest file.

[0161] The meta information includes but is not limited to track ID, resource type, codec type, priority, display time range, etc. This part of information is crucial for the client to identify and process specific resources. The meta information can be extracted from the first track file (such as the encapsulated bullet screen, special effect track file) using a special media file parsing library (such as FFmpeg, Gst-Parse, etc.) or a custom parsing script, and then parsed and written into the first field of the resource manifest file.

[0162] The first field usually refers to the part that directly describes the basic information of a track resource, such as each item in the tracks array in the manifest. The meta information of the first track file is written into the first field; different types of rich media resources (such as bullet screen, special effects, subtitles, etc.) may have different control logics, for example, bullet screen may need to consider user identity, playback speed adjustment, display density control, while special effects may involve user device capability and version compatibility determination. The process of determining the control logic determines the corresponding enabling rules, display priority, rendering order and dependency according to the characteristics of the resource type and the business strategy of the platform. The second field may include conditional expressions, priority sorting, dependency declaration, etc. By writing the control logic information into the second field, the client can obtain resource processing guidelines while receiving the media container file, thereby achieving intelligent resource scheduling and rendering control.

[0163] Optionally, the following is the main structure and field description of the manifest file, which can be encapsulated in JSON, YAML or binary metadata format:

[0164] {

[0165]

[0166]

[0167] id: Unique identifier of the track, used for internal scheduling and referencing, ensuring that clients can accurately identify and control each track.

[0168] type: Resource type, categorizing audio, video, subtitles, bullet screens, stickers, and special effects for differentiated processing and rendering.

[0169] codec: Decoding format or resource packaging sub-format, defining how to decode and render resources, ensuring compatibility with client decoding capabilities.

[0170] priority: Rendering priority, with smaller values indicating higher priority, used to control resource loading order and resolve resource conflicts.

[0171] conditions: Control rules, including device type, client version, user tags, and feature switches, to determine when to load and display specific tracks.

[0172] render_layer: Resource rendering layer on the playback interface, ensuring correct superposition, such as the main screen, subtitle layer, and special effect layer.

[0173] dependencies: Resource rendering dependencies, indicating which tracks need to be loaded first to achieve coordinated resource display.

[0174] The resource manifest file not only describes static resource information but also supports dynamic control of resource behavior. Control logic ranges include:

[0175] 1. Loading behavior control:

[0176] Intelligently select appropriate tracks based on client device type (mobile, TV, PC).

[0177] Automatically implement fallback strategies based on minimum client version requirements. Low-version clients may skip incompatible tracks to maintain basic playback functionality.

[0178] Enable experimental or specific function tracks through specific feature flags, enabling gray-scale publishing and A / B testing of functions.

[0179] 2. Rendering priority control:

[0180] Set the priority of sticker resources lower than subtitles and main video to avoid covering important information.

[0181] Adjust the priority of bullet screen tracks based on user subscription status, such as VIP users enjoying higher-definition or more timely bullet screen display.

[0182] 3. Time range and trigger mechanism:

[0183] Limit the playback time range of the track to avoid interfering with the viewing experience at unexpected time points.

[0184] Set conditional trigger mechanisms, such as automatically loading specific tracks at certain time points to achieve precise resource scheduling.

[0185] 4. Multiple track exclusion and combination strategy:

[0186] In some cases, define the exclusion relationship between tracks to prevent resource conflicts, such as advanced filters and certain interactive instructions cannot exist simultaneously.

[0187] Through dependency trees or combination scenarios, intelligently schedule track loading and rendering to achieve complex resource combination and hierarchical display.

[0188] Through the above integrated resource manifest file structure and control logic, the video streaming resource is uniformly packaged, dynamically scheduled, and consistently played across platforms, greatly improving resource management efficiency and user experience.

[0189] In this embodiment, a media resource loading method is also provided, which is applied to a client, Figure 3 is a flowchart of the media resource loading method according to the embodiments of the present application (two), as shown in Figure 3 The flowchart includes the following steps:

[0190] Step S302: receiving the media file and resource manifest file sent by the service device, wherein the cloud platform generates a first track file corresponding to the rich media resource and a second track file of the multimedia resource, encapsulates the first track file and the second track file into a target container to generate a media file, generates a resource manifest file according to the control logic of the first track file, and the rich media resource is a resource superimposed on the interface corresponding to the multimedia resource;

[0191] Step S304: loading the multimedia resource according to the media file, and loading the rich media resource according to the media file and the resource manifest file.

[0192] In one example embodiment, loading the multimedia resource according to the media file and loading the rich media resource according to the media file and the resource manifest file comprises: parsing the resource manifest file to determine the control logic of the first track file, and parsing the media file to obtain the first track file and the second track file; decoding the first track file and the second track file to obtain the multimedia resource and the rich media resource; loading the multimedia resource to the client through a target interface and loading the rich media resource to the client according to the control logic.

[0193] In one example embodiment, after loading the multimedia resource according to the media file and loading the rich media resource according to the media file and the resource manifest file, the method further comprises: in the case that an updated media file and an updated resource manifest file are received, parsing the updated media file to obtain the first track file, the second track file and a third track file, wherein the third track file is a track file corresponding to an updated rich media resource; determining whether the client satisfies a preset condition of rendering the third track file; in the case that the client satisfies the preset condition of rendering the third track file, loading the multimedia resource according to the updated media file and loading the updated rich media resource according to the updated media file and the updated resource manifest file; in the case that the client does not satisfy the preset condition of rendering the third track file, loading the multimedia resource according to the updated media file and loading the rich media resource according to the updated media file and the updated resource manifest file.

[0194] In the embodiments of the present application, the client reads the resource manifest when starting, parses and determines whether to load track_danmaku_beta. If the user satisfies the condition, the track is loaded and rendered; otherwise, the old danmaku track is loaded by default or skipped.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the embodiments of the present application.

[0196] In the embodiment, a media resource loading apparatus is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described herein. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0197] Figure 4 Fig. 1 is a structural block diagram of a media resource loading apparatus according to an embodiment of the present application, which is applied to a service device, such as a server, as shown in the figure, the apparatus comprises: Figure 4

[0198] A first generating module 42 is configured to generate a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulate the first track file and the second track file into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource.

[0199] A second generating module 44 is configured to generate a resource manifest file according to a control logic of the first track file.

[0200] A sending module 46 is configured to send the media file and the resource manifest file to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file.

[0201] Through the above apparatus, a first track file corresponding to a rich media resource and a second track file of a multimedia resource are generated, and the first track file and the second track file are encapsulated into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; a resource manifest file is generated according to a control logic of the first track file; the media file and the resource manifest file are sent to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file. In the present application, multiple tracks are uniformly encapsulated in a container to form a single download / cache object, which reduces the number of requests and loading delay, and thus, the problem that rich media resources are usually managed and loaded independently of mainstream media resources, and thus, are usually out of synchronization with the playing of the multimedia resources, can be solved.

[0202] ​In one example embodiment, the first generating module 42 is configured to determine attribute information of each sub-resource in the rich media resource, generate a track event of each sub-resource according to the attribute information of each sub-resource, and generate a first track file of the rich media resource according to display time of each sub-resource and the track event of each sub-resource.

[0203] In one example embodiment, the first generating module 42 is configured to convert the first track file into a first data file in a target format, convert the second track file into a second data file in the target format, and write the first data file and the second data file into target areas of a target container respectively, and execute a target packaging command to package the target container in which the first data file and the second data file are written into the media file.

[0204] In one example embodiment, the first generating module 42 is configured to determine an update state of the rich media resource, generate a third track file according to the updated rich media resource in a case where the update state of the rich media resource indicates that the rich media resource has an update, and package the third track file into a target container to generate an updated media file.

[0205] In one example embodiment, the second generating module 44 is configured to generate an updated resource manifest file according to meta information and control logic of the third track file, and send the updated media file and the updated resource manifest file to a client to enable the client to load the multimedia resource according to the updated media file and load the updated rich media resource according to the updated media file and the updated resource manifest file.

[0206] In one example embodiment, the second generating module 44 is configured to obtain meta information of the first track file and write the meta information into a first field in the resource manifest file, and determine the control logic according to a resource type of the rich media resource and write the control logic into a second field in the resource manifest file.

[0207] Figure 5 is a structural block diagram of a media resource loading device according to an embodiment of the present application, which is applied to a client, as shown in Figure 5 The device comprises:

[0208] The receiving module 52 is configured to receive the media file and the resource manifest file sent by the service device, wherein the cloud platform generates a first track file corresponding to the rich media resource and a second track file of the multimedia resource, encapsulates the first track file and the second track file into a target container to generate the media file, and generates the resource manifest file according to the control logic of the first track file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource, and the multimedia resource at least includes one of a video resource and an audio resource.

[0209] The loading module 54 is configured to load the multimedia resource according to the media file, and load the rich media resource according to the media file and the resource manifest file.

[0210] In an example embodiment, the loading module 54 is configured to, in a case where the updated media file and the updated resource manifest file are received, parse the updated media file to obtain the first track file, the second track file and a third track file, wherein the third track file is a track file corresponding to an updated rich media resource; determine whether the client satisfies a preset condition for rendering the third track file; in a case where the client satisfies the preset condition for rendering the third track file, load the multimedia resource according to the updated media file, and load the updated rich media resource according to the updated media file and the updated resource manifest file; and in a case where the client does not satisfy the preset condition for rendering the third track file, load the multimedia resource according to the updated media file, and load the rich media resource according to the updated media file and the updated resource manifest file.

[0211] In an example embodiment, the loading module 54 is configured to parse the resource manifest file to determine the control logic of the first track file, and parse the media file to obtain the first track file and the second track file; decode the first track file and the second track file to obtain the multimedia resource and the rich media resource; and load the multimedia resource to the client through a target interface and load the rich media resource to the client according to the control logic.

[0212] It should be noted that each of the above modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: all the above modules are located in the same processor; or the above modules are located in different processors in any combination.

[0213] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0214] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0215] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the method embodiments.

[0216] In an example embodiment, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0217] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to perform the steps in any of the method embodiments.

[0218] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps in any of the method embodiments.

[0219] The embodiment of the present application further provides a computer program, which comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the steps in any of the method embodiments.

[0220] The specific examples in the embodiment can refer to the examples described in the above embodiments and example embodiments, and the embodiment will not be described here again.

[0221] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with general computing devices, which can be centralized on a single computing device or distributed on a network of multiple computing devices, and which can be implemented with program codes executable by the computing devices, so that they can be stored in storage devices and executed by the computing devices, and in some cases, the steps shown or described can be executed in different orders than shown, or made into individual integrated circuit modules, or made into a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0222] The preferred embodiments of the present application described above are only used to explain the principles of the present application and not limit the present application. Any modification, equivalent replacement, improvement, etc. within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for loading a media resource, characterized in that, The application is applied to a service device, comprising: generating a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulating the first track file and the second track file into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; generating a resource list file according to the control logic of the first track file; sending the media file and the resource list file to a client to enable the client to load the multimedia resource according to the media file and load the rich media resource according to the media file and the resource list file.

2. The method of claim 1, wherein, The application is applied to a service device, comprising: determining attribute information of each sub-resource in the rich media resource; generating a track event of each sub-resource according to the attribute information of each sub-resource, and generating a first track file of the rich media resource according to the display time of each sub-resource and the track event of each sub-resource.

3. The method of claim 1, wherein, The application is applied to a service device, comprising: converting the first track file into a first data file in a target format and converting the second track file into a second data file in the target format, and writing the first data file and the second data file into target areas of the target container respectively; executing a target encapsulation command to encapsulate the target container in which the first data file and the second data file are written into the media file.

4. The method of claim 1, wherein, After the media file and the resource list file are sent to the client, the method further comprises: determining an update state of the rich media resource; generating a third track file according to the updated rich media resource in the case that the update state of the rich media resource indicates that the rich media resource exists update; encapsulating the third track file into a target container to generate an updated media file.

5. The method of claim 4, wherein, After the third track file is encapsulated into the target container to generate the updated media file, the method further comprises: generating an updated resource list file according to the meta information and the control logic of the third track file; sending the updated media file and the updated resource list file to the client to enable the client to load the multimedia resource according to the updated media file and load the updated rich media resource according to the updated media file and the updated resource list file.

6. The method of claim 1, wherein, The application is applied to a service device, comprising: obtaining meta information of the first track file and writing the meta information into a first field in the resource list file; and determining the control logic according to the resource type of the rich media resource, and writing the control logic into a second field in the resource list file.

7. A method for loading a media resource, characterized in that, The application is applied to a client, comprising: receive a media file and a resource manifest file sent by a service device, wherein a cloud platform generates a first track file corresponding to a rich media resource and a second track file of a multimedia resource, encapsulates the first track file and the second track file into a target container to generate the media file, and generates a resource manifest file according to control logic of the first track file, the rich media resource being a resource superimposed on an interface corresponding to the multimedia resource, the multimedia resource including at least one of a video resource and an audio resource; load the multimedia resource according to the media file, and load the rich media resource according to the media file and the resource manifest file.

8. The method of claim 7, wherein, After the multimedia resource is loaded according to the media file, and the rich media resource is loaded according to the media file and the resource manifest file, the method further includes: In a case where the updated media file and the updated resource manifest file are received, the updated media file is parsed to obtain the first track file, the second track file, and a third track file, wherein the third track file is a track file corresponding to an updated rich media resource; determine whether the client satisfies a preset condition for rendering the third track file; In a case where the client satisfies the preset condition for rendering the third track file, the multimedia resource is loaded according to the updated media file, and the updated rich media resource is loaded according to the updated media file and the updated resource manifest file; in a case where the client does not satisfy the preset condition for rendering the third track file, the multimedia resource is loaded according to the updated media file, and the rich media resource is loaded according to the updated media file and the updated resource manifest file.

9. A media resource loading apparatus, characterized by comprising: applied to a service device, and including: a first generation module configured to generate a first track file corresponding to a rich media resource and a second track file of a multimedia resource, and encapsulate the first track file and the second track file into a target container to generate a media file, wherein the rich media resource is a resource superimposed on an interface corresponding to the multimedia resource; a second generation module configured to generate a resource manifest file according to control logic of the first track file; a sending module configured to send the media file and the resource manifest file to a client, so that the client loads the multimedia resource according to the media file, and loads the rich media resource according to the media file and the resource manifest file.

10. A media resource loading apparatus, characterized by comprising: applied to a client, and including: a receiving module configured to receive a media file and a resource manifest file sent by a service device, wherein a cloud platform generates a first track file corresponding to a rich media resource and a second track file of a multimedia resource, encapsulates the first track file and the second track file into a target container to generate the media file, and generates a resource manifest file according to control logic of the first track file, the rich media resource being a resource superimposed on an interface corresponding to the multimedia resource, the multimedia resource including at least one of a video resource and an audio resource; a loading module, configured to load the multimedia resource according to the media file, and load the rich media resource according to the media file and the resource manifest file.

Images