Media resource rendering method and apparatus, computer device, and storage medium

By dividing media resources into essential and non-essential resources, prioritizing the processing of essential resources and rendering non-essential resources while downloading, the problems of long rendering loading time and poor user experience in the existing technology are solved, and faster Media resource loading and smoother playback experience.

WO2024240174A1PCT designated stage expired Publication Date: 2024-11-28BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/094659
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-05-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

When rendering audio and video data containing complex effects, the prior art needs to wait for all resource packages to be downloaded before playing, resulting in long loading time and poor user experience, especially when the network is unstable, the problem of resource download failure is prone to occur.

Method used

By dividing media resources into essential rendering resources and non-essential rendering resources, priority is given to the necessary resources to present the media interface, and download and render non-essential resources during playback, dynamically add playback effects, reduce loading time and Improve user experience.

Benefits of technology

It significantly reduces the loading time of rendering templates, avoids user waiting, and improves the preview experience of media effects, especially when network conditions are poor, which can still ensure smooth playback effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024094659_28112024_PF_FP_ABST
    Figure CN2024094659_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides a media resource rendering method and apparatus, a computer device, and a storage medium. The method comprises: acquiring a plurality of media rendering resources in a rendering template from a media editing interface; generating first media data on the basis of the plurality of media rendering resources, the first media data comprising a first rendering resource set, and the first rendering resource set comprising at least some of the media rendering resources; playing the first media data in the media editing interface; and in the process of playing the first media data, adding a playback effect of a second rendering resource set in real time, the second rendering resource set comprising media rendering resources among the plurality of media rendering resources other than the first rendering resource set.
Need to check novelty before this filing date? Find Prior Art

Description

Media resource rendering method, device, computer equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims priority to the Chinese patent application with application number 202310582866.7 and application date May 22, 2023. The disclosed content of the Chinese patent application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a method and apparatus for rendering media resources, a computer device, and a storage medium. Background Art

[0004] When rendering audio and video data containing complex effects, a corresponding rendering template is usually set. This rendering template includes special effects, filters, stickers, transitions, music, animations, etc. Each effect has a corresponding resource package, which is stored on a remote server and needs to be downloaded before use.

[0005] Summary of the Invention

[0006] Embodiments of the present disclosure provide a method, apparatus, computer equipment, and storage medium for rendering media resources.

[0007] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a method for rendering media resources, the method comprising: obtaining multiple media rendering resources in a rendering template in a media editing interface; generating first media data based on the multiple media rendering resources, the first media data comprising a first rendering resource set, the first rendering resource set comprising at least part of the media rendering resources; playing the first media data in the media editing interface; and in the process of playing the first media data, adding a playback effect of a second rendering resource set in real time, the second rendering resource set comprising other media rendering resources in the multiple media rendering resources except the first rendering resource set.

[0008] According to another aspect of the present disclosure, an embodiment of the present disclosure provides a rendering device for media resources, the device including: a resource acquisition module for acquiring multiple media rendering resources in a rendering template in a media editing interface; a generation module for generating first media data based on multiple media rendering resources, the first media data including a first rendering resource set, the first rendering resource set including at least part of the media rendering resources; a playback module for playing the first media data in the media editing interface; an adding module for adding the playback effect of a second rendering resource set in real time during the playback of the first media data, the second rendering resource set including other media rendering resources in the multiple media rendering resources except the first rendering resource set.

[0009] According to another aspect of the present disclosure, an embodiment of the present disclosure provides a computer device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the media resource rendering method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0010] According to another aspect of the present disclosure, an embodiment of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for rendering media resources of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] FIG1 is a flow chart of a method for rendering media resources according to some embodiments of the present disclosure;

[0013] FIG2 is a flow chart of another method for rendering media resources according to some embodiments of the present disclosure;

[0014] FIG3 is a flowchart of another method for rendering media resources according to some embodiments of the present disclosure;

[0015] FIG4 is a schematic diagram of experimental results of a method for rendering media resources according to some embodiments of the present disclosure;

[0016] FIG5 is a structural block diagram of a device for rendering media resources according to an embodiment of the present disclosure;

[0017] FIG6 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0019] When rendering audio and video data that includes complex effects such as special effects, filters, stickers, transitions, music, animations, etc., the rendering template needs to include resource packages such as special effects, filters, stickers, transitions, music, animations, etc., and these resource packages usually exist on remote servers and need to be downloaded for use. Therefore, when rendering a file with complex effects and the effect resource package exists on a remote server, a description of the remote resource package can be defined in the rendering template, such as an http link; all remote resources involved in the rendering template are traversed and downloaded in order of playback time. After the download is complete, the local file path is updated to the rendering template; media effects are replayed based on the rendering template to obtain audio and video effects.

[0020] Because rendering templates can involve a large number of resources, using the above solution, if you need to play a preview effect at a specific time, you must wait until all the resources corresponding to that time are downloaded before you can replay the media effect. This results in a long loading time, and the worse the network, the longer the loading time, which seriously affects the user experience. Furthermore, it is inevitable that resource downloads may fail due to poor network signal, making it impossible to restore the media effects of the rendering template.

[0021] In the present disclosure, however, it is not necessary to render all media effects at once when restoring the media effects corresponding to the rendering template. Some media effects can be restored progressively over the playback time, such as certain special effects, stickers, and filter effects. Thus, for example, if there are many sticker resources in the rendering template, the video can be rendered first, and then stickers can be downloaded while the video is playing. Each sticker is rendered after it is downloaded, giving the user the feeling that the effects in the template are being dynamically added during the playback process. Thus, the time to load the rendering template is greatly reduced without affecting the user's experience of previewing the media effects.

[0022] Based on this, this technical solution divides the rendering resources into essential and non-essential rendering resources, gives priority to the essential rendering resources to present the corresponding media interface to the user, avoiding long waiting times for the user. For non-essential rendering resources, they can be downloaded and rendered based on the needs of the playback time point, without affecting the display of media effects, thereby improving the user's preview experience of media effects.

[0023] According to an embodiment of the present disclosure, an embodiment of a method for rendering media resources is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] In this embodiment, a method for rendering media resources is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, and computers. FIG1 is a flow chart of the method for rendering media resources according to an embodiment of the present disclosure. As shown in FIG1 , the process includes the following steps:

[0025] Step S101: Acquire multiple media rendering resources in a rendering template in a media editing interface.

[0026] The media editing interface is used to configure the rendering template accordingly. The rendering template is used to define the description for the media rendering resource and to lay out the media rendering resources so that the media data can be loaded according to the rendering template. The media rendering resource is the resource package required to generate the media effect. Specifically, the media rendering resource may include a video resource package, an audio resource package, a sticker resource package, a filter resource package, a special effect resource package, an animation resource package, a transition resource package, etc. There is no limitation on the type of media rendering resources here, and those skilled in the art can make a choice according to actual needs.

[0027] The technician can pre-configure the rendering template in the media editing interface and configure the required media rendering resources in the rendering template. Correspondingly, the media editing interface can obtain multiple media rendering resources contained in the rendering template by parsing the configuration information of the rendering template.

[0028] Step S102: Generate first media data based on multiple media rendering resources.

[0029] The first media data includes a first rendering resource set, and the first rendering resource set includes at least part of the media rendering resources.

[0030] Part of the media rendering resources included in the first rendering resource set is input into the rendering layer. The rendering layer may render the part of the media rendering resources according to the layout associated with the rendering template to generate corresponding first media data.

[0031] Step S103: Play the first media data in the media editing interface.

[0032] The first media data is displayed in a media editing interface, so that a user can preview the first media data associated with the first rendering resource set.

[0033] Taking the example of a first rendering resource set including video resources, after identifying the video resources in the rendering template, the video resources are downloaded and directly sent to the rendering layer for video rendering, so that the video data corresponding to the video resources are displayed in the media editing interface, and the video data is played in the media editing interface, so that the user can view the video playback screen. It is understandable that the video resources included in the first rendering resource set can be distributed at any time node or time period of the video data, that is, the determination of the first rendering resource set can be independent of the time position and time sequence of its playback.

[0034] Step S104: During playback of the first media data, a playback effect of a second rendering resource set is added in real time, wherein the second rendering resource set includes other media rendering resources in the plurality of media rendering resources except the first rendering resource set.

[0035] The second rendering resource set is a set of media rendering resources required for playback of the first media data. Specifically, the second rendering resource set includes other media rendering resources from the plurality of media rendering resources, excluding the first rendering resource set. The other media rendering resources included in the second rendering resource set can be determined based on the playback time of the first media data.

[0036] Specifically, during playback of the first media data, the media rendering resources contained in the second rendering resource set are fed into the rendering layer for rendering processing, resulting in the playback effects associated with the second rendering resource set. This allows the playback effects corresponding to the second rendering resource set to be added in real time while the first media data is being played.

[0037] For example, if the first rendering resource set includes video resources and the second rendering resource set includes sticker resources, when the video resource rendering is completed and the video playback screen is displayed, the sticker resources in the rendering template can be downloaded and rendered in the rendering layer to obtain sticker data. The sticker data is then merged with the video data to dynamically add sticker effects to the video playback screen.

[0038] The rendering method of media resources provided in this embodiment generates and plays some of the first media data from the multiple media rendering resources obtained in the media editing interface, and processes other media rendering resources based on the playback of the first media data, so as to add the corresponding playback effects of other media rendering resources in real time during the playback of the first media data. As a result, the media resources in the rendering template do not need to wait for all downloads to be completed before rendering, and can be downloaded and rendered at any time, thus reducing the loading time of the media resources. Moreover, during the rendering process, some media rendering resources can be processed first to present the corresponding media data to the user, reducing the user's waiting time, and other media rendering resources can be rendered and played at the same time, so that the corresponding playback effects can be dynamically added based on the playback of the first media data, reducing the impact on the display of media effects and improving the user's preview experience of media effects.

[0039] In this embodiment, a method for rendering media resources is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, and computers. FIG2 is a flow chart of the method for rendering media resources according to an embodiment of the present disclosure. As shown in FIG2 , the process includes the following steps:

[0040] Step S201: Acquire multiple media rendering resources in a rendering template in a media editing interface. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0041] Step S202: Generate first media data based on multiple media rendering resources.

[0042] The first media data includes a first rendering resource set, and the first rendering resource set includes at least part of the media rendering resources.

[0043] Specifically, the first rendering resource set includes essential rendering resources among the multiple media rendering resources. Accordingly, the above step S202 may include:

[0044] Step S2021: Identify at least one essential rendering resource from a plurality of media rendering resources.

[0045] Required rendering resources are the starting playback resources required for rendering a media effect. These resources are the resources that the media effect prioritizes for playback. For example, if a video is required to start playback, the required rendering resource is a video resource; if an audio resource is required to start playback, the required rendering resource is an audio resource.

[0046] The multiple media rendering resources include essential rendering resources and non-essential rendering resources. The essential rendering resources are identified from the multiple media rendering resources according to a preset rule. The preset rule is a pre-set rule for identifying the essential rendering resources.

[0047] In some implementations, step S2021 may include:

[0048] Step a1: Identify whether each media rendering resource among the multiple media rendering resources carries a necessary identifier.

[0049] The mandatory identifier is the identifier information carried by the mandatory rendering resource. When the media rendering resources in the rendering template are obtained, the identifier information carried by each media rendering resource is detected to determine whether the identifier information carried by the media rendering resource is a mandatory identifier.

[0050] Step a2: determining one or more media rendering resources carrying a mandatory identifier among the multiple media rendering resources as mandatory rendering resources.

[0051] In response to determining that a media rendering resource carrying a mandatory flag exists in the rendering template, the media rendering resource is determined to be a mandatory rendering resource, such as all video clips of the master track. In response to determining that multiple media rendering resources carrying mandatory flags exist in the rendering template, the multiple media rendering resources are determined to be mandatory rendering resources, such as only the first video clip of the master track, all clips of the master track, the audio track, the sticker track, etc. At the same time, resources other than the mandatory rendering resources are determined to be non-mandatory rendering resources.

[0052] Therefore, by identifying the necessary identifiers carried by the media rendering resources, the necessary rendering resources can be quickly identified and loaded first, thereby ensuring the loading speed of the necessary rendering resources.

[0053] Step S2022: Process the at least one required rendering resource to generate first media data.

[0054] The first media data includes the at least one necessary rendering resource.

[0055] The required rendering resources are input into the rendering layer, which then renders them to generate first media data containing the required rendering resources. For example, if the required rendering resource is a video resource, the rendering layer renders the required rendering resource to generate a video image corresponding to the required rendering resource. The user can then preview the first media data associated with the required rendering resource through the video image. It is understood that the required rendering resources can be distributed at any time point or time period of the video data, meaning that the determination of the required rendering resources can be independent of their playback time position or time sequence.

[0056] Step S203: Play the first media data in the media editing interface. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0057] Step S204: During playback of the first media data, playback effects of a second rendering resource set are added in real time. The second rendering resource set includes the other media rendering resources in the plurality of media rendering resources, excluding the first rendering resource set. For detailed descriptions, please refer to the corresponding descriptions of the above embodiments and will not be repeated here.

[0058] In some embodiments, the above step 104 or step S204 may include: generating second media data based on the first media data and the second rendering resource set.

[0059] In some embodiments, the second media data includes the first media data and at least part of the media rendering resources in the second rendering resource set added in real time.

[0060] After playing the first media data, the media rendering resources in the second rendering resource set are input into the rendering layer for rendering processing, thereby fusing the first media data with the media data generated by rendering the second rendering resource set to generate the second media data. Thus, when displaying the second media data, the user can preview the first media data and the playback effects of the other media rendering resources added to the first media data in real time.

[0061] In some embodiments, the second rendering resource set includes non-essential rendering resources among the plurality of media rendering resources. Accordingly, the step of generating the second media data based on the first media data and the second rendering resource set may include:

[0062] Step B: within a preset time period after processing the essential rendering resources, process one or more non-essential rendering resources to obtain a processing result, and based on the processing result and the first media data, generate and play second media data including the non-essential rendering resources.

[0063] The preset duration is the pre-set interval between rendering the required rendering resources. Specifically, after the required rendering resources are rendered, the non-essential rendering resources of the second rendering resource set are rendered. Specifically, the non-essential rendering resources are downloaded and sent to the rendering layer for rendering, resulting in media data including the non-essential rendering resources. Subsequently, the media data corresponding to the second rendering resource set is merged with the first media data, and the second media data is played in addition to the first media data.

[0064] In this implementation, by dividing media resources into essential rendering resources and non-essential rendering resources, the essential rendering resources are prioritized during rendering to generate corresponding first media data for playback in the media editing interface. The non-essential rendering resources are then processed to generate corresponding second media data, and the second media data is played based on the first media data. Thus, when rendering media resources, essential rendering resources can be prioritized to present the corresponding media editing interface to the user, reducing user waiting time. Non-essential rendering resources can be rendered and played simultaneously, allowing the media data corresponding to the second rendering resource set to be dynamically added in real time based on the playback of the first media data. This reduces the impact on the presentation of media effects and enhances the user's preview experience of media effects.

[0065] In some embodiments, the above step B may include:

[0066] Step b1: When the first media data is played for a preset time period, a playback position of the first media data is determined.

[0067] The preset duration is the pre-set playback duration of the first media data, and the playback position is the position reached after the first pair of media data has played for the preset duration. Specifically, the first media data is played according to a corresponding playback track, which sets the playback time of the media data according to the time dimension, i.e., there is a one-to-one correspondence between the playback time and the media data. Based on the preset duration of the first media data, the position of the first media data on the playback track can be determined; this position is the playback position of the first media data.

[0068] Taking the first media data as video data as an example, the video data is played according to the video track, and each frame of video corresponds to a start time and an end time. When the video data is played for a preset time, the current playback position of the video data can be determined by combining the preset time, the start time of each frame of video, and the end time of each frame of video.

[0069] Step b2: Based on the playback position, determine the target resource required for the next playback position from the non-essential rendering resources.

[0070] While the first media data is playing, each non-essential rendering resource can be downloaded and loaded into the current first media data according to pre-set start and end times. To ensure loading continuity and create a dynamic addition effect, during the playback of the first media data, it can be detected in real time whether the next playback position requires loading non-essential rendering resources.

[0071] The non-essential rendering resources required to load the next playback position are downloaded as target resources. Even in poor network conditions, this method can download non-essential rendering resources according to playback requirements without waiting for all non-essential rendering resources to be downloaded. This can reduce the waiting time for resource loading and ensure the media playback effect.

[0072] In some optional implementations, the above step b2 may include:

[0073] Step b21: Obtain the loading time corresponding to each non-essential rendering resource, and determine the target playback position of each non-essential rendering resource based on the first media data based on the loading time.

[0074] Step b22 : comparing the target playback position with the next playback position, and selecting target resources required for the next playback position from non-essential rendering resources based on the comparison result.

[0075] The loading time represents the display time of each non-essential rendering resource. The loading time is pre-configured according to actual needs. There is no specific limitation on the loading time here. Those skilled in the art can set the loading time of each non-essential rendering resource on the first media data based on debugging results or experience.

[0076] The playback position of each playback moment of the first media data on the playback track is determined based on the playback time of the first media data. The target playback position of the loading time on the playback track can be determined based on the loading time of the non-essential rendering resource.

[0077] If the next playback position of the first media data is the same as the target playback position of the non-essential rendering resource, it means that the next playback position needs to load the corresponding non-essential rendering resource, and the non-essential rendering resource is determined as the target resource required for the next playback position.

[0078] In this embodiment, by determining the loading time corresponding to the non-essential rendering resources, its target playback position based on the first media data can be determined. Combined with the target playback position and the next playback position of the first media data, the required target resources can be determined, thereby realizing dynamic scheduling of non-essential rendering resources, making it convenient to dynamically obtain corresponding target resources according to the actual playback status.

[0079] Step b3: Process the target resource according to the preset priority to generate second media data corresponding to the target resource.

[0080] The preset priority is a pre-set download order for each target resource. The second media data is generated based on the rendering of the target resource. Specifically, each target resource is downloaded according to the preset priority. Each target resource is rendered after it is downloaded. This means that a simultaneous download and rendering operation is performed for each target resource, thereby sequentially obtaining the second media data associated with each target resource.

[0081] In some optional implementations, the above step b3 may include:

[0082] Step b31: construct multiple download tasks corresponding to the target resources.

[0083] Step b32: Prioritize the multiple download tasks based on the preset priority to obtain a priority ranking result.

[0084] Step b33: trigger the multiple download tasks in sequence according to the priority sorting results to generate second media data.

[0085] Construct multiple download tasks corresponding to the target resources, add each download task to the task queue, and prioritize each download task in the task queue according to the preset priority to obtain the priority sorting result. During downloading, the download task is triggered in combination with the priority sorting result. Specifically, different types of target resources have different corresponding priorities. The target resources of each type are arranged in descending order of priority, that is, the higher the priority of the target resource, the higher the priority of the corresponding download task.

[0086] For example, the priority for video resources is 100, the priority for audio resources is 99, the priority for filter resources is 98, the priority for special effects resources is 97, and the priority for sticker resources is 96. Different types of target resources correspond to one or more download tasks. The download tasks are sorted by the priority of each type of target resource to obtain the priority sorting results for each download task. In this example, the video resource has the highest priority, so the download task corresponding to the video resource is triggered first.

[0087] When a download task is completed, it means that the resource package corresponding to the download task has been downloaded locally. At this time, the downloaded resource package is rendered to obtain the media data associated with the resource package.

[0088] In this embodiment, by triggering each download task in sequence according to the priority ranking result, it is ensured that the required target resources are loaded first, thereby improving the rendering effect of the media resources.

[0089] In some optional implementations, the above step b33 may include:

[0090] Step c1: detecting whether the number of the plurality of download tasks exceeds a preset concurrent quantity.

[0091] Step c2: in response to the number of the plurality of download tasks exceeding the preset concurrency amount, determining one or more current download tasks according to the preset concurrency amount.

[0092] Step c3: trigger the one or more current download tasks in sequence according to the priority sorting results to generate second media data.

[0093] The preset concurrency is the maximum number of concurrent tasks pre-configured in the business layer, such as 3, 4, 5, etc. There is no limit on the preset concurrency here, and those skilled in the art can determine it according to actual needs.

[0094] Count the number of download tasks corresponding to the target resource and compare the number of download tasks with the preset concurrency limit to determine whether the number of download tasks corresponding to the target resource exceeds the preset concurrency limit. If the number of download tasks exceeds the preset concurrency limit, select several download tasks with higher priorities from the multiple download tasks based on the preset concurrency limit and the priority sorting result, execute these download tasks, trigger the corresponding resource download process, and render the downloaded resources to obtain the corresponding media data.

[0095] Specifically, each time a download task is completed, the highest priority download task is taken out from the task queue to meet the maximum task concurrency and continue to download resources, thereby enabling progressive rendering of the target resource.

[0096] It should be noted that following the playback process of the first media data, the non-essential rendering resources required for the next playback position are detected in real time according to the playback time and playback position of the first media data, so as to give priority to downloading the non-essential rendering resources involved in the content to be played, and re-prioritize the download tasks corresponding to the non-essential rendering resources, and trigger each download task in sequence according to the priority sorting results after re-sorting.

[0097] In this embodiment, when the number of download tasks corresponding to the target resource exceeds the preset concurrency, multiple download tasks corresponding to the target resource are scheduled according to the preset priority and the preset concurrency. Each time a download task is completed, a download task with the highest priority is scheduled from the remaining download tasks. This enables progressive rendering of the target resource, greatly reducing the loading time of the resource without affecting the user's preview of the media effect.

[0098] Step b4: Play the second media data based on the first media data.

[0099] On the basis of playing the first media data through the media editing interface, the second media data obtained by rendering the target resource is loaded into the current media editing interface. In this way, on the basis of playing the first media data, the second media data corresponding to each target resource can be dynamically added to the first media data in turn for playback, thereby realizing the dynamic loading and playback of the second media data.

[0100] Taking the first media data as video data and the second media data as audio data and sticker data as an example, the playback time of audio resources and sticker resources is pre-set. For example, the audio data and sticker data are added 2s after the video data is played. When it is detected that the video data is played to 1.9s, the audio resources and sticker resources required for the next playback position (i.e. the position corresponding to 2s) are determined from the non-essential rendering resources, and the priority of the audio resources is greater than the priority of the sticker resources. Here, the audio resources and sticker resources are used as target resources and downloaded in the order of audio resources first and sticker resources second. After the download of the audio resources is completed, the audio data is rendered and loaded into the video data for audio playback. At the same time, the downloaded sticker resources are dynamically rendered, and the sticker data can be dynamically added to the current video data, and the feeling presented to the user is the effect of dynamic addition of stickers.

[0101] Of course, if the audio data is added after 2s of video data playback, and the sticker data is added after 4s of video data playback, then when it is detected that the video data is played to 1.9s, the audio resources required for the next playback position (i.e., the position corresponding to 2s) are determined from the non-essential rendering resources for downloading, and the downloaded audio resources are rendered so that the audio data can be played while the video data is playing. Similarly, when it is detected that the video data is played to 3.9s, the sticker resources required for the next playback position (i.e., the position corresponding to 4s) are determined from the non-essential rendering resources for downloading, and the downloaded sticker resources are rendered so that the sticker data can be dynamically displayed on the video screen while the video data and audio data are playing.

[0102] It is understandable that if there are factors such as network delays, the playback time point after the initial dynamic loading of the media rendering resource may be later than the actual playback node set by the rendering template. For example, in the above example, the sticker display time period is 4s to 8s. When it is detected that the video data is played to 3.9s, the sticker resources required for the next playback position (i.e. the position corresponding to 4s) should be determined from the non-essential rendering resources for download and playback. However, due to network reasons, the sticker resource download is not completed at 4s. After the sticker resource download is completed at 4.2s, the sticker effect can be played again until the sticker display is canceled at 8s.

[0103] The media resource rendering method provided in this embodiment detects the current playback position to determine the target resource required to be loaded at the next playback position. This allows the required target resource to be dynamically determined from non-essential rendering resources, prioritizing the loading of the media resource to be played, eliminating the need to sequentially download all non-essential rendering resources and saving loading time. Furthermore, the determined target resources are processed sequentially according to a preset priority, enabling progressive rendering of non-essential rendering resources. This allows the target resource to be dynamically added to the first media data, achieving a dynamic loading effect and enhancing the user's preview experience of the media effect.

[0104] In this embodiment, a method for rendering media resources is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, and computers. FIG3 is a flow chart of the method for rendering media resources according to an embodiment of the present disclosure. As shown in FIG3 , the process includes the following steps:

[0105] Step S301: Acquire multiple media rendering resources in a rendering template in a media editing interface.

[0106] Specifically, the above step S301 includes:

[0107] Step S3011: Obtain resource description information in the rendering template in the media editing interface.

[0108] Resource description information is a description of the download path for the remote resource package, pre-defined by the technician in the rendering template protocol. All media rendering resources included in the rendering template have corresponding resource description information. Specifically, upon obtaining a rendering template, the resource description information corresponding to all media rendering resources can be obtained by parsing the protocol information corresponding to the rendering template.

[0109] Step S3012: Acquire corresponding media remote resources based on the resource description information.

[0110] Remote media resources are media rendering resources stored in remote servers. Based on the resource description information, the download path of the remote media resources can be determined. Based on the download path, the corresponding remote server can be located, and all remote media resources can be obtained from the remote server pointed to by the download address.

[0111] Step S3013: Convert the media remote resource according to a preset conversion protocol to obtain a media rendering resource.

[0112] The default conversion protocol is the data conversion protocol between the remote server and the rendering template. Because remote media resources may come from different servers, and different remote servers have different data definitions, to ensure data consistency, various remote media resources can be converted according to the default conversion protocol to obtain media rendering resources that the rendering template can recognize. These media rendering resources are then populated into the rendering template for subsequent rendering.

[0113] Step S302: Generate first media data based on multiple media rendering resources. The first media data includes a first rendering resource set, which includes at least some of the media rendering resources. Detailed descriptions are provided in the corresponding descriptions of the above embodiments, which will not be repeated here.

[0114] Step S303: Play the first media data in the media editing interface. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.

[0115] Step S304: During playback of the first media data, playback effects of a second rendering resource set are added in real time. The second rendering resource set includes the other media rendering resources in the plurality of media rendering resources, excluding the first rendering resource set. For detailed descriptions, please refer to the corresponding descriptions of the above embodiments and will not be repeated here.

[0116] The rendering method of media rendering resources provided in this embodiment performs protocol conversion on the acquired media remote resources to generate media rendering resources supported by the current rendering template, thereby improving compatibility with various remote resources.

[0117] As a specific application embodiment of the present disclosure, when replaying a media effect according to a rendering template, the implementation method is as follows:

[0118] 1) Download the necessary resources in the prepared rendering template (i.e., NLEModel) in advance. These necessary resources are related to audio and video rendering, such as the first main track video clip to be played. Here, preset rules are combined to identify the necessary resources in the rendering template.

[0119] 2) After downloading the necessary resources, the NLEModel is directly handed over to the rendering layer (i.e., the NLEMedia layer) for rendering of the necessary resources. The NLEMedia layer can render the main track video based on the existing necessary resources, so that the user can see the video screen.

[0120] 3) There are many non-essential rendering resources in the NLEModel template. If you download them all at once, it may cause slow resource loading. At this time, you need to download each non-essential rendering resource in sequence according to the pre-priority, such as other video resources, audio resources, sticker / filter resources, various style resources, etc., and the preset priority can support business configuration.

[0121] Specifically, when playing a video, non-essential rendering resources are prioritized based on the current playback position of the video, with priority given to downloading the non-essential rendering resources related to the content that is about to be played. Based on this, resources are selected based on the priority of various types of non-essential rendering resources. For example, if video resources have the highest priority, they will be downloaded first.

[0122] 4) The downloaded non-essential rendering resources are rendered through the NLEMedia layer to add the effects corresponding to the non-essential rendering resources to the audio and video rendering engine for playback.

[0123] We conducted several experiments using this solution, yielding the results shown in Figure 4. Analysis of these results revealed that the average rendering time per template was reduced by 455ms, the effects download process was optimized by 20.2%, and the overall rendering template application process was optimized by 7.1%. This demonstrates that this solution improves the loading speed of rendering templates and enhances the user experience.

[0124] In this embodiment, a rendering device for media resources is also provided, which is used to implement the above-mentioned embodiments and preferred implementation methods. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0125] This embodiment provides a rendering device for media rendering resources, as shown in FIG5 , including:

[0126] The resource acquisition module 401 is used to acquire multiple media rendering resources in the rendering template in the media editing interface.

[0127] The generating module 402 is configured to generate first media data based on a plurality of media rendering resources, wherein the first media data includes a first rendering resource set, and the first rendering resource set includes at least some of the media rendering resources.

[0128] The playing module 403 is configured to play the first media data in the media editing interface.

[0129] The adding module 404 is configured to add the playback effect of the second rendering resource set in real time during the playback of the first media data, wherein the second rendering resource set includes other media rendering resources in the plurality of media rendering resources except the first rendering resource set.

[0130] In some optional implementations, the generating module 402 may include:

[0131] The identification unit is configured to identify at least one essential rendering resource from a plurality of media rendering resources.

[0132] The resource processing unit is configured to process the necessary rendering resources to generate first media data, wherein the first media data includes the necessary rendering resources.

[0133] In some optional implementations, the identification unit may include:

[0134] The identification unit is used to identify whether each media rendering resource carries a necessary identification.

[0135] The mandatory determining unit is configured to determine one or more media rendering resources carrying mandatory identifiers as mandatory rendering resources.

[0136] In some optional implementations, the adding module 404 may include:

[0137] The data generation unit is configured to generate second media data based on the first media data and the second rendering resource set, wherein the second media data includes the first media data and at least part of the media rendering resources in the second rendering resource set added in real time.

[0138] In some implementations, the second rendering resource set includes non-essential rendering resources among the plurality of media rendering resources, and the data generating unit may include:

[0139] The rendering subunit is used to process one or more non-essential rendering resources within a preset time period after processing the essential rendering resources to obtain a processing result, and based on the processing result, generate and play second media data including the non-essential rendering resources on the basis of the first media data.

[0140] In some embodiments, the rendering sub-unit is further used to: determine the playback position of the first media data when the first media data is played for a preset duration; based on the playback position, determine the target resources required for the next playback position from non-essential rendering resources; process the target resources according to a preset priority to generate second media data corresponding to the target resources; and play the second media data based on the first media data.

[0141] In some optional embodiments, the above-mentioned rendering sub-unit is further specifically used to: obtain the loading time corresponding to each non-essential rendering resource, and determine the target playback position of each non-essential rendering resource based on the first media data based on the loading time; compare the target playback position with the next playback position, and filter out the target resources required for the next playback position from the non-essential rendering resources according to the comparison results.

[0142] In some optional embodiments, the above-mentioned rendering sub-unit is further used to: construct multiple download tasks corresponding to the target resources; prioritize the multiple download tasks based on preset priorities to obtain priority sorting results; trigger each download task in turn according to the priority sorting results to generate second media data.

[0143] In some optional embodiments, the above-mentioned rendering sub-unit is further specifically used to: detect whether the number of download tasks exceeds the preset concurrency; when the number of download tasks exceeds the preset concurrency, determine one or more current download tasks according to the preset concurrency; trigger one or more current download tasks in sequence according to the priority sorting results to obtain the second media data corresponding to each download task.

[0144] In some optional implementations, the resource acquisition module 401 may include:

[0145] The information acquisition unit is used to obtain resource description information in the rendering template in the media editing interface.

[0146] The remote resource acquisition unit is used to acquire the corresponding media remote resource based on the resource description information.

[0147] The resource conversion unit is used to convert the media remote resources according to the preset conversion protocol to obtain media rendering resources.

[0148] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0149] The rendering device of the media resources in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0150] The rendering device for media resources provided in this embodiment generates and plays a portion of the first media data from the multiple media rendering resources obtained in the media editing interface, and processes other media rendering resources based on the playback of the first media data, so as to add the corresponding playback effects of other media rendering resources to the first media data in real time. As a result, the media resources in the rendering template do not need to wait for all downloads to be completed before rendering, and can be downloaded and rendered at any time, thereby reducing the loading time of the media resources. Moreover, during the rendering process, some media rendering resources can be processed first to present the corresponding media data to the user, reducing the user's waiting time, and other media rendering resources can be rendered and played at the same time, so that the corresponding playback effects can be dynamically added based on the playback of the first media data, reducing the impact on the display of the media effects, and improving the user's preview experience of the media effects.

[0151] The embodiment of the present disclosure further provides a computer device having the rendering apparatus for media resources shown in FIG. 5 .

[0152] Please refer to Figure 6, which is a structural diagram of a computer device provided by an optional embodiment of the present disclosure. As shown in Figure 6, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 takes a processor 10 as an example.

[0153] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0154] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0155] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0157] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means, and FIG6 takes the bus connection as an example.

[0158] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0159] The computer device also includes a communication interface for data communication between the computer device and other devices or a communication network.

[0160] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0161] The embodiments of the present disclosure further provide a computer program, which includes computer instructions. When the computer instructions are executed by the aforementioned processor (eg, processor 10), the methods shown in the above embodiments are implemented.

[0162] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for rendering a media resource, comprising: Get multiple media rendering resources in the rendering template in the media editing interface; Generate first media data based on the multiple media rendering resources, the first media data comprising a first rendering resource set, the first rendering resource set comprising at least part of the media rendering resources; Playing the first media data in the media editing interface; as well as During the playing of the first media data, a playing effect of a second rendering resource set is added in real time, where the second rendering resource set includes other media rendering resources among the multiple media rendering resources except the first rendering resource set.

2. The method according to claim 1, wherein: The first rendering resource set includes essential rendering resources among the multiple media rendering resources, and generating first media data based on the multiple media rendering resources includes: identifying at least one required rendering resource from the plurality of media rendering resources; and The at least one required rendering resource is processed to generate first media data, where the first media data includes the at least one required rendering resource.

3. The method according to claim 2, wherein: The identifying at least one essential rendering resource from the plurality of media rendering resources comprises: identifying whether each of the plurality of media rendering resources carries a required identifier; and One or more media rendering resources carrying the essential identifier are determined as the essential rendering resources.

4. The method according to any one of claims 1 to 3, wherein: The adding the playback effect of the second rendering resource set in real time during the playback of the first media data includes: Based on the first media data and the second rendering resource set, second media data is generated, where the second media data includes the first media data and at least part of the media rendering resources in the second rendering resource set added in real time.

5. The method according to claim 4, wherein: The second rendering resource set includes non-essential rendering resources among the plurality of media rendering resources, and the generating second media data based on the first media data and the second rendering resource set includes: Within a preset time after processing the essential rendering resources, one or more of the non-essential rendering resources are processed to obtain a processing result, and based on the processing result and the first media data, second media data including the non-essential rendering resources are generated and played.

6. The method according to claim 5, wherein: Within a preset time after the essential rendering resource is processed, one or more of the non-essential rendering resources are processed to obtain a processing result, and based on the processing result and the first media data, second media data including the non-essential rendering resources is generated and played, including: When the first media data is played for a preset time, determining a playback position of the first media data; Based on the playback position, determining target resources required for a next playback position from the non-essential rendering resources; Processing the target resource according to a preset priority to generate the second media data corresponding to the target resource; and Based on the first media data, the second media data is played.

7. The method according to claim 6, wherein: The determining, based on the playback position, target resources required for the next playback position from the non-essential rendering resources includes: Obtaining a loading time corresponding to each of the non-essential rendering resources, and determining a target playback position of each of the non-essential rendering resources based on the first media data based on the loading time; and Compare the target playback position with the next playback position, and select the non-essential playback position according to the comparison result. The target resource required for the next playback position is filtered out from the standby rendering resources.

8. The method according to claim 6, wherein: Processing the target resource according to a preset priority to generate the second media data corresponding to the target resource includes: Constructing multiple download tasks corresponding to the target resources; Prioritizing the multiple download tasks based on the preset priorities to obtain a priority ranking result; and The multiple downloading tasks are triggered in sequence according to the priority sorting result to generate the second media data.

9. The method according to claim 8, wherein: The triggering each of the download tasks in sequence according to the priority sorting result to generate the second media data includes: Detecting whether the number of the multiple download tasks exceeds a preset concurrent amount; In response to determining that the number of the plurality of download tasks exceeds the preset concurrency amount, determining one or more current download tasks according to the preset concurrency amount; and The one or more current download tasks are triggered in sequence according to the priority sorting result to obtain the second media data corresponding to each download task.

10. The method according to any one of claims 1 to 3, wherein: The step of obtaining multiple media rendering resources in a rendering template in a media editing interface includes: Obtain resource description information in the rendering template in the media editing interface; Acquire corresponding media remote resources based on the resource description information; and The media remote resource is converted according to a preset conversion protocol to obtain the media rendering resource.

11. A media resource rendering device, comprising: The resource acquisition module is used to obtain multiple media rendering resources in the rendering template in the media editing interface. source; A generating module, configured to generate first media data based on the plurality of media rendering resources, wherein the first media data comprises a first rendering resource set, and the first rendering resource set comprises at least part of the media rendering resources; A playing module, used for playing the first media data in the media editing interface; as well as An adding module is used to add a playback effect of a second rendering resource set in real time during the playback of the first media data, where the second rendering resource set includes other media rendering resources among the multiple media rendering resources except the first rendering resource set.

12. A computer device comprising: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for rendering media resources according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, wherein computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed by a computer, the computer executes the method for rendering media resources according to any one of claims 1 to 10. 14 . A computer program, comprising computer instructions, which, when executed by a computer, enable the computer to execute the method for rendering a media resource according to claim 1 .

Citation Information

Patent Citations

  • Multimedia information playing method and system

    CN102929971A

  • Multimedia resource display method and device, terminal and storage medium

    CN112966130A

  • Video preview method and device, medium, equipment and computer program product

    CN115967824A

  • Page display method, device, and system, and page display assist method and device

    US20180253406A1