High-efficiency browser-based rendering method and system for video playback

Abstract

The present invention relates to the field of video playback rendering technology, and in particular to a browser-based high-efficiency rendering method and system for video playback, including: S1-reading the timeline of a video, judging whether all elements in a CG track can be supported by the browser, and if all can be supported, passing the CG track to the browser, and passing the A track content and the V track content in the video to a synthesis rendering module; S2-creating a channel between the browser CEF and the synthesis rendering module, and creating a JS timeline renderer-which is a rendered CG track; S3-superimposing the rendered V track with the rendered CG track, and then superimposing the A track. The beneficial effects achieved by the present invention are: since the synthesis rendering module uses the browser CEF to assist in creating the JS timeline renderer, the number of interactions is reduced, and the synthesis rendering efficiency is improved, the resource loading performance is optimized, the visual effect is enhanced, and the development and maintenance are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of video playback and rendering, and in particular to a browser-based high-efficiency rendering method and system for video playback. Background Art

[0002] In the process of video playback, in order to improve the fluency, it is usually necessary to render the played video. The current rendering methods include rendering by timeline, by entry and exit point area, and by the area where the clip is located.

[0003] In the method of composite rendering by timeline, it is divided into CG track, V track and A track in detail. The CG track mainly contains elements such as text and images (for example, CG special effects, barrage text, etc.), the V track mainly includes videos in various formats (mainly images), and the A track mainly includes audios in various formats.

[0004] 1. In the technology of rendering video playback according to the timeline, the original technology does not involve the browser (CEF), and all videos, audios, images, and subtitles are implemented through fixed protocol parameters in the timeline. This method is suitable for simple subtitles many years ago, but with the development of video playback technology, dynamic subtitles (such as barrage) have emerged. 2. In the dynamic subtitle technology, when using the timeline for rendering in video playback, the video track image in the V track, the audio track data in the A track, part of the text and image of the CG track and other elements are completed using a special synthesis rendering program, and then the other text and image of the CG track are completed through the browser (ie CEF). Finally, the result of the special synthesis rendering program is integrated with the structure completed by the browser (CEF) to complete the rendering. In layman's terms, in dynamic subtitle technology, the other part of the text and image in the CG track actually refers to the barrage (the barrage includes the text itself, as well as the special shapes and colors included in the text). During the rendering process, all subtitle effects are rendered by the browser. The advantage is that the synthesis rendering program no longer cares about the subtitle parameter content, no longer cares about how the subtitles are displayed, and only needs to obtain image data from the browser when needed (each interaction to obtain the image takes about 10 milliseconds, which basically meets the synthesis needs).

[0005] However, this dynamic subtitle technology also has shortcomings: because the interaction between the synthesis rendering program and the browser (CEF) is one dynamic subtitle corresponding to one browser process, if there are too many dynamic subtitles on the timeline CG track, the number of times the browser process is started and closed will also increase, causing performance loss. If there are multiple tracks of dynamic subtitles, there will be interactions with multiple browsers at the same time to obtain images, which will increase the time consumption exponentially. It takes 10 milliseconds for one track and about 20 milliseconds for two tracks, and the synthesis efficiency will drop sharply. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a browser-based high-efficiency rendering method and system for video playback, which solves the problem in video playback technology that the synthesis rendering efficiency drops sharply due to too many dynamic subtitles.

[0007] It should be noted that in existing dynamic subtitles, since the interaction between the conventional rendering program (conventional rendering module) and the browser (CEF) is one dynamic subtitle corresponding to one browser process, if there are too many dynamic subtitles, there will be multiple browser processes. In this way, the interaction between the conventional rendering program (conventional rendering module) and the browser (CEF) is also multiple times, resulting in low synthesis efficiency.

[0008] Well, considering that current browsers support processing of multiple image formats, such as jpg\png\gif\webp, etc., which basically meet the image format support in the timeline, can we strip all the CG track content and let the browser process it (not only strip dynamic subtitles, but also other CG special effects); in addition, it is also considered that the CG track image always exists above the V track. Therefore, this plan proposes that all elements of the CG track be rendered by the browser, and only one browser process is started for the synthesis of a timeline. You only need to superimpose the image data obtained from the browser on the screen of the V track, and then merge the audio data to generate a video.

[0009] The purpose of the present invention is achieved through the following technical solutions:

[0010] In a first aspect, a browser-based efficient rendering method for video playback is disclosed, comprising the following steps:

[0011] S1-When the synthesis rendering module receives a playback request from the front end, it determines whether the CG track in the timeline of the video contains elements that are not supported by the browser; if all elements in the CG track are supported by the browser, the browser CEF is started, the CG track is transmitted to the browser, and the CG track rendering work is prepared, and the A track content and the V track content in the timeline of the video are left in the synthesis rendering module for subsequent processing;

[0012] S2-establishes a channel between the browser CEF and the synthesis rendering module, through which the synthesis rendering module obtains the CG track content, thereby creating a main container canvas; then the CG track is rendered on the main container canvas through the channel to form a CG track canvas, which is the JS timeline renderer; in addition, an interaction mode is established between the JS timeline renderer and the V track processed and the A track processed in the synthesis rendering module;

[0013] S21-After starting the browser CEF, write the webpage address where the JS timeline renderer to be created in the synthesis rendering module is located, and establish data interaction between the browser CEF and the JS timeline renderer to be created in the synthesis rendering module;

[0014] S22-Create JS timeline renderer: S221. When the browser CEF is started, the synthesis rendering program will inject the CG track data in the timeline of the video into the window node, so that the JS timeline renderer can immediately obtain the CG track data in the timeline when it is created, thereby starting the creation of the timeline renderer and the creation of the JS time renderer; S222. First parse the CG track data in the timeline through JavaScript to obtain the timeline information - that is, obtain the height and width of the timeline track, and then create a canvas of the required size for the timeline, which is the main container; S223. Parse the CG track data in the timeline through JavaScript again, read all the segment elements in the CG track, each segment element has its own entry point and exit point attributes; create a corresponding display object for each segment element, and add one to the main container after each display object is created; in this way, the CG track is rendered on the main container canvas according to the timeline to form a CG track canvas, which is the JS timeline renderer;

[0015] S3 - Synthetic rendering program generates the finished video;

[0016] Render the V track to get the V track canvas;

[0017] Overlay the CG canvas with the V-track canvas to get the final image at a certain moment;

[0018] Overlay the A track with the final picture at a certain moment to finally get the finished video file.

[0019] Further, in said S1, when reading the timeline of the video, if it is determined that the CG track contains elements that are not supported by the browser, the CG track is not passed to the browser for processing, but is rendered in the conventional rendering module. The conventional rendering module here refers to another program module existing in the backend server, which executes a common rendering method - that is, directly rendering and synthesizing the CG track, the A track, and the V track.

[0020] Further, in the above S221, when the browser CEF is started, the browser CEF will pass the timeline data of the CG track to the synthesis rendering program, and then the synthesis rendering program will inject the CG track timeline data into the window node. When the browser CEF passes the timeline data of the CG track to the synthesis rendering program: all the contents to be loaded in the CG track are queued and loaded; and the loading progress status is fed back to the synthesis rendering module; wherein the loading progress is expressed as a percentage from 0% to 100%, 0% indicates the initialization state, 0%-100% indicates the initialization state, 100% indicates the initialization completion state, and in addition, there is an initialization error state.

[0021] Explanation of the initialization error status: Since the CG track is image or text data, it is necessary to load image resources or font resources through the network. There may be a loading failure problem. At this time, it is necessary to return an error status to let the synthesis rendering module retry or set it to failure.

[0022] Further, after all the contents to be loaded in the CG track have been loaded - that is, after the JS timeline renderer is initialized: analyze and generate rendering range data: analyze the entry point, exit point and total length of the timeline of different types of data in the CG track, and mark all points of the timeline with continuous rendering types. The specific types are divided into dynamic area, static area and empty area. The time range of the divided area is the rendering range data. Analyze and generate the number of interactions: in the dynamic area, according to the number of different pictures at different time points, the corresponding number of interactions is determined; in the static area, one interaction is determined; in the empty area, it is determined that no interaction is generated. The rendering range data and the number of interactions are called back to the synthesis rendering module and recorded.

[0023] Further, when the rendering range data and the number of interactions are called back to the synthesis rendering module and recorded: start to obtain data from the loaded JS time renderer: each time data is obtained is an interaction, and the time used for an interaction includes the rendering time of the JS time renderer and the time for transmitting image data. In the process of obtaining data: for dynamic areas, interactions are performed according to the determined number of interactions; if it is a static area, only one interaction is performed according to the determined number of interactions; if it is an empty area, no interaction is generated according to the determined number of interactions.

[0024] Furthermore, in the S2: the different types of data in the CG track include dynamic subtitles and image data. After the different types of data in the CG track are created as corresponding segment elements, the segment elements have a hierarchical attribute, and their hierarchy is consistent with the CG track hierarchy, so that the high-level segment elements of the CG track are always located above the low-track hierarchy. In the CG track, the segment elements of each track contain their own resources, which are loaded through the web address of the JS timeline renderer written in S1.

[0025] Further, the rendering range data: after the initialization in the JS timeline renderer is completed, the entry point, exit point and total length of the timeline of each segment element in the timeline CG track are analyzed; then the timeline CG track is edited into a splicing form of multiple dynamic areas, multiple static areas, and multiple empty areas; then the timeline CG track is divided into a time interval range of multiple dynamic areas, multiple static areas, and multiple empty areas - that is, the rendering range data.

[0026] It should be noted that a segmentation element is either dynamic or static; there is no segmentation element in the empty area, all segmentation elements are static in the static area, and at least one of all segmentation elements is dynamic in the dynamic area.

[0027] Further, in said S3, specifically: Rendering the V track: Decoding the V track through the synthesis rendering module, and according to the special effects on the V track, finally drawing the picture on the V track on the canvas of the V track, that is, rendering to form the V track canvas. Superimposing the CG track canvas and the V track canvas: According to the interactive method in S2, superimposing the CG track canvas and the V track canvas to obtain the final picture at a certain moment. Superimposing the A track: Sending the final picture at a certain moment - that is, this frame picture, to the encoder ffmpeg or the SDK of the synthesis rendering module, and at the same time, sending the audio data in the A track to the encoder ffmpeg or the SDK of the synthesis rendering module, encoding continuously from left to right along the timeline, and finally obtaining the finished video file.

[0028] In a second aspect, a system based on the above-mentioned browser-based high-efficiency rendering method for video playback is also disclosed, including a back-end processor, a back-end video storage, and a front-end browser;

[0029] The back-end video storage is used to store video materials;

[0030] The back-end processor: after receiving a play request from the browser, the video material is called from the video storage: a. Open the timeline data; b. Parse the timeline information, obtain the height and width of the timeline, and create a rendering main container; c. Parse the timeline track information and create a timeline canvas for each track; d. Parse the segment elements of each track and create different display objects for different types of elements; e. Wait for the initialization of the display object, the picture needs to be loaded into the memory for decoding, and the video needs to load the video parsing header to analyze the height and width of the video; f. After the main container waits for all display objects to be initialized, it starts rendering from the beginning, obtains all canvases of this frame and combines them into one canvas, and sends it to the encoder; g. The encoder starts writing the video stream, and the main program notifies the synthesis progress externally; h. Render to the last frame, end the encoder, generate a video file, and complete the timeline rendering synthesis task;

[0031] The front-end browser receives the CG track and transmits the CG track data to the back-end processor, thereby assisting the back-end processor in creating a real-time canvas.

[0032] Technical explanation:

[0033] Synthesis rendering module: It is essentially a synthesis rendering program run by the processor of the backend server, through which the above steps are executed; it is written as "synthesis rendering module / synthesis rendering program" just for the convenience of description. In this solution, the program module corresponding to all steps S1 to S3 executed by the backend server when it determines that all elements of the CG track can be rendered with the assistance of browser CEF is called the synthesis rendering module; when the backend server determines that some elements of the CG track cannot be processed with the assistance of CEF, it is processed by another program block of the backend server. This part of the program module is called the conventional synthesis module / ordinary synthesis module. The conventional synthesis module synthesizes the CG track, A track, and V track together in the backend according to the common method of rendering them together.

[0034] Timeline data: refers to the data formed by the timeline of the video. It is a complete data that can describe the entire synthesis task. It contains the timeline information (height, width, frame rate, etc.), track, overall entry point, exit point, etc.; the most important is the track (or track data), which is divided into CG track, V track, and A track. Each track has different segment elements (or segment data).

[0035] Segment elements: refers to each piece of data contained in the timeline track, which may be pictures, videos, audios, texts, etc. Although CG tracks can only have texts and pictures, V tracks can only have videos and pictures, and A tracks can only have audios, they all have the same basic properties, such as relative to the timeline's in and out points, and only the data in this interval will be rendered.

[0036] The present invention has the following advantages:

[0037] (1) All CG tracks are rendered by the browser CEF. The synthesis rendering program only needs to obtain its image data and no longer cares about the content of the CG track data. It only needs to care about the V track and A track data, which reduces the difficulty of development. For CG track materials, developers can focus more on the realization of page effects without paying too much attention to the underlying rendering details, thereby simplifying the development process.

[0038] (2) The original timeline may have multiple operations that interact with the browser CEF. The current solution only requires one, avoiding repeated startup of the CEF process to consume resources. When multiple dynamic subtitles use the same font resource, it also reduces the repeated loading of font resources. The original timeline exists in multiple CG tracks. One track will occupy a canvas or decoding resource of the synthesis rendering program. This solution can draw all CG tracks on the same canvas. The browser CEF stacks multiple track elements into one layer and manages them through the track level. It can compress unnecessary synthesis layers and prevent the occurrence of layer explosion. The final output is on one canvas, realizing multiple operations combined into one.

[0039] (3) By marking the timeline data, the timeline is divided into a dynamic area, a static area, and an empty area. The dynamic area is acquired every frame, the static area is acquired only once, and the empty area is no longer acquired, thereby reducing the number of interactions between the synthetic rendering program and the browser (CEF), further improving the efficiency of synthetic rendering.

[0040] (4) In summary, using the browser rendering timeline CG track for synthesis can improve synthesis rendering efficiency, optimize resource loading performance, enhance visual effects, and facilitate development and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A simple schematic diagram of using the existing technology for timeline-based composite rendering in the field of video playback;

[0042] Figure 2 A flowchart for synthesizing and rendering by timeline using the prior art in the field of video playback;

[0043] Figure 3 A flowchart of rendering and synthesis according to the timeline in the present invention in the field of video playback;

[0044] Figure 4 It is a way to interact with the JS time renderer after loading the CG data;

[0045] Figure 5 The graphical result of rendering range data. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0048] In the field of video playback, if the existing "synthetic rendering by timeline" technology is used, the synthetic rendering efficiency will drop sharply when there are too many dynamic subtitles.

[0049] It should be noted that in the composite rendering of the timeline, three types of content will be composited and rendered, namely the Cg track - mainly including text, images and other elements, the V track - mainly including videos in various formats, and the A track - mainly including audio in various formats.

[0050] The following is a detailed description of the shortcomings of the prior art "composite rendering by timeline":

[0051] The working process and principle of the timeline composite rendering in the prior art are described in detail. Figure 1 and Figure 2 , Figure 1 It is a simple timeline. The following operations are performed through the back-end synthesis rendering program: Render from left to right in sequence. At this time, the program will create four canvases, corresponding to the four tracks of CG3, CG2, CG1, and V1. When the picture at a certain moment is obtained, each canvas automatically renders the picture at this moment to its own canvas. Finally, the main program combines the four canvases into one to complete the synthesis of the picture at a certain moment. During this process, dynamic subtitle 1 will start the browser process to interactively obtain the image. The corresponding CG1 canvas will take about 10 milliseconds. After completion, the browser process will be closed, and then dynamic subtitle 2 and dynamic subtitle 3 will come. Since these two dynamic subtitles appear in the picture at the same time, two browser processes will be opened at this time, and the CG1 canvas and CG2 canvas will be drawn simultaneously, which takes about 20 milliseconds. Then it comes to "Image A", "Image B" and "Image C" (referring to the attached manual Figure 1 The "Image A", "Image B" and "Image C" shown in the picture content of the CG1 canvas, the CG2 canvas and the CG3 canvas are drawn at the same time. Since the synthesis rendering program is generally developed in C++ language, the decoding speed of graphic images is relatively high, so this position takes less time. Finally, it comes to dynamic subtitle 4. At this time, the synthesis process of dynamic subtitle 1 will be repeated, and the browser process will be started to draw the subtitle screen to the CG1 canvas. The time keeps increasing until the last frame of the timeline, and the work of synthesizing the video on the timeline is finally completed.

[0052] From the above content, we can see that in the existing timeline-based synthesis rendering technology: when the timeline is initially synthesized and rendered, there is no participation of the browser CEF, and all videos, audios, images, and subtitles are implemented through fixed protocol parameters in the timeline. Because the original subtitle attributes were too simple, dynamic subtitles were developed later, and all subtitle effects were rendered by the browser. The advantage is that the synthesis rendering program no longer cares about the subtitle parameter content, no longer cares about how the subtitles are displayed, and only needs to obtain image data from the browser when needed. Each interaction to obtain the image takes about 10 milliseconds, which basically meets the synthesis requirements.

[0053] However, since the interaction between the synthesis rendering program and the browser CEF is one dynamic subtitle corresponding to one browser process, if there are too many dynamic subtitles on the timeline CG track, the number of browser process startup and shutdown will also increase, resulting in performance loss. If there are multiple tracks of dynamic subtitles, there will be interactions with multiple browsers at the same time to obtain images, which will increase the time consumption exponentially. It takes 10 milliseconds for one track and about 20 milliseconds for two tracks, and the synthesis efficiency will drop sharply.

[0054] Therefore, it can be briefly said that in the existing timeline-based composite rendering technology, when dynamic subtitles exist, the composite rendering program will repeatedly enable or even simultaneously enable the browser for interaction, which greatly reduces efficiency.

[0055] To this end, this specific implementation provides a browser-based high-efficiency rendering method for video playback, by creating a JS timeline renderer, which supports stripping all CG tracks for rendering. The synthetic rendering module (essentially a synthetic rendering program) only needs to start a browser CEF to communicate with the JS timeline renderer to obtain the CG track content, and the rest only needs to care about the V track and A track of the existing renderer, merge the rendered CG track with the rendered V track, and then merge the two together. The content is merged with the A track to finally get the finished video.

[0056] It should be noted that the synthesis rendering module / synthesis rendering program referred to in the present invention is essentially a program run by the processor of the back-end server, through which corresponding operations are performed; for the convenience of description, these running programs are written as synthesis rendering modules / synthesis rendering programs, and "back-end server" can be used to replace "synthesis rendering module / synthesis rendering program" to understand this solution.

[0057] First, as Figure 3 As shown, a browser-based high-efficiency rendering method for video playback is disclosed, comprising the following steps:

[0058] S1-judgment process.

[0059] The judgment process is mainly used for compatibility with the historical timeline: if there are elements in the CG track that are not supported by the browser - which is a rare case, the original synthesis process is followed (the current conventional method is to synthesize and render the CG track, V track, and A track on the back-end server); if the elements (or content) in the CG track can all be supported by the browser, the CG track will be passed to the browser, and the subsequent process in this solution will be followed, leaving the A track content and V track content of the CG track in the synthesis rendering module for subsequent processing.

[0060] It should be noted that when judging, it is the backend that directly calls the stored video for analysis, rather than the browser CEF passing the corresponding data to the synthesis rendering module for further analysis. In layman's terms, in the field of video playback, videos are stored in the backend server. When a request is received, the backend directly analyzes the video to determine whether the CG track in the video has elements that are not supported by the browser. The elements here refer to different types of data in the CG track, such as dynamic subtitles, image data, etc.

[0061] It should be noted that the original synthesis process usually has two methods: the first method is to render the V track, A track, and CG track according to the timeline through the synthesis rendering module; the second method is to process the elements of the CG track that can be supported by the browser through the browser CEF, and process the elements of the CG track that cannot be supported by the browser through the synthesis rendering module in a conventional manner. In addition, the V track and A track are also processed through the synthesis rendering module in a conventional manner. However, in the existing second processing method, there is a problem of too many interactions between the browser CEF and the synthesis rendering module during rendering.

[0062] S2-Establish a channel between the browser CEF and the backend. The backend obtains the CG track content through the channel, thereby creating the main container canvas. Then, the CG track is rendered on the main container canvas through the channel to form a CG track canvas. The CG track canvas is the JS timeline renderer. It can be said that the JS timeline renderer is formed after the CG track is rendered. In addition, the interaction between the JS timeline renderer and the V track and A track processed in the backend is also established.

[0063] S2 is further described below.

[0064] S21-After starting the browser CEF, write the webpage address where the JS timeline renderer to be created in the backend is located, and establish data interaction between the browser CEF and the JS timeline renderer to be created in the backend.

[0065] It should be further explained that the browser CEF is the core of the browser and is a callable program; when the browser CEF is started, it can write the web address of the JS timeline renderer to be created in the backend, and establish a channel with the backend through the web address. During the creation of the JS timeline renderer, the browser CEF passes the CG track content to the JS timeline renderer being created through the web address, thereby completing the creation of the JS timeline renderer.

[0066] S22. Create a timeline renderer.

[0067] S221. When the browser CEF is started, the synthesis rendering module directly obtains the video stored in the backend, and then injects the CG track data in the timeline of the video into the window node, so that the JS timeline renderer can immediately obtain the CG track data in the timeline when it is created, thereby starting the creation of the timeline renderer and the creation of the JS time renderer.

[0068] 1. It should be noted that when the browser CEF is started, the synthesis rendering module directly calls the video from the backend, and then injects the CG track data in the timeline of the video into the window node. When the browser CEF passes the CG track data in the timeline to the synthesis rendering program: all the contents to be loaded in the CG track data are queued and loaded; and the loading progress status is fed back to the synthesis rendering module; among them, the loading progress is expressed as a percentage from 0% to 100%, 0% indicates the initialization state, 0%-100% indicates the initialization state, 100% indicates the initialization completed state, and there is also an initialization error state. Due to the amount of all the content to be loaded in the CG track and the network light, the loading time cannot be determined. By feeding back the loading accuracy to the synthesis rendering module, it can avoid the synthesis rendering module from consistently handling the pseudo-death state.

[0069] Initialize error status: Since the CG track is image or text data, it is necessary to load image resources or font resources through the network. There may be a loading failure problem. At this time, it is necessary to return an error status to let the synthesis rendering program retry or set it to failure.

[0070] Second, it should also be noted that after all the content to be loaded on the CG track is loaded, that is, after the JS timeline renderer is initialized, the following steps are performed:

[0071] (1) It is necessary to analyze and obtain rendering range data: that is, it is necessary to analyze the entry point, exit point and total length of the timeline of each segment element in the CG track. Then edit the timeline CG track into a splicing form of multiple dynamic areas, multiple static areas, and multiple empty areas; then divide the timeline CG track into multiple dynamic areas, multiple static areas, and multiple empty areas - that is, rendering range data.

[0072] a. The following explains the entry point, exit point and total length of the timeline of each segment element in the CG track.

[0073] The data types in the CG track include dynamic subtitles and image data. Each data type can be cut and processed by the browser CEF to form multiple segments, and each small segment formed after cutting is a segment element. The segment element has its own resources. For example, the data of the dynamic subtitle type is cut and processed in the browser CEF, thus forming multiple small segment elements. These segment elements themselves carry information such as font files, image elements, and image files that need to be loaded; then the browser CEF loads the segment elements formed by the cutting process to the synthesis rendering module through the web page address (the synthesis rendering module itself does not need to be cut).

[0074] Each segment element has its own entry point and exit point attributes. For example, the entry point of a segment element is 3000 milliseconds, the exit point is 5000 milliseconds, and the total duration is (5000-3000) milliseconds.

[0075] b. The following is an explanation of how to edit the timeline CG track into a form of splicing multiple dynamic areas, multiple static areas, and multiple empty areas.

[0076] First of all, it should be noted that a segmentation element is either dynamic or static. For example, if a segmentation element remains unchanged from 3000 milliseconds to 5000 milliseconds, it is static; if a switching element changes from 3000 milliseconds to 5000 milliseconds. To give a more popular example, for example, "a subtitle in the interval of 3000 milliseconds to 5000 milliseconds in the video" is used as a segmentation element. If this subtitle is always displayed on the screen during this time interval, it is static. If this subtitle is displayed word by word during this time interval, it is dynamic.

[0077] Editing the CG track in the timeline into multiple dynamic areas, multiple static areas, and multiple empty areas means: ① In the timeline, the CG track is divided into multiple time segments, each of which contains multiple segment elements; ② However, there are corresponding rules for dividing the CG track into multiple time segments: In the timeline, if there is no segment element in a certain time period, this time period will be divided out to form an empty area; if all segment elements in a certain time period are static, this time period will be divided out to form a static area; if at least one segment element in a certain time period is dynamic, this time period will be divided out to form a dynamic area.

[0078] (2) It is also necessary to analyze and know the number of interactions required - that is, the method of obtaining image data: After the above (1) operation, edit the CG track in the timeline into multiple dynamic areas, multiple static areas, and multiple empty areas, and then you can know the number of interactions required in the time analysis and subsequent generation. The rule for generating the number of interactions is: no interaction in the empty area, one interaction in the static area; in the dynamic area, you can choose to interact once per frame, so as to perform multiple interactions.

[0079] For example, the method of obtaining rendering range data in (1) and the number of required interactions in (2) above, that is, the method of obtaining image data, is described as follows:

[0080] The synthetic rendering program starts parsing the rendering range data at this time. By parsing the calibration in this data, we can know that 0-5000 milliseconds is the dynamic area, and each time point is a different picture, and each interaction is essential; 5000 milliseconds to 8000 milliseconds is the empty area, and there is no picture at all in this interval, so there is no need for interaction; and 13000 milliseconds to 15000 milliseconds is the static area, and this interval only needs to take the picture of a certain time and record it. When the program needs to take the picture of this interval next time, the recorded picture will be used, and no interaction with the JS timeline renderer will be performed;

[0081] If the synthesis rendering program is synthesized according to the timeline shown in the figure, this timeline is a 22000 millisecond timeline, starting from 0 to take the screen image, according to the frame rate of 25, that is, taking a frame of image data every 40 milliseconds, it is necessary to take 22000 / 40=550 times. However, if the interaction is controlled according to the analysis of the rendering range data, excluding the number of empty areas and static areas, only about 377 interactions are required (three dynamic areas 15000 / 40=375, plus two static areas once each), calculated at 10 milliseconds per interaction, saving 1730 milliseconds of synthesis rendering time, further improving synthesis efficiency.

[0082] 3. After the JS timeline renderer is initialized, the rendering range data obtained in (1) and the number of interactions required obtained in (2) - that is, the method for obtaining image data - are called back to the synthesis rendering program. The synthesis rendering program records both and begins to prepare for interactive data acquisition.

[0083] For example, the above three are described by way of example:

[0084] After the JS timeline renderer is initialized, the calculated rendering range data is passed to the composite rendering program. Specifically, the data structure can be represented by JSON parameters:

[0085] {"renderRanges":[{"trackIn":0,"trackOut":5000,"renderType":"dynamic"},{"trackIn":5000,"trackOut":8000,"renderType":"empty"},{"trackIn":8000,"trackOut":13000,"renderType":" dynamic "},{"trackIn":13000,"trackOut":15000,"renderType":"static"}

[0086] …]}

[0087] Finally, graphical results such as Figure 5 shown.

[0088] 4. After the above operations 1, 2, and 3 are completed, start to obtain data from the loaded JS time renderer: each time data is obtained is an interaction. The time used for an interaction includes the rendering time of the JS time renderer and the time to transmit image data.

[0089] For example, the above four are described as follows:

[0090] like Figure 4 As shown, the first dynamic area has dynamic subtitles, so the picture at each moment in this time zone is different. In this interval, the synthesis rendering program and the JS time renderer must interact. Then we come to the empty area 1. In this interval, no elements need to be displayed, and the synthesis rendering program and the JS time renderer do not need to interact once. Finally, we come to the static area 1. In this interval, "Image E", "Image F", and "Image G" are static png images (referring to the attached manual). Figure 4 In the example above, all images in this interval are exactly the same, so the composite renderer only needs to interact with the JS time renderer once. If this interval originally required 10 interactions, now it only needs one, which can greatly improve the efficiency of composite rendering.

[0091] S222. First parse the CG track data in the timeline through JavaScript to obtain the timeline information, that is, obtain the height and width of the timeline track, and then create a canvas of the required size for the timeline, which is the main container.

[0092] S223. Parse the CG track data in the timeline through JavaScript, read all the segment elements in the CG track, each segment element has its own entry point and exit point attributes; create a corresponding display object for each segment element, and add one to the main container after each display object is created; in this way, the CG track is rendered on the main container canvas according to the timeline to form a CG track canvas, which is the JS timeline renderer.

[0093] For example, S223 is described as follows:

[0094] For example, if the entry point of a segment element is 3000 milliseconds and the exit point is 5000 milliseconds, then this segment element will only be displayed between 3000 milliseconds and 5000 milliseconds in the timeline, and hidden at other times. After each segment element is created, one is added to the main container, so that each segment element has a hierarchical attribute, and its hierarchy is consistent with the hierarchy of the CG track, ensuring that the high-level segment elements of the CG track are always located above the low-level track hierarchy, thereby ensuring the consistency of timeline rendering.

[0095] S3 - Composition Rendering program produces the finished video.

[0096] After the above S2, all the image data of the CG track can be obtained through the browser CEF. In addition, in the above S2, the synthesis rendering module will create a special CG track, which contains only one object, namely the JS timeline renderer. In addition, the method of obtaining the CG track image data, that is, the interactive method, is also used by the above S2 to draw the image data on the canvas corresponding to this CG track. At the same time, in the original timeline synthesis process, the V track is also decoded, and according to the special effects on the V track, the picture is finally drawn on the canvas of the V track. The CG track canvas and the V track canvas are superimposed and synthesized to obtain the final picture at a certain moment. This frame of the picture is sent to the encoder ffmpeg or the rendering program's own SDK, and the corresponding audio data is also sent to the encoder. Encoding is performed continuously from left to right along the timeline, and finally a finished video file is obtained.

[0097] This specific embodiment also provides a system for a browser-based efficient rendering method for video playback, comprising: a back-end processor, a back-end video storage, and a front-end browser;

[0098] The back-end video storage is used to store video materials;

[0099] Backend processor: after receiving a playback request from the browser, it calls the video material from the video storage: a. Open the timeline data; b. Parse the timeline information, obtain the height and width of the timeline, and create a rendering main container; c. Parse the timeline track information and create a timeline canvas for each track; d. Parse the segment elements of each track and create different display objects for different types of elements; e. Wait for the initialization of the display object. The picture needs to be loaded into the memory for decoding, and the video needs to load the video parsing header to analyze the height and width of the video; f. After the main container waits for all display objects to be initialized, it starts rendering from the beginning, obtains all canvases of this frame and combines them into one canvas, and sends it to the encoder; g. The encoder starts writing the video stream, and the main program notifies the synthesis progress; h. Render to the last frame, end the encoder, generate a video file, and complete the timeline rendering and synthesis task;

[0100] The front-end browser receives the CG track and passes the CG track data to the back-end processor to assist the back-end processor in creating the real-time canvas.

[0101] Performance Testing:

[0102] It has been verified that when synthesizing a 10-second 1080P timeline video, the synthesis time using the existing synthesis method is more than 10 seconds; however, using the technology in this solution, the synthesis time required is only about 5 seconds.

[0103] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A browser-based high-efficiency rendering method for video playback, characterized in that: The following steps are involved: S1-When the synthesis rendering module receives a playback request from the front end, it determines whether the CG track in the timeline of the video contains elements that are not supported by the browser; if all elements in the CG track are supported by the browser, the browser CEF is started, the CG track is transmitted to the browser, and the CG track rendering work is prepared, and the A track content and the V track content in the timeline of the video are left in the synthesis rendering module for subsequent processing; In the above S1, when reading the timeline of the video, if it is determined that the CG track contains elements that are not supported by the browser, the CG track is not passed to the browser for processing, but is rendered in a conventional rendering module; The conventional rendering module refers to another program module existing in the backend server. The conventional rendering method of the program module is to directly render and synthesize the CG track, A track, and V track; S2-establishes a channel between the browser CEF and the synthesis rendering module, through which the synthesis rendering module obtains the CG track content, thereby creating a main container canvas; then the CG track is rendered on the main container canvas through the channel to form a CG track canvas, which is the JS timeline renderer; in addition, an interaction mode is established between the JS timeline renderer and the V track processed and the A track processed in the synthesis rendering module; S21-After starting the browser CEF, write the webpage address where the JS timeline renderer to be created in the synthesis rendering module is located, and establish data interaction between the browser CEF and the JS timeline renderer to be created in the synthesis rendering module; S22-Create JS timeline renderer: S221. When the browser CEF is started, the synthesis rendering module will inject the CG track data in the timeline of the video into the window node, so that the JS timeline renderer can immediately obtain the CG track data in the timeline when it is created, thereby starting the creation of the timeline renderer and the creation of the JS time renderer; S222. First parse the CG track data in the timeline through JavaScript, obtain the timeline information - that is, obtain the height and width of the timeline track, and then create a canvas of the required size for the timeline, which is the main container; S223. Parse the CG track data in the timeline through JavaScript again, read all the segment elements in the CG track, each segment element has its own entry point and exit point attributes; create a corresponding display object for each segment element, and add one to the main container after each display object is created; in this way, the CG track is rendered on the main container canvas according to the timeline to form a CG track canvas, which is the JS timeline renderer; S3 - Synthetic rendering program generates the finished video; Render the V track to get the V track canvas; Overlay the CG canvas with the V-track canvas to get the final image at a certain moment; Overlay the A track with the final picture at a certain moment to finally get the finished video file.

2. The browser-based high-efficiency rendering method for video playback according to claim 1, characterized in that: In the above S221, when the browser CEF is started, the browser CEF will pass the timeline data of the CG track to the synthesis rendering program, and then the synthesis rendering program will inject the timeline data of the CG track into the window node; When the browser CEF passes the timeline data of the CG track to the synthesis rendering program: all the contents to be loaded in the CG track are queued for loading; and the loading progress status is fed back to the synthesis rendering module; the loading progress is expressed as a percentage from 0% to 100%, 0% indicates the initialization state, 0%-100% indicates the initialization state, 100% indicates the initialization completion state, in addition, there is also an initialization error state.

3. The browser-based high-efficiency rendering method for video playback according to claim 2, characterized in that: When all the contents to be loaded in the CG track are loaded - that is, after the JS timeline renderer is initialized: Analyze and generate rendering range data: Analyze the entry and exit points of different types of data in the CG track and the total length of the timeline, mark all points of the timeline with continuous rendering types, and divide the specific types into dynamic area, static area, and empty area. The time range of the divided area is the rendering range data; Analyze and generate the number of interactions: In the dynamic area, the number of interactions is determined according to the number of different screens at different time points; In the static area, identify one interaction; In the empty area, it is determined that no interaction occurs; The rendering range data and interaction times are called back to the synthesis rendering module and recorded.

4. The browser-based high-efficiency rendering method for video playback according to claim 3, characterized in that: When the rendering range data and interaction times are called back to the synthesis rendering module and recorded: Start to obtain data from the loaded JS time renderer: each time data is obtained is an interaction. The time used for one interaction includes the rendering time of the JS time renderer and the time to transfer image data. In the process of obtaining data: if it is a dynamic area, interaction is performed according to a certain number of interactions; if it is a static area, only one interaction is performed according to the certain number of interactions; if it is an empty area, no interaction is generated according to the certain number of interactions.

5. The browser-based high-efficiency rendering method for video playback according to any one of claims 1 to 4, characterized in that: In said S2: The different types of data in the CG track include dynamic subtitles and image data; After the different types of data in the CG track are created as corresponding segment elements, the segment elements have a hierarchical attribute, and their hierarchical attributes are consistent with the hierarchical attributes of the CG track, so that the segment elements at the high level of the CG track are always located above the hierarchical attributes of the low track; In the CG track, each segment element of the track contains its own resources, and the resources are loaded through the web page address of the JS timeline renderer written in S1.

6. The browser-based high-efficiency rendering method for video playback according to claim 3 or 4, characterized in that: The rendering range data described is: After the JS timeline renderer is initialized, analyze the entry point, exit point and total length of each segment element in the timeline CG track; Then edit the timeline CG track into a spliced ​​form of multiple dynamic areas, multiple static areas, and multiple empty areas; then divide the timeline CG track into a time interval range of multiple dynamic areas, multiple static areas, and multiple empty areas - that is, rendering range data.

7. The browser-based high-efficiency rendering method for video playback according to claim 5, characterized in that: In S3, specifically: Rendering the V-track: Decoding the V-track through the synthesis rendering module, and finally drawing the image on the V-track on the canvas of the V-track according to the special effects on the V-track, that is, rendering to form the V-track canvas; Overlaying the CG track canvas and the V track canvas: According to the interactive method in S2, overlay the CG track canvas and the V track canvas to get the final picture at a certain moment; A track overlay: The final picture at a certain moment - that is, this frame of the picture, is sent to the encoder ffmpeg or the SDK of the synthesis rendering module. At the same time, the audio data in the A track is also sent to the encoder ffmpeg or the SDK of the synthesis rendering module, and encoded continuously from left to right along the timeline to finally obtain the finished video file.

8. The system of the browser-based high-efficiency rendering method for video playback according to any one of claims 1 to 7, characterized in that: Including back-end processor, back-end video storage, and front-end browser; The back-end video storage is used to store video materials; The back-end processor: upon receiving a playback request from a browser, calls the video material from the video storage: a. opens the timeline data; b. parses the timeline information, obtains the height and width of the timeline, and creates a rendering main container; c. Parse the timeline track information and create a timeline canvas for each track; d. Parse the segment elements of each track and create different display objects for different types of elements; e. Waiting for the initialization of display objects. Images need to be loaded into memory for decoding. Videos need to load the video parsing header to analyze the height and width of the video. f. The main container waits for all display objects to be initialized, starts rendering from the beginning, obtains all canvases of this frame and combines them into one canvas, and sends it to the encoder. g. The encoder starts writing the video stream, and the main program notifies the synthesis progress. h. Render to the last frame, end the encoder, generate a video file, and complete the timeline rendering and synthesis task; The front-end browser receives the CG track and transmits the CG track data to the back-end processor, thereby assisting the back-end processor in creating a real-time canvas.

Images