Web video player optimization method supporting H265 coding format

Abstract

The invention belongs to the technical field of video players, and discloses a Web video player optimization method supporting an H265 coding format, which comprises the following specific steps: step 1, a player is mainly divided into four parts and three threads, namely a UI (User Interface), a Loader, a data processing part and a data rendering part, and one thread is a main thread; one thread is a data loading thread; the last thread is a data processing thread; and step 2, UI: displaying by a player, wherein the UI comprises two parts, i.e., a screen and a controlbar. A traditional Web player can be optimized to improve user experience, H265 video decoding is supported, high-definition video playing in a low-bandwidth environment is achieved, a user-friendly interactive interface is provided, a user can conveniently conduct video playing control, playing performance is optimized, delay and lagging phenomena are reduced, complex video processing tasks are conducted through Web Workers, and the user experience is improved. The method has the advantages of improving page response speed, automatically loading and caching subsequent related videos during playing, improving playing speed and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of video players, and in particular relates to a method for optimizing a Web video player that supports the H265 encoding format. Background Art

[0002] Web video player is an online streaming network player, which is a web page plug-in. After running the prepared page in the browser, it will call the system's built-in player to play the pre-set audio (video) files, such as MP3 songs and other audio files. When the Web video player is running, it will first download the playback plug-in and enter the directory after decompression. In this process, music.htm is the playback bar page loaded on the homepage, list.htm is the pop-up page for viewers to view the play list, the js folder contains several playback control files, and img contains some playback interface image files, which supports the H264 video encoding protocol.

[0003] With the development of video encoding technology, compared with H264, H265 has many advantages such as half the volume, half the bandwidth, and finer image quality with the same image quality. However, Web browsers do not yet support H265 decoding and playback. Therefore, H265 playback on the Web is implemented based on WebAssembly (encapsulating FFmpeg), JS decapsulation, Canvas projection, and AudioContext. When playing high-definition videos, traditional Web players often experience freezes and delays due to bandwidth limitations and limited computing resources. In addition, high-definition videos cannot be played effectively in low-bandwidth environments. The user interaction interface is unfriendly and it is inconvenient for users to control video playback, which will further affect the user's viewing experience and bring a poor user experience to users. Summary of the invention

[0004] The purpose of the present invention is to provide a method for optimizing a Web video player supporting the H265 encoding format to solve the problems raised in the above-mentioned background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for optimizing a Web video player supporting the H265 encoding format, the specific steps are as follows:

[0006] Step 1:

[0007] The player is mainly divided into four parts: UI, Loader, data processing, and data rendering, and three threads. One is the main thread, which includes interface control, download control, data flow control, audio and video control, etc.; one is the data loading thread, which completes the request for metadata and data fragments; the last one is the data processing thread, which completes data decapsulation and decoding.

[0008] Step 2:

[0009] UI: Player display, including screen and controlbar. The screen includes video image display, pop-up window, poster image, etc. The controlbar includes components such as progress bar, play button, volume control, etc.

[0010] Step 3:

[0011] Loader: responsible for loading and parsing media data, realizing data request through worker. After loading is completed, the requested ts data is stored according to the set cache size. When the cache limit is reached, loading stops. After the decoder obtains ts from the ts data queue, Hls Loader will release the requested ts and continue to load the next ts. Loading stops when the maximum cache limit is reached.

[0012] Step 4:

[0013] Data processing: mainly includes data decapsulation and H265 decoding. Decapsulation is implemented by the demuxe.js library, and H265 decoding is implemented by wasm software decoding generated by ffmpeg packaging. The CPU usage rate is high.

[0014] Step 5:

[0015] Data rendering: includes video rendering and audio rendering. Video rendering uses ImagePlayer to render the decoded YUV data directly to the canvas. Audio uses AudioPlayer to decode the AAC data and then play the audio. Finally, pts is used to synchronize audio and video. The synchronization strategy uses audio as a reference to determine the difference between the current video pts and the acquired audio pts to adjust the video display time to achieve audio and video synchronization.

[0016] As a preferred technical solution of the present invention, the data decapsulation described in step 1 requires a decapsulator, which implements the decapsulation of video data through JS, which is a lightweight, interpreted or just-in-time compiled programming language with function priority.

[0017] As a preferred technical solution of the present invention, the decoder described in step three implements soft decoding of H265 data through ffmpeg, which is an open source computer program that can be used to record and convert digital audio and video and convert them into streams, using LGPL or GPL licenses.

[0018] As a preferred technical solution of the present invention, the ImagePlayer described in step five realizes YUV data rendering through yuv-canvas. There will be a queue in the ImagePlayer to store YUV data and calculate the time length of the current yuv data. When it is greater than or equal to the value of readybufferLength, the ImagePlayerReady event will be triggered.

[0019] As a preferred technical solution of the present invention, the implementation of the AudioPlayer described in step 5 is based on the WebAudio API, including decoding of the AAC audio stream and playback of PCM data.

[0020] As a preferred technical solution of the present invention, the decoding described in step 1, step 4 and step 5 refers to when the Audio Player module receives the AAC data stream sent by the decapsulation module, it decodes it into PCM data to be played subsequently through the decodeAudioData API of AudioContext.

[0021] As a preferred technical solution of the present invention, some audio nodes of the Audio Context provide a playbackRate attribute to implement the function of multiple-speed playback. When the audio is played at multiple speeds, the pitch of the audio is also synchronously increased / decreased.

[0022] As a preferred technical solution of the present invention, the Loader described in step three only supports the HLS protocol. The HLS protocol is a streaming media transmission protocol based on http and is a protocol interaction mode for transmitting audio and video.

[0023] As a preferred technical solution of the present invention, the H265 described in step 4 is a video coding standard, also known as high-efficiency video coding. H265 can realize 720P (resolution 1280*720) ordinary high-definition audio and video transmission at a transmission speed of 1-2Mbps.

[0024] As a preferred technical solution of the present invention, the wasm described in step 4 is a new format that is portable, small in size, fast in loading, and compatible with the Web. Native code can be executed in a browser through wasm.

[0025] The beneficial effects of the present invention are as follows:

[0026] The present invention can optimize the traditional Web player to improve the user experience, support H265 video decoding, realize high-definition video playback in a low-bandwidth environment, provide a user-friendly interactive interface, facilitate users to control video playback, optimize playback performance, reduce delays and freezes, use Web Workers to perform complex video processing tasks, improve page response speed, automatically load and cache subsequent related videos during playback, and improve playback speed, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Construct a flow chart for the present invention as a whole;

[0028] Figure 2 This is the main flow chart of the player of the present invention;

[0029] Figure 3 This is a flowchart of the specific playback process of the present invention;

[0030] Figure 4 This is the ImagePlayer flow chart;

[0031] Figure 5 This is the AudioPLayer flow chart. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] like Figures 1 to 5 As shown, the embodiment of the present invention provides a method for optimizing a Web video player supporting the H265 encoding format, and the specific steps are as follows:

[0034] Step 1:

[0035] The player is mainly divided into four parts: UI, Loader, data processing, and data rendering, and three threads. One is the main thread, which includes interface control, download control, data flow control, audio and video control, etc.; one is the data loading thread, which completes the request for metadata and data fragments; the last one is the data processing thread, which completes data decapsulation and decoding.

[0036] Step 2:

[0037] UI: Player display, including screen and controlbar. The screen includes video image display, pop-up window, poster image, etc. The controlbar includes components such as progress bar, play button, volume control, etc.

[0038] Step 3:

[0039] Loader: responsible for loading and parsing media data, realizing data request through worker. After loading is completed, the requested ts data is stored according to the set cache size. When the cache limit is reached, loading stops. After the decoder obtains ts from the ts data queue, Hls Loader will release the requested ts and continue to load the next ts. Loading stops when the maximum cache limit is reached.

[0040] Step 4:

[0041] Data processing: mainly includes data decapsulation and H265 decoding. Decapsulation is implemented by the demuxe.js library, and H265 decoding is implemented by wasm software decoding generated by ffmpeg packaging. The CPU usage rate is high.

[0042] Step 5:

[0043] Data rendering: includes video rendering and audio rendering. Video rendering uses ImagePlayer to render the decoded YUV data directly to the canvas. Audio uses AudioPlayer to decode the AAC data and then play the audio. Finally, pts is used to synchronize audio and video. The synchronization strategy uses audio as a reference to determine the difference between the current video pts and the acquired audio pts to adjust the video display time to achieve audio and video synchronization.

[0044] The present invention provides a Web player supporting H265 encoding, including an H265 decoder, a streaming media transmission module, a Web Workers processing module, etc., which realizes high-definition video playback in a low-bandwidth environment and improves user experience. The player has the following characteristics: supports H265 video decoding and can realize high-definition video playback in a low-bandwidth environment; provides a user-friendly interactive interface to facilitate users to perform video playback, pause, fast-forward and other operations; optimizes playback performance and reduces delays and freezes.

[0045] Among them, the data decapsulation in step 1 requires the use of a decapsulator, which implements the decapsulation of video data through JS, which is a lightweight, interpreted or just-in-time compiled programming language with function priority.

[0046] By using the decapsulator, independent video (H265) data and audio (AAC) data can be obtained, and the demuxer module is already available.

[0047] Among them, the decoder in step three implements soft decoding of H265 data through ffmpeg. ffmpeg is an open source computer program that can be used to record, convert digital audio and video, and convert them into streams, using LGPL or GPL licenses.

[0048] ffmpeg provides a complete solution for recording, converting and streaming audio and video. It includes the advanced audio / video codec library libavcodec. ffmpeg is developed on the Linux platform, but it can also be compiled and run in other operating system environments, including Windows, Mac OS X, etc., and has a wide range of applications.

[0049] Among them, the ImagePlayer in step 5 realizes YUV data rendering through yuv-canvas. There will be a queue in ImagePlayer to store YUV data and calculate the time length of the current yuv data. When it is greater than or equal to the value of readybufferLength, the ImagePlayerReady event will be triggered.

[0050] When both the audio and video players are in the ready state, the H265 player triggers the dataReady event and starts calling the play method to play the video.

[0051] The implementation of AudioPlayer in step 5 is based on the Web Audio API, including decoding of AAC audio stream and playback of PCM data.

[0052] The Web Audio API is an advanced JavaScript API for processing and synthesizing audio on the web. The Web Audio API draft specification is customized by the W3C audio working group and is supported and optimized by the C++ engine at the bottom. The Web Audio API provides a very rich interface for developers to process web audio on the web platform. Using the Web Audio API, web developers can implement audio effects such as audio sound effects, audio visualization, 3D audio, etc. on the web platform, and the operation is quick and convenient.

[0053] The decoding in step 1, step 4 and step 5 refers to when the Audio Player module receives the AAC data stream sent by the decapsulation module, it decodes it into PCM data to be played later through the decodeAudioData API of AudioContext.

[0054] AudioNode is the audio data "processor" in Web Audio API. Web Audio API processes audio data serially through different "processors" and finally plays it through speakers (audioContext.destination). Web Audio API provides a variety of audio source nodes (ScriptProcessorNode) to provide audio data, such as AudioBufferSourceNode, MediaElementAudioSourceNode, etc. Considering the demand for double-speed playback, the present invention adopts ScriptProcessorNode to request audio data through the onaudioprocess callback function.

[0055] Among them, some audio nodes of Audio Context provide the playbackRate attribute to realize the function of double-speed playback. When the audio is played at double speed, the pitch of the audio is also increased / decreased synchronously.

[0056] In order to achieve speed change without changing pitch, the audio data processing library SoundTouchJS is used to provide the processed audio data without changing pitch to ScriptProcessorNode. The present invention uses this plug-in to call the code example:

[0057]

[0058] The Loader in step 3 only supports the HLS protocol. The HLS protocol is an HTTP-based streaming media transmission protocol and a protocol interaction method for transmitting audio and video.

[0059] The HLS protocol cuts the audio and video streams into small ts slices and generates an m3u8 playlist file. The playback client downloads the playlist file through the http protocol, downloads the slice files and plays them in the order specified in the playlist file, thereby achieving download-while-playing, similar to the effect of real-time online playback.

[0060] H265 in step 4 is a video coding standard, also known as high-efficiency video coding. H265 can achieve 720P (resolution 1280*720) ordinary high-definition audio and video transmission at a transmission speed of 1-2Mbps.

[0061] The goal of H265 is to provide higher video compression efficiency, enabling it to provide higher video quality and higher resolution under the same bandwidth and storage space. H265 uses more advanced encoding technology, which can reduce the bit rate by about 50% compared to H264, while providing better image quality.

[0062] Among them, wasm in step 4 is a new format that is portable, small in size, fast in loading, and compatible with the Web. Through wasm, native code can be executed in the browser.

[0063] wasm is a virtual instruction set architecture (virtual ISA). The overall architecture includes the core ISA definition, binary encoding, definition and execution of program semantics, and application programming interfaces (WebAssembly API) for different embedded environments (such as the Web). Its initial goal is to compile programs written in languages ​​​​such as C / C++ and run them better on the Web platform while ensuring security and running speeds close to native applications.

[0064] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0065] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for optimizing a Web video player supporting the H265 encoding format, characterized in that: The specific steps are as follows: Step 1: The player is mainly divided into four parts: UI, Loader, data processing, and data rendering, and three threads. One is the main thread, which includes interface control, download control, data flow control, audio and video control, etc.; one is the data loading thread, which completes the request for metadata and data fragments; the last one is the data processing thread, which completes data decapsulation and decoding. Step 2: UI: Player display, including screen and controlbar. The screen includes video image display, pop-up window, poster image, etc. The controlbar includes components such as progress bar, play button, volume control, etc. Step 3: Loader: responsible for loading and parsing media data, realizing data request through worker. After loading is completed, the requested ts data is stored according to the set cache size. When the cache limit is reached, loading stops. After the decoder obtains ts from the ts data queue, Hls Loader will release the requested ts and continue to load the next ts. Loading stops when the maximum cache limit is reached. Step 4: Data processing: mainly includes data decapsulation and H265 decoding. Decapsulation is implemented by the demuxe.js library, and H265 decoding is implemented by wasm software decoding generated by ffmpeg packaging. The CPU usage rate is high. Step 5: Data rendering: includes video rendering and audio rendering. Video rendering uses ImagePlayer to render the decoded YUV data directly to the canvas. Audio uses AudioPlayer to decode the AAC data and then play the audio. Finally, pts is used to synchronize audio and video. The synchronization strategy uses audio as a reference to determine the difference between the current video pts and the acquired audio pts to adjust the video display time to achieve audio and video synchronization.

2. According to claim 1, a method for optimizing a Web video player supporting the H265 encoding format is characterized in that: The data decapsulation described in step 1 requires a decapsulator, which implements the decapsulation of video data through JS, which is a lightweight, interpreted or just-in-time compiled programming language with function priority.

3. According to claim 1, a method for optimizing a Web video player supporting the H265 encoding format is characterized in that: The decoder described in step 3 implements soft decoding of H265 data through ffmpeg. ffmpeg is an open source computer program that can be used to record, convert digital audio and video, and convert them into streams, using LGPL or GPL licenses.

4. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The ImagePlayer described in step 5 implements YUV data rendering through yuv-canvas. There will be a queue in ImagePlayer to store YUV data and calculate the time length of the current yuv data. When it is greater than or equal to the value of readybufferLength, the ImagePlayerReady event will be triggered.

5. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The implementation of AudioPlayer described in step 5 is based on the Web Audio API, including decoding of AAC audio streams and playback of PCM data.

6. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The decoding described in step 1, step 4 and step 5 refers to when the Audio Player module receives the AAC data stream sent by the decapsulation module, it decodes it into PCM data to be played later through the decodeAudioData API of AudioContext.

7. The method for optimizing a Web video player supporting the H265 encoding format according to claim 6, characterized in that: Some audio nodes of the Audio Context provide a playbackRate attribute to implement a function of double-speed playback. When the audio is played at double speed, the pitch of the audio is also increased / decreased synchronously.

8. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The Loader described in step 3 only supports the HLS protocol. The HLS protocol is an HTTP-based streaming media transmission protocol and a protocol interaction method for transmitting audio and video.

9. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The H265 described in step 4 is a video coding standard, also known as high-efficiency video coding. H265 can achieve 720P (resolution 1280*720) ordinary high-definition audio and video transmission using a transmission speed of 1-2Mbps.

10. The method for optimizing a Web video player supporting the H265 encoding format according to claim 1, characterized in that: The wasm described in step 4 is a new format that is portable, small in size, fast in loading, and compatible with the Web. Through wasm, native code can be executed in the browser.

Images