Webassembly-based browser audio and video playing method

Abstract

The application provides a webassembly-based browser audio and video playing method, which decodes data in units of GOP and evaluates decoding capability of a current decoder in real time based on a decoding delay gradient algorithm in an audio and video playing process, and adjusts decoding capability of the decoder by selecting different adjustment strategies according to an evaluation result. The application automatically reduces the number of decoding threads to release CPU resources and reduce waste of CPU resources when video decoding capability demand is low, automatically increases the number of decoding threads to meet decoding demand when video decoding capability demand is high, so that high-definition video can be smoothly played on browsers of different configuration machines, and adopts a frame dropping strategy when decoding capability reaches a peak value, so as to relieve decoding pressure of the decoder and ensure real-time video playing.

Description

Technical Field

[0001] This invention relates to the field of audio and video stream decoding and playback technology, and in particular to a browser-based audio and video playback method based on WebAssembly. Background Technology

[0002] With the rapid development of video playback applications, people have higher requirements for the smoothness of video playback. Many video websites have also provided corresponding web players for video playback on the browser side. However, how to improve the decoding capability of web-based videos to achieve smooth video playback has become a technical problem that the industry urgently needs to solve, especially the decoding capability of H.265 high-definition videos on the web.

[0003] The patent application CN 112291628 A discloses a method of allocating video frames to different threads for parallel decoding, with the number of decoding threads matching the number of CPU cores. This technique increases the number of frames that can be decoded on a multi-core CPU per unit time, improving video decoding capabilities to a certain extent. However, this method has the following drawbacks: Since video resolution, frame rate, image complexity, and machine performance all affect decoding capabilities, using the same number of decoding threads as the number of CPU cores for parallel decoding requires sorting the decoded frames even for videos with lower decoding requirements, leading to significant playback delays and wasting CPU resources. For videos with higher decoding requirements, even if the maximum decoding capacity of the decoding threads is insufficient, playback delays and stuttering will still occur. If the current machine performance is poor and the number of CPU cores is low, a number of decoding threads matching the number of CPU cores will also be insufficient to meet the current decoding requirements, similarly causing playback delays and stuttering, impacting user experience. Therefore, this application is filed. Summary of the Invention

[0004] The main objective of this invention is to address the aforementioned problems and shortcomings by providing a browser-based audio and video playback method based on WebAssembly. This method evaluates the current decoder's decoding capability in real time during audio and video playback and selects different adjustment strategies to achieve an adaptive balance between decoding capability requirements and the number of decoding threads.

[0005] A browser-based audio and video playback method based on WebAssembly, the method comprising:

[0006] S1. Obtain the audio and video stream to be played, and decapsulate it to obtain the valid audio and video data in the audio and video stream;

[0007] S2. Start the decoding thread and call the decoder to decode the valid audio and video data;

[0008] S3. Start the rendering thread to play the decoded audio and video;

[0009] S4. Evaluate the current decoder's decoding capability in real time, and select different adjustment strategies to adjust the decoder's decoding capability based on the evaluation results.

[0010] Furthermore, in step S4, the real-time evaluation of the current decoder's decoding capability is based on the decoding delay gradient algorithm to calculate the average decoding delay gradient within a preset time period, and the decoding capability of the decoder is evaluated by the numerical range of the average decoding delay gradient; the adjustment strategy includes adjusting the number of decoding threads, maintaining the current number of decoding threads, or a frame dropping strategy.

[0011] Furthermore, the process of real-time evaluation of the current decoder's decoding capability in step S4 is as follows:

[0012] S41. Calculate the decoding delay gradient Dm(i) between two adjacent video frames in the effective audio and video data. The decoding delay gradient Dm(i) is defined as the change obtained by subtracting the time interval between the decoding completion time of two adjacent video frames from the time interval between their decoding timestamps DTS.

[0013] S42. Calculate the average value of the decoding delay gradient Dm(i) between all two adjacent video frames in the effective audio and video data within the preset time period to obtain the average decoding delay gradient △Dm(j). Evaluate the decoding capability of the current decoder by the average decoding delay gradient △Dm(j).

[0014] S43. When △Dm(j)>0, the current decoder's decoding capability is insufficient. The decoding capability of the current decoder can be improved by increasing the number of decoding threads.

[0015] S44. When △Dm(j)<0, the current decoder's decoding capability meets the playback requirements. Therefore, the number of decoding threads will not be adjusted, and / or the number of decoding threads will be reduced to free up CPU resources.

[0016] Furthermore, the formula for calculating the decoding delay gradient Dm(i) is as follows:

[0017] Dm(i)=[(Ti–Ti-1)–(DTSi–DTSi-1)] / (DTSi–DTSi-1);

[0018] In the formula,

[0019] i≥2;

[0020] DTSi: Decoding timestamp of the i-th video frame;

[0021] DTSi-1: Decoding timestamp of the (i-1)th video frame;

[0022] Ti: Time to complete decoding of the i-th video;

[0023] Ti-1: The time to complete the decoding of the (i-1)th video.

[0024] Furthermore, in step S43, when increasing the decoding capability of the current decoder by increasing the number of decoding threads, the audio and video playback rate is also adjusted. The process is as follows: first, the audio and video playback rate PlayRate is calculated, and then the playback time T of the audio and video data is calculated at PlayRate. The calculation formula is: PlayRate = 1 / (1 + number of threads increased), T = (1 + number of threads increased) × GOP length.

[0025] Furthermore, after step S43, which improves the decoding capability of the current decoder by increasing the number of decoding threads, an additional step is added: the average decoding delay gradient ΔDm(j+1) in the next preset time period after the decoding capability adjustment is calculated and compared with ΔDm(j). When |ΔDm(j+1)–ΔDm(j)|<0.05, the decoding capability of the current decoder reaches its peak, and a frame dropping strategy is adopted to ensure the real-time performance of audio and video playback.

[0026] Furthermore, the maximum number of decoding threads is equal to the number of CPU cores.

[0027] Furthermore, step S1 includes,

[0028] S11. Start the streaming thread, obtain the audio and video streams to be played, and call the decapsulation module to decapsulate the obtained audio and video streams;

[0029] S12. Determine whether the audio and video stream is an FLV format audio and video stream and whether it contains audio and video data;

[0030] S13. Identify the FlvTag in the data packet corresponding to the FLV format audio and video stream based on the FLV format;

[0031] S14. Perform data parsing in units of FLV tags to extract the valid audio and video data corresponding to the FLV format audio and video streams.

[0032] Furthermore, step S1 also includes compiling the decoder based on Webassembly to generate a wasm decoding module for browser use. The wasm decoding module is obtained by compiling a decoding library written in C / C++ using the Emscripten compilation tool.

[0033] Furthermore, the process of calling the decoder to decode the valid audio and video data in step S2 includes,

[0034] S21. Initialize the decoding thread by calling the valid audio and video data described in step S1;

[0035] S22. The main thread allocates the decapsulated audio and video data from step S1 to the decoding thread in units of GOPs, and decodes the audio and video data into the original audio and video data.

[0036] S23. The decoding thread sends the original audio and video data back to the Javascript module and caches it in the original data queue for rendering and playback.

[0037] In summary, the browser audio and video playback method based on WebAssembly provided by this invention has the following advantages compared with the prior art:

[0038] (1) During the audio and video playback process in the browser, the decoding capability of the current decoder is evaluated in real time based on the decoding delay gradient algorithm. Different adjustment strategies are selected to adjust the decoding capability of the decoder. The relationship between the decoding capability requirement and the number of decoding threads is dynamically adjusted. When the video decoding capability requirement is low, the number of decoding threads is automatically reduced to release CPU resources and reduce the waste of CPU resources.

[0039] (2) By decoding data in units of GOP and dynamically adjusting the relationship between the decoding capability and the number of decoding threads, the decoding capability of the decoder is greatly improved, especially the decoding capability of H.265 encoded video. This enables smooth playback on browsers with different configurations, including some machines with slightly lower performance, greatly improving the user experience.

[0040] (3) After increasing the number of decoding threads, determine whether the current decoder's decoding capability has reached its peak. When the decoding capability reaches its peak, adopt a frame dropping strategy to alleviate the decoding pressure on the decoder and ensure the real-time performance of video playback.

[0041] (4) After the first GOP raw video data after decoding enters the raw video queue, the rendering thread immediately calls the data to render and play it on the browser. There is no need to wait for a certain amount of data to be cached in the raw video queue before rendering and playing, which shortens the video first screen playback time and achieves the effect of video opening in seconds.

[0042] (5) When playing H.265 format video on the browser using the browser method of the present invention, no plugins need to be installed, and high-definition video can be played smoothly and smoothly. This avoids compatibility issues and limitations of decoding types caused by installing plugins and improves security performance. Attached image description:

[0043] Figure 1 : Flowchart of the browser video playback method based on WebAssembly of the present invention;

[0044] Figure 2 : Schematic diagram of the video decoding capability adjustment principle of the present invention;

[0045] Figure 3 : Schematic diagram of the decoding delay gradient Dm(i) calculation method of the present invention. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] This invention provides a browser-based audio and video playback method based on WebAssembly, supporting playback of audio and video streams in various encoding formats, such as common video encoding formats like H.265 and H.264, and common audio encoding formats like AAC, G711, and Opus. This embodiment uses the playback of H.265 encoded video in a browser to illustrate the browser video playback method provided by this invention. First, the CodecID of the HTTP-FLV protocol is extended to support the parsing of H.265 encoded video streams. Then, the video stream to be played is encapsulated on the server side using the HTTP-FLV protocol, awaiting retrieval. Figure 1 As shown, the browser video playback method based on WebAssembly includes the following steps:

[0049] S1. Obtain the audio and video stream to be played, and decapsulate it to obtain the valid audio and video data in the audio and video stream. The specific steps are as follows:

[0050] S11. Start the streaming thread in the main thread, asynchronously obtain the video stream using the fetch method of Javascript according to the address of the video stream, and call the decapsulation module to decapsulate the obtained audio and video streams.

[0051] S12. Determine whether the video stream is an FLV format video stream and whether it contains audio and video data. Specifically, this includes using the Probe method to determine whether the first three bytes of the video stream header information are FLV, i.e., whether the first three bytes are 0x46, 0x4C, and 0x56, and whether the version number is 0x01, thereby determining whether the video stream is an FLV format video stream. At the same time, parse TypeFlagsAudio and TypeFlagsVideo to determine whether it contains audio and video data.

[0052] S13. Identify the FlvTag in the data packet corresponding to the FLV format video stream according to the FLV format;

[0053] S14. Perform data parsing on a per-FLV tag basis to extract the valid video data corresponding to the FLV format video stream. The FLV tag contains audio and video metadata, audio and video data, and scripts. The data parsing specifically includes the following steps.

[0054] S1401: Determine whether the current tag is an Audio Tag or a Video Tag by parsing the TagType; obtain the timestamp by parsing the Timestamp for playback control;

[0055] S1402. For video data, the video encoding type is determined by parsing the CodecId, and the video frame type is determined by parsing the FrameType, such as I-frame or P-frame.

[0056] S1403. For H.265 encoded video with CodecId 12, further parse PacketType to determine whether the Tag is HEVCdata or HEVCDecoderConfigurationRecord;

[0057] S1404. For tags of type HEVCDecoderConfigurationRecord, further parse and obtain information such as video resolution and frame rate.

[0058] Furthermore, step S1 also includes compiling the decoder based on Webassembly to generate a wasm decoding module for browser use. The wasm decoding module is obtained by compiling a decoding library written in C / C++ using the Emscripten compiler. This step ensures that the browser video playback method provided by this invention can be applied to the playback of audio and video streams with various encoding formats, especially H.265 encoded videos. The decoding library is one of the ffmpeg decoding library, libde265 decoding library, or libhevc decoding library. Preferably, this embodiment uses the Emscripten compiler to compile the FFmpeg decoding library written in C to obtain the wasm decoding module. Specifically, the decoding work is implemented by relying on the general interface of the FFmpeg library. First, the ffmpeg library is compiled into a static library using the Emscripten compiler. Then, C language decoder code is written to call the decoding library in ffmpeg. Finally, the code is compiled into a wasm decoding module for browser use.

[0059] S2. JavaScript starts a Worker decoding thread in the browser's main thread, calling the decoder corresponding to the encoding format of the video to be played to decode the valid audio and video data. The specific process is as follows:

[0060] S21. Initialize the Woke decoding thread by calling the valid video data obtained after decapsulating the video stream in step S1. The valid video data includes, but is not limited to, information such as CodecId, resolution, and frame rate.

[0061] S22. The main thread allocates the decapsulated video data from step S1 to the decoding thread in units of GOPs, and decodes the video data into the original video data.

[0062] S23. The decoding thread sends the raw video data back to the Javascript module and caches it in the raw data queue for rendering and playback.

[0063] S3. Start the rendering thread and play the decoded video according to the playback timestamp PTS;

[0064] After the first decoded GOP video data enters the raw data queue, the rendering thread immediately calls the raw GOP video data for rendering and playback on the browser. There is no need to wait for a certain amount of data to be cached in the raw video queue before rendering and playback, which shortens the first screen playback time of the video and achieves the effect of instant video playback. After the first GOP raw video data has finished playing, the raw GOP video data sorted by playback timestamp PTS is obtained from the raw data queue for rendering and playback.

[0065] S4. Real-time evaluation of the current decoder's decoding capability. Based on the evaluation results, different adjustment strategies are selected to adjust the decoder's decoding capability. These adjustment strategies include adjusting the number of decoding threads, maintaining the current number of decoding threads, or employing a frame dropping strategy. Specifically, an algorithm based on the decoding latency gradient is used to calculate the average decoding latency gradient over a preset time period. The decoder's decoding capability is evaluated using the numerical range of the average decoding latency gradient. Based on the evaluation results, different adjustment strategies are selected to adjust the decoder's decoding capability. These adjustment strategies include adjusting the number of decoding threads, maintaining the current number of decoding threads, or employing a frame dropping strategy. Figure 2 and Figure 3 As shown, the process of real-time evaluation of the current decoder's decoding capability is as follows:

[0066] S41. Calculate the decoding delay gradient Dm(i) between two adjacent video frames in the effective audio and video data. The decoding delay gradient Dm(i) is defined as the change obtained by subtracting the time interval between the decoding completion times of two adjacent video frames from the time interval between their decoding timestamps DTS. The formula for calculating the decoding delay gradient Dm(i) is: Dm(i) = [(Ti – Ti-1) – (DTSi – DTSi-1)] / (DTSi – DTSi-1).

[0067] In the formula, i ≥ 2;

[0068] DTSi: Decoding timestamp of the i-th video frame;

[0069] DTSi-1: Decoding timestamp of the (i-1)th video frame;

[0070] Ti: Time to complete decoding of the i-th video;

[0071] Ti-1: The time to complete the decoding of the (i-1)th video;

[0072] S42. Calculate the average value of the decoding delay gradient Dm(i) between all two adjacent video frames in the effective audio and video data within the preset time period to obtain the average decoding delay gradient △Dm(j). Evaluate the decoding capability of the current decoder by the average decoding delay gradient △Dm(j).

[0073] S43. When △Dm(j)>0, the current decoder's decoding capability is insufficient. The decoding capability of the current decoder can be improved by increasing the number of decoding threads. The maximum number of decoding threads is the number of CPU cores.

[0074] S44. When △Dm(j)≤0, the current decoder's decoding capability meets the playback requirements. Therefore, the number of decoding threads is not adjusted, and / or the number of decoding threads is reduced to free up CPU resources.

[0075] In step S4, the browser's decoding capability is evaluated in real time during audio and video playback. The relationship between decoding capability requirements and the number of decoding threads is dynamically adjusted so that the number of decoding threads and decoding capability requirements are adaptive. This satisfies the decoding capability and video playback requirements while making reasonable use of CPU resources and reducing CPU resource waste.

[0076] Furthermore, in steps S43 and S44, △Dm(j) can be further evaluated to determine whether the number of decoding threads needs to be increased, decreased, or maintained at the current number. This ensures sufficient decoding capability of the decoder without wasting CPU resources. The specific rules are as follows:

[0077] When △Dm(j) < -0.5, the current decoder's decoding capability is sufficient, and one Worker decoding thread can be reduced.

[0078] When -0.5 < △Dm(j) ≤ 0, the current decoder has sufficient decoding capability, and the number of decoding threads does not need to be adjusted, maintaining the current number of decoding threads.

[0079] When 0 < △Dm(j) ≤ 0.25, the current decoder's decoding capability is insufficient, and one additional Worker decoding thread needs to be added.

[0080] When 0.25 < △Dm(j) ≤ 0.5, the current decoder's decoding capability is insufficient, and two additional Worker decoding threads are needed.

[0081] When △Dm(j)>0.5, the current decoder's decoding capability is severely insufficient; four additional Worker decoding threads are needed.

[0082] It should be noted that the number of decoding threads mentioned in this rule is at least 1 and at most equal to the number of CPU cores. When the total number of decoding threads is 1, the number of decoding threads will not be adjusted even if ΔDm(j) < -0.5. Similarly, when the total number of decoding threads is equal to the number of CPU cores, the number of decoding threads will not be increased even if ΔDm(j) > 0.

[0083] Furthermore, in step S43, when increasing the number of decoding threads to improve the decoding capability of the current decoder, in order to fully utilize the parallel capability of multi-threaded decoding, it is necessary to adjust the video playback rate. The process is as follows: first, calculate the video playback rate PlayRate, PlayRate = 1 / (1 + number of threads increased), then calculate the time T for playing the video at PlayRate, T = (1 + number of threads increased) × GOP length, and after playing at PlayRate rate for time T, resume normal rate rendering and playback.

[0084] Furthermore, after improving the current decoder's decoding capability by increasing the number of decoding threads in step S43, an additional step is added: the average decoding delay gradient ΔDm(j+1) in the next preset time period after the decoding capability adjustment is calculated and compared with ΔDm(j) to determine whether the current decoder's decoding capability has reached its peak. When |ΔDm(j+1) – ΔDm(j)| < 0.05, a frame dropping strategy is adopted. This is because the decoding capability of the decoder and the number of Worker decoding threads are not linearly positively related. When the number of threads is configured to a certain level, the decoding capability will reach a peak, and further increasing the number of threads will not improve the decoding capability. In fact, the decoding capability may even be reduced due to CPU resource scheduling. Therefore, even if ΔDm(j+1) > 0, the number of Worker decoding threads will not be increased. Instead, a frame dropping strategy will be adopted, which means that all video frames before the most recent I-frame in the current time buffer queue will be discarded to alleviate the decoding pressure on the decoder and ensure the real-time performance of video playback.

[0085] Preferably, the preset time period can be a fixed sliding time window Twnd, where Twnd = number of threads × GOP length. The decoding capability of the current decoder is evaluated by calculating the average decoding delay gradient ΔDm(j) within a fixed sliding time window.

[0086] In another implementation of this embodiment, the decoding capability of the current decoder is evaluated by replacing the average decoding delay gradient ΔDm(j) with the decoding delay gradient Dm(w) between the last video frame and the first video frame in the fixed sliding time window Twnd. Substituting the decoding delay gradient Dm(w) into steps S43 and S44 can also evaluate the decoding capability of the decoder. The process of selecting different adjustment strategies to adjust the decoding capability of the decoder based on the evaluation results remains unchanged.

[0087] The present invention further provides a computer-readable storage medium storing a computer program for the above-described webassembly-based browser audio and video playback method.

[0088] This invention further provides a browser-based audio and video playback device based on WebAssembly. The playback device includes an acquisition module, a decapsulation module, a decoding module, and a playback module. The acquisition module acquires audio and video stream data to be played. The decapsulation module parses the acquired audio and video stream data in units of FLV tags to extract valid audio and video data corresponding to the FLV format audio and video stream. The decoding module calls a decoder corresponding to the encoding format of the audio and video to be played to decode the valid audio and video data. The decoding module includes a wasm decoding module compiled based on WebAssembly to ensure that H.265 encoded video can be played smoothly and efficiently in a browser without plugins. The playback module acquires the decoded original audio and video data, renders the original audio and video data, and loads it into a web-based audio and video player for playback. The various modules of the playback device use the browser-based audio and video playback method based on WebAssembly as described throughout the text for audio and video playback. The computer program for the browser-based audio and video playback method based on WebAssembly can be stored in the controller of the playback device or in the aforementioned computer-readable storage medium.

[0089] The present invention further provides a computer device for implementing the browser audio and video playback method described above. The computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. The computer device includes, but is not limited to, a memory / readable storage medium, a processor, and a network interface. The memory / readable storage medium stores program code for a webassembly-based browser audio and video playback method that can run on the processor. The processor executes the computer program to implement the steps of the webassembly-based browser audio and video playback method described above. The network interface is used to establish a communication connection with a server and / or other computer devices to obtain the audio and video data stream to be played.

[0090] In summary, the browser audio and video playback method based on WebAssembly provided by this invention has the following advantages compared with the prior art:

[0091] (1) During the audio and video playback process in the browser, the decoding capability of the current decoder is evaluated in real time based on the decoding delay gradient algorithm. Different adjustment strategies are selected to adjust the decoding capability of the decoder. The relationship between the decoding capability requirement and the number of decoding threads is dynamically adjusted. When the video decoding capability requirement is low, the number of decoding threads is automatically reduced to release CPU resources and reduce the waste of CPU resources.

[0092] (2) By decoding data in units of GOP and dynamically adjusting the relationship between the decoding capability and the number of decoding threads, the decoding capability of the decoder is greatly improved, especially the decoding capability of H.265 encoded video. This enables smooth playback on browsers with different configurations, including some machines with slightly lower performance, greatly improving the user experience.

[0093] (3) After increasing the number of decoding threads, determine whether the current decoder's decoding capability has reached its peak. When the decoding capability reaches its peak, adopt a frame dropping strategy to alleviate the decoding pressure on the decoder and ensure the real-time performance of video playback.

[0094] (4) After the first GOP raw video data after decoding enters the raw video queue, the rendering thread immediately calls the data to render and play it on the browser. There is no need to wait for a certain amount of data to be cached in the raw video queue before rendering and playing, which shortens the video first screen playback time and achieves the effect of video opening in seconds.

[0095] (5) When playing H.265 format video on the browser using the browser method of the present invention, no plugins need to be installed, and high-definition video can be played smoothly and smoothly. This avoids compatibility issues and limitations of decoding types caused by installing plugins and improves security performance.

[0096] As described above, similar technical solutions can be derived from the given solutions. Any simple modifications, equivalent changes, or alterations made to the above embodiments based on the technical essence of the invention, without departing from the scope of the invention, shall still fall within the scope of the invention.

Claims

1. A browser-based audio and video playback method based on WebAssembly, characterized in that: The method includes, S1. Obtain the audio and video stream to be played, and decapsulate it to obtain the valid audio and video data in the audio and video stream; S2. Start the decoding thread and call the decoder to decode the valid audio and video data; S3. Start the rendering thread to play the decoded audio and video; S4. Evaluate the current decoder's decoding capability in real time, and select different adjustment strategies to adjust the decoder's decoding capability based on the evaluation results; In step S4, the real-time evaluation of the current decoder's decoding capability is based on the decoding delay gradient algorithm to calculate the average decoding delay gradient within a preset time period, and the decoding capability of the decoder is evaluated by the numerical range of the average decoding delay gradient; the adjustment strategy includes adjusting the number of decoding threads, maintaining the current number of decoding threads, or a frame dropping strategy. The process of real-time evaluation of the current decoder's decoding capability in step S4 is as follows: S41. Calculate the decoding delay gradient Dm(i) between two adjacent video frames in the effective audio and video data. The decoding delay gradient Dm(i) is defined as the change obtained by subtracting the time interval between the decoding completion time of two adjacent video frames from the time interval between their decoding timestamps DTS. S42. Calculate the average value of the decoding delay gradient Dm(i) between all two adjacent video frames in the effective audio and video data within the preset time period to obtain the average decoding delay gradient △Dm(j). Evaluate the decoding capability of the current decoder by the average decoding delay gradient △Dm(j). S43. When △Dm(j)>0, the current decoder's decoding capability is insufficient. The decoding capability of the current decoder can be improved by increasing the number of decoding threads. Improving the decoding capability of the current decoder by increasing the number of decoding threads also includes adjusting the audio and video playback rate. The process is as follows: first, calculate the audio and video playback rate PlayRate, and then calculate the playback time T of the audio and video data played at PlayRate. The calculation formula is: PlayRate = 1 / (1 + number of threads increased), T = (1 + number of threads increased) × GOP length; S44. When △Dm(j)<0, the current decoder's decoding capability meets the playback requirements. Therefore, the number of decoding threads will not be adjusted, and / or the number of decoding threads will be reduced to free up CPU resources.

2. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: The formula for calculating the decoding delay gradient Dm(i) is as follows: Dm(i) = [(T i – T i-1 ) – (DTS i – DTS i-1 )] / (DTS i – DTS i-1 ); In the formula, i≥2； DTS i : Decoding timestamp of the i-th video frame; DTS i-1 : Decoding timestamp of the (i-1)th video frame; T i : The decoding completion time of the i-th video; T i-1 : The time for decoding the (i-1)th video.

3. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: After step S43, which improves the decoding capability of the current decoder by increasing the number of decoding threads, an additional step is added: calculate the average decoding delay gradient ΔDm(j+1) in the next preset time period after the decoding capability adjustment and compare it with ΔDm(j). When |ΔDm(j+1) – ΔDm(j)| < 0.05, the decoding capability of the current decoder reaches its peak, and a frame dropping strategy is adopted to ensure the real-time performance of audio and video playback.

4. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: The maximum number of decoding threads is equal to the number of CPU cores.

5. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: Step S1 includes, S11. Start the streaming thread, obtain the audio and video streams to be played, and call the decapsulation module to decapsulate the obtained audio and video streams; S12. Determine whether the audio and video stream is an FLV format audio and video stream and whether it contains audio and video data; S13. Identify the FlvTag in the data packet corresponding to the FLV format audio and video stream based on the FLV format; S14. Perform data parsing in units of FLV tags to extract the valid audio and video data corresponding to the FLV format audio and video streams.

6. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: Step S1 also includes compiling the decoder based on Webassembly to generate a wasm decoding module for browser use. The wasm decoding module is obtained by compiling a decoding library written in C / C++ using the Emscripten compiler.

7. The browser audio and video playback method based on WebAssembly as described in claim 1, characterized in that: The process of calling the decoder to decode the valid audio and video data in step S2 includes the following steps: S21. Initialize the decoding thread by calling the valid audio and video data described in step S1; S22. The main thread allocates the decapsulated audio and video data from step S1 to the decoding thread in units of GOPs, and decodes the audio and video data into the original audio and video data. S23. The decoding thread sends the original audio and video data back to the Javascript module and caches it in the original data queue for rendering and playback.

Images