Multimedia playback method, browser and electronic device

Abstract

The present embodiment provides a multimedia playback method, a browser, and an electronic device. The multimedia playback method includes obtaining an encapsulated video bitstream to be played in a browser, the browser integrating a hardware decoder and a web assembly Wasm module, calling the Wasm module to analyze the encapsulated video bitstream and obtain encoded video data in the encapsulated video bitstream, calling an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder for hard decoding to obtain initial video data, rendering the initial video data to obtain a video screen, and displaying the video screen. This can improve compatibility and reduce system resources occupied for video playback.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority from a Chinese patent application bearing application number 202211708270.9 and entitled "Multimedia Playback Method, Browser and Electronic Device," filed with the State Intellectual Property Office of the People's Republic of China on December 29, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of multimedia technology, and in particular to a multimedia playback method, a browser and an electronic device. [Background technology]

[0003] Since the amount of video data is usually large, in order to facilitate the transmission of the video bitstream, the video data in the video bitstream is usually encoded. Therefore, the video bitstream needs to be decoded in the process of being played by a browser. In the related art, the browser performs the decoding by executing a decoder such as a Video tag.

[0004] If the decoder implemented by the browser is a hardware decoder, it can only decode video formats supported by the hardware, resulting in low compatibility. If the decoder implemented by the browser is a software decoder, the amount of calculation required for software decoding is large, resulting in too much system resource being occupied for video playback. Therefore, how to improve compatibility and reduce system resource occupation for video playback has become a technical issue that needs to be resolved as soon as possible. Summary of the Invention

[0005] The present embodiment aims to provide a multimedia playback method, a browser, and an electronic device that can improve compatibility and reduce the system resources required for video playback. The specific technical solutions are as follows:

[0006] A first aspect of the present application is Obtaining an encapsulated video bitstream to be played in a browser, the browser having integrated therein a hardware decoder and a web assembly Wasm module; calling the Wasm module to parse the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; calling an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder for hard decoding to obtain initial video data; rendering the initial video data to obtain a video screen; and displaying said video screen. A multimedia playback method is provided.

[0007] In one possible embodiment, invoking an interface provided to the hardware decoder and inputting the encoded video data into the hardware decoder for hard decoding to obtain initial video data comprises: Invoking an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder running in a background process and hard-decoding it to obtain initial video data, the background process being a thread independent of a main thread for running the browser.

[0008] In one possible embodiment, rendering the initial video data to obtain a video screen comprises: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0009] In one possible embodiment, the initial video data is bitmap data, and the multimedia playback method comprises: further comprising storing the initial video data in a preset bitmap linked list; Rendering the initial video data to obtain a video screen includes: extracting bitmap data from the bitmap linked list; and rendering the extracted bitmap data to obtain a video screen.

[0010] In one possible embodiment, the step of calling an interface provided to the hardware decoder and inputting the encoded video data to the hardware decoder running in a background process to hard-decode the encoded video data to obtain initial video data includes: The method includes calling an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder executed in a background process by a post message PostMessage, and hard-decoding the encoded video data to obtain initial video data.

[0011] In one possible embodiment, invoking the Wasm module to parse the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream comprises: calling a demultiplexer DEMUX module in the Wasm module to parse the encapsulated video bitstream to obtain raw video data; storing the video raw data in a preset raw data linked list as encoded video data in the encapsulated video bitstream.

[0012] In one possible embodiment, rendering the initial video data to obtain a video screen comprises: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0013] In one possible embodiment, the multimedia playback method comprises: obtaining an encapsulated audio bitstream to be played on the browser; calling the Wasm module to parse the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream; decoding the encoded audio data to obtain initial audio data; rendering the initial audio data to obtain an audio signal; and playing the audio signal.

[0014] In one possible embodiment, the initial audio data is Pulse Code Modulation (PCM) data, and the multimedia playback method comprises: further comprising storing the initial audio data in a preset PCM linked list; Rendering the initial audio data to obtain an audio signal includes: extracting PCM data from the PCM link list; and rendering the extracted PCM data to obtain an audio signal.

[0015] In one possible embodiment, decoding the encoded audio data to obtain initial audio data comprises: calling the Wasm module to decode the encoded audio data to obtain initial audio data; or Invoking an interface provided to the hardware decoder, inputting the encoded audio data into the hardware decoder and hard-decoding it to obtain initial audio data.

[0016] A second aspect of the present application provides a browser including a main thread module, a hard decryption module, and a web assembly Wasm module; The main thread module obtains an encapsulated video bitstream to be played in a browser; the Wasm module parses the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; The hard decoding module calls an interface provided to a hardware decoder, inputs the encoded video data into the hardware decoder and hard decodes it to obtain initial video data, renders the initial video data to obtain a video screen, and displays the video screen.

[0017] In one possible embodiment, the hard decoding module calls an interface provided to a hardware decoder, and inputting the encoded video data into the hardware decoder and hard decoding it to obtain initial video data, the hard decoding module comprising: Invoking an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder running in a background process and hard-decoding it to obtain initial video data, the background process being a thread independent of a main thread for running the browser.

[0018] In one possible embodiment, the hard decoding module may further include: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0019] In one possible embodiment, the initial video data is bitmap data, and the hard decoding module stores the initial video data in a preset bitmap linked list; The hard decoding module renders the initial video data to obtain a video screen, extracting bitmap data from the bitmap linked list; and rendering the extracted bitmap data to obtain a video screen.

[0020] In one possible embodiment, the hard decoding module may further include: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0021] In one possible embodiment, the hard decoding module calls an interface provided to a hardware decoder, and inputs the encoded video data to the hardware decoder running in a background process to hard-decode the encoded video data to obtain initial video data, The method includes calling an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder executed in a background process by a post message PostMessage, and hard-decoding the encoded video data to obtain initial video data.

[0022] In one possible embodiment, the Wasm module parses the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream, the parsing comprising: The method includes invoking a demultiplexer DEMUX module in the Wasm module to parse the encapsulated video bitstream to obtain raw video data, and storing the raw video data in a preset raw data linked list as encoded video data in the encapsulated video bitstream.

[0023] In one possible embodiment, the main thread module further includes: obtaining an encapsulated audio bitstream to be played in the browser; The Wasm module further parses the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream, decodes the encoded audio data to obtain initial audio data, renders the initial audio data to obtain an audio signal, and plays the audio signal.

[0024] In one possible embodiment, the initial audio data is Pulse Code Modulation (PCM) data, and the Wasm module further stores the initial audio data in a preset PCM linked list; The Wasm module renders the initial audio data to obtain an audio signal, extracting PCM data from the PCM link list; and rendering the extracted PCM data to obtain an audio signal.

[0025] In one possible embodiment, the main thread module further includes: obtaining an encapsulated audio bitstream to be played in the browser; the Wasm module further parses the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream; The hard decoding module further invokes an interface provided to a hardware decoder, inputs the encoded audio data into the hardware decoder and hard-decodes it to obtain initial audio data, renders the initial audio data to obtain an audio signal, and plays back the audio signal.

[0026] A third aspect of the present application is a memory storing a computer program; a processor that, when executing a program stored in the memory, implements the method according to any one of the first aspects; Electronic equipment is also provided.

[0027] A fourth aspect of the present application further provides a computer-readable recording medium having a computer program recorded thereon, the computer program implementing the method according to any one of the first aspects when executed by a processor.

[0028] Beneficial Effects of the Present Embodiments The multimedia playback method, browser, and electronic device provided in the present application parse an encapsulated video bitstream using a Wasm module and input the parsed encoded video data to the hardware decoder via an interface provided to the hardware decoder. The hardware decoder then decodes the decapsulated encoded video data, thereby obtaining initial video data and displaying the video screen through rendering. Because the Wasm module is developed based on a high-level programming language, it can realize more parsing methods than native hardware methods. Therefore, it can parse various encapsulated video bitstreams with different encapsulations. The hardware decoder does not need to consider the encapsulation format of the video bitstream and only needs to decode the encoded video data. Therefore, the multimedia playback method according to the present application can be applied to various encapsulated video bitstreams with different encapsulations, thereby effectively improving compatibility. Furthermore, because the multimedia playback method according to the present application uses a hardware decoder for decoding, it effectively reduces the amount of calculation required for decoding, thereby effectively reducing the system resources required for video playback. In other words, the multimedia playback method according to the present application improves compatibility and reduces the system resources required for video playback.

[0029] Of course, it is not necessary for all of the above advantages to be achieved simultaneously in order to practice any product or method herein.

[0030] The drawings described herein are intended to provide a further understanding of the present application and are intended to constitute a part of the present application. The exemplary embodiments and the description thereof are intended to be illustrative, not limiting, of the present application. [Brief explanation of the drawings]

[0031] [Figure 1a]FIG. 1a is a schematic diagram of an application scenario of the multimedia playback method according to the present application. [Figure 1b] FIG. 1b is a schematic diagram of the video playback method provided in the browser according to the present application. [Figure 1c] FIG. 1c is a schematic diagram of another video playback method provided in the browser according to the present application. [Figure 2] FIG. 2 is a flow diagram of the multimedia playback method according to the present application. [Figure 3] FIG. 3 is a schematic diagram of the system framework of the browser according to the present application. [Figure 4] FIG. 4 is a schematic diagram showing the configuration of an electronic device according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Of course, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present application, all other embodiments that a person skilled in the art can obtain based on the present application also fall within the scope of protection of the present application.

[0033] In order to more clearly explain the multimedia playback method according to the present application, one possible application scenario of the multimedia playback method according to the present application will be exemplarily described below. It is understood that the following example is only one possible application scenario of the multimedia playback method according to the present application, and in other possible embodiments, the multimedia playback method according to the present application can also be applied to other possible application scenarios, and is not limited to the following example.

[0034] As shown in FIG. 1a, a user remotely accesses an image collection device through a browser and watches a video collected by the image collection device. The image collection device collects images of a specific area and performs pet detection on the images to determine whether a pet is present in the images. The image collection device encodes the collected initial video data to obtain coded video data, encapsulates the coded video data and the detection result of the pet detection on the initial video data to obtain an encapsulated video bitstream, and sends the encapsulated video bitstream to the browser. Accordingly, when the browser plays the video, it needs to sequentially perform decapsulation (hereinafter also referred to as parsing), decoding, and rendering.

[0035] The browser offers two video playback methods:

[0036] About Method 1 As shown in Figure 1b, the browser inputs the encapsulated video bitstream into a Video tag provided in HTML5 (HyperText Markup Language 5), parses the encapsulated video bitstream using the Video tag to obtain encoded video data, decodes the encoded video data using the Video tag to obtain initial video data, and renders the initial video data using the Video tag, thereby allowing the browser to display the video screen.

[0037] About Method 2 As shown in Figure 1c, a developer first develops and obtains a functional module with video playback capabilities using C, C++, or other high-level programming languages, then uses Emscripten (a compilation tool) to compile the functional module into a Web Assembly (Wasm) module, which is then integrated into a browser. The browser inputs the encapsulated video bitstream into the Wasm module, which parses the encapsulated video bitstream to obtain encoded video data, decodes the encoded video data to obtain initial video data, and then renders the initial video data using the Wasm module. This allows the browser to display the video screen.

[0038] In contrast to Method 1, the Video tag realizes video playback based on the hardware's native method. Therefore, it can only analyze information supported by the hardware, such as coded video data in a specific format, from the encapsulated video bitstream. It cannot analyze information not supported by the hardware, such as coded video data in a specific format or the above-mentioned detection results, from the encapsulated video bitstream. As a result, Method 1 can only play a portion of the encapsulated video bitstream, but cannot acquire or display the detection results. In other words, Method 1 has low compatibility and does not support displaying the detection results.

[0039] Compared to Method 2, the Wasm module uses soft decoding and does not fully rely on native hardware methods to play videos. It therefore needs to call the CPU to perform some operations, which consumes a lot of system resources.

[0040] Based on this, the present application provides a multimedia playback method, which is applicable to any electronic device with multimedia playback function, including but not limited to computers, smartphones, tablets, etc. The multimedia playback method provided in the present application, as shown in Figure 2, includes the following steps:

[0041] In S201, an encapsulated video bitstream to be played back in a browser is obtained. The browser has a hardware decoder and a Wasm module integrated.

[0042] In S202, the Wasm module is called to parse the encapsulated video bitstream to obtain the encoded video data in the encapsulated video bitstream.

[0043] In S203, an interface provided to the hardware decoder is called, and the coded video data is input to the hardware decoder and hard-decoded to obtain the initial video data.

[0044] In S204, the initial video data is rendered to obtain a video screen.

[0045] In S205, the video screen is displayed.

[0046] According to this embodiment, an encapsulated video bitstream is parsed using a Wasm module, and the resulting encoded video data is input to the hardware decoder via an interface provided to the hardware decoder. The hardware decoder then decodes the decapsulated encoded video data, thereby obtaining initial video data and displaying the video screen through rendering. Because the Wasm module is developed based on a high-level programming language, it can implement more parsing methods than native hardware methods. Therefore, it can parse various encapsulated video bitstreams with different encapsulations. The hardware decoder does not need to consider the encapsulation format of the video bitstream and only needs to decode the encoded video data. Therefore, the multimedia playback method according to the present application can be applied to various encapsulated video bitstreams with different encapsulations, thereby effectively improving compatibility. Furthermore, because the multimedia playback method according to the present application uses a hardware decoder for decoding, it effectively reduces the amount of calculation required for decoding, thereby effectively reducing the system resources required for video playback. In other words, the multimedia playback method according to the present application not only improves compatibility but also reduces the system resources required for video playback.

[0047] Below, the above steps S201 to S205 will be explained one by one. In S201, various hardware decoders may be integrated depending on the type of browser. A hardware decoder in this specification refers to a decoder that supports hard decoding. The Wasm module should at least have the function of extracting encoded video data from the encapsulated video bitstream through analysis. Furthermore, for data (hereinafter referred to as proprietary data) formed by an image capture device (e.g., a network video camera, a smartphone, etc.) or an image storage device (e.g., a network video recorder, a hard disk recorder, etc.) processing the video data (e.g., target detection, adding intelligence information, etc.) that is further included in the encapsulated video bitstream, the Wasm module may also have the function of extracting proprietary data from the encapsulated video bitstream through analysis. For example, in a scenario in which the detection results are applied, the Wasm module may be configured with a proprietary protocol corresponding to the proprietary data, and the Wasm extracts the proprietary data from the encapsulated video bitstream through analysis according to the analysis method defined by the proprietary protocol.

[0048] In S202, the Wasm module parses the encapsulated video bitstream to obtain at least the encoded video data, and then parses the encapsulated video bitstream using a proprietary protocol supported by the Wasm module to obtain proprietary data such as the detection result. This solves the problem of not being able to display the detection result in Method 1. Depending on actual requirements, Wasm can support encapsulated video bitstreams in various encapsulation formats. Preferably, Wasm supports all common encapsulation formats, such as PS (Program Stream), RTP (Real-time Transport Protocol), FLV (Flash Video, a streaming media video format), MP4 (an audiovisual information compression coding standard), and HIK (an encapsulation format).

[0049] In step S203, the interface provided to the hardware decoder is an interface that can directly call existing media components and codecs in the browser. For example, in the Chrome browser, the interface is WebCodecsAPI (browser codec interface). Through this interface, video data can be input to the hardware decoder and hard-decoded by the hardware decoder to obtain the initial video data.

[0050] The data format of the initial video data may vary depending on the application scenario. For example, in one possible embodiment, the initial video data is vector graphic data, and in another possible embodiment, the initial video data is bitmap data. If the initial video data is bitmap data, the initial video data obtained by decoding may be stored in a preset bitmap linked list. Then, when rendering the initial video data, the bitmap data is extracted from the bitmap linked list, and the extracted bitmap data is rendered to obtain a video screen.

[0051] A bitmap linked list is a linked list. A linked list contains multiple entries. Each entry is used to store data and stores a pointer to the next entry. Taking a bitmap linked list as an example, each entry in the bitmap linked list is used to store bitmap data and stores a pointer to the next entry.

[0052] When storing bitmap data in the bitmap linked list, the bitmap data is stored in the current write entry, and the entry pointed to by the pointer in the current entry becomes the new current write entry. The current write entry is initially the initial bitmap entry. The initial bitmap entry can be any entry in the bitmap linked list, but to make full use of the storage space, the initial bitmap entry should be as close to the first entry in the bitmap linked list as possible.

[0053] When bitmap data is read from the bitmap linked list, the bitmap data stored in the current read entry is read from the bitmap linked list, and the entry pointed to by the pointer in the current storage entry becomes the new current storage entry. The current storage entry is initially the initial bitmap entry.

[0054] To prevent the renderer for rendering the initial video data from occupying too much cache space by caching the bitmap data due to the renderer reading too much bitmap data at once, the renderer can read one bitmap data from the bitmap linked list every preset period. The preset period depends on the frame rate of the video bitstream and should be equal to or less than the interval between two consecutive frames. For example, if the frame rate of the video bitstream is 20 frames per second, i.e., one video frame is played every 50 ms, the preset period should be equal to or less than 50 ms, preferably close to 50 ms.

[0055] A hardware decoder is a decoder integrated into a browser, and is understood to decode and obtain initial video data by executing the decoder in the browser's main thread. However, when a hardware decoder is executed in the browser's main thread, the main thread's processing power is limited, and the main thread may not be able to respond to user operations in a timely manner due to the execution of the hardware decoder, which may cause the browser's user interface to stutter. For example, when a user is watching a video in a browser and clicks a toolbar button on the browser to expand the toolbar, because the main thread is executing the hardware decoder, the main thread cannot respond to the user's click in a timely manner, and it takes a long time for the browser to expand the toolbar.

[0056] In one possible embodiment, an interface provided to the hardware decoder is called to input the encoded video data to the hardware decoder running in a background process for hard decoding to obtain the initial video data. By running the hardware decoder in a background process to perform hard decoding, the stuttering of the user interface caused by the main thread running the hardware decoder can be avoided and the browser response rate can be improved.

[0057] The background process is any thread that is independent of the main thread and can execute a hardware decoder. For example, the background process is a WebWorker process. The WebWorker process exchanges data with the main thread using PostMessage (a function for posting a message). In one possible embodiment, after completing the decoding of the encoded video data, the WebWorker process sends the initial video data obtained by the decoding to the main thread using PostMessage for rendering. In another example, the background process of the present application may be a shared worker process, and is not limited to the present application.

[0058] In another possible embodiment, after the WebWorker process completes decoding of the encoded video data, it renders the initial video data obtained by decoding by calling the interface provided to the browser, and sends the video screen obtained by rendering to the main thread by PostMessage for display.

[0059] In this embodiment, both decoding and rendering are performed in background processes, further reducing the amount of computation required to run on the main thread, thereby further improving the browser response rate and reducing the likelihood of stuttering.

[0060] In S204, rendering may be performed in a background process or on the main thread, as described above in S203.

[0061] In S205, only the video screen may be displayed, or the video screen and other information other than the video screen, such as the detection result, may be displayed simultaneously.

[0062] To more clearly explain the multimedia playback method according to the present application, the following will be described exemplarily with reference to the system framework of a browser. Referring to Figure 3, Figure 3 illustrates the system framework of a browser according to the present application, which includes a web layer and a Wasm layer. The web layer includes a main thread module and a WebWorker module, and the Wasm layer includes a Wasm module. The main thread module includes an API interface, instantiation, logic control, information exchange, bitstream reception, module management, and worker management. The API interface connects to the WebWorker module. The instantiation instantiates the main thread program. The logic control controls the calculation logic of the main thread. The information exchange realizes data exchange between modules. The bitstream reception realizes data transfer between modules. The module management manages each module. The worker management manages the WebWorker module.

[0063] The main thread module exchanges information with the user interface, receives video data sent from the device, and manages each module. The Wasm module includes a Demux module, a Decoder module, and a Render module.

[0064] The Demux module parses the encapsulated audio-video bitstream to obtain encoded video data, encoded audio data, and proprietary data from the encapsulated audio-video bitstream. The Decoder module decodes the encoded audio data to obtain initial audio data. The Render module renders the initial audio data to obtain an audio signal.

[0065] The WebWorker module includes a hardware decoder (video hard decoding) and a hard decoding display module. The hardware decoder hard decodes the encoded video data to obtain initial video data. The hard decoding display module renders the initial video data to obtain a video screen.

[0066] In this example, the video playback flow is as follows:

[0067] In step 1, the main thread module obtains the encapsulated video bitstream to be played, that is, realizes the above step S201.

[0068] In step 2, the main thread exchanges data with the Wasm module to send the encapsulated video bitstream to the Wasm module.

[0069] In step 3, the Wasm module parses the encapsulated video bitstream using the Demux module to obtain raw video data. The raw video data corresponds to the encoded video data. This step implements step S202. The Wasm module then sends the encoded video data to the WebWorker module using the PostMessage message transmission method.

[0070] In step 4, the WebWorker module uses WebCodecs to call the hardware decoder to hard-decode the raw video data to obtain the initial video data, that is, to realize the above step S203.

[0071] In step 5, the WebWorker module calls the browser interface through the hard decoding display module to render the initial video data, obtain the video screen, and display the video screen, that is, realizes the above steps S204 and S205.

[0072] Here, the video raw data is stored in a raw data link list preset as an encapsulated video bitstream. The raw data link list is a linked list, and its format is the same as the bitmap linked list, except that the data stored in each entry in the raw data link list is video raw data, while the data stored in each entry in the bitmap linked list is bitmap data.

[0073] When storing video raw data in the raw data linked list, the video raw data is stored in the current write entry, and the entry pointed to by the pointer in the current entry becomes the new current write entry. The current write entry is initially the initial raw data entry. The initial raw data entry can be any entry in the raw data linked list, but to make full use of the storage space, the initial raw data entry should be as close to the first entry in the raw data linked list as possible.

[0074] When reading raw data from the raw data link list, the raw data stored in the current read entry is read from the raw data link list, and the entry pointed to by the pointer in the current storage entry becomes the new current storage entry. The current storage entry is initially the initial raw data entry.

[0075] To prevent a hardware decoder for decoding raw video data from occupying an excessive cache space due to caching the raw video data caused by reading excessive raw video data at once, the hardware decoder can call back one raw video data from the raw video link list every preset period. The preset period depends on the number of raw video data to be decoded per unit time of the hardware decoder. For example, if a video bitstream decodes 20 raw video data per second, that is, one raw video data per 50 ms, the preset period should be less than 50 ms and preferably close to 50 ms.

[0076] The system framework shown in Figure 3 can also play audio, not just video. The audio playback flow is as follows:

[0077] In step 6, the main thread module obtains the encapsulated audio bitstream to be played.

[0078] In step 7, the main thread exchanges data with the Wasm module to send the encapsulated audio bitstream to the Wasm module.

[0079] In step 8, the Wasm module parses the encapsulated audio bitstream through the Demux module to obtain raw audio data (hereinafter referred to as encoded audio data).

[0080] In step 9, the Wasm module decodes the encoded audio data using the Decoder module to obtain the initial audio data.

[0081] In step 10, the Wasm module renders the initial audio data using the Render module to obtain an audio signal and plays the audio signal.

[0082] Here, the initial audio data may be PCM (Pulse Code Modulation) data. The initial video data obtained by decoding may be stored in a preset PCM linked list. When rendering the initial audio data, the PCM data is extracted from the PCM linked list and the extracted PCM data is rendered to obtain an audio signal.

[0083] The PCM linked list is a linked list whose format is the same as the bitmap linked list, except that the data stored in each entry in the PCM linked list is PCM data, while the data stored in each entry in the bitmap linked list is bitmap data.

[0084] When storing PCM data in the PCM linked list, the PCM data is stored in the current write entry, and the entry pointed to by the pointer in the current entry becomes the new current write entry. The current write entry is initially the initial PCM entry. The initial PCM entry can be any entry in the PCM linked list, but to make full use of storage space, the initial PCM entry should be as close to the first entry in the PCM linked list as possible.

[0085] When reading PCM data from the PCM link list, the PCM data stored in the current read entry is read from the PCM link list, and the entry pointed to by the pointer in the current storage entry becomes the new current storage entry. The current storage entry is initially the initial PCM entry.

[0086] To prevent the renderer for rendering the initial audio data from occupying too much cache space by reading too much PCM data at once and caching the PCM data, the renderer can read one PCM data from the PCM link list for each preset period. The preset period depends on the frame rate of the audio bitstream. For example, if the frame rate of the audio bitstream is 20 frames per second, i.e., if one audio frame is played every 50 ms, the preset period should be less than 50 ms and preferably close to 50 ms.

[0087] It is understood that steps 6 to 10 are only the audio playback method of the present application, and in other possible embodiments, the audio data can be decoded and rendered by the WebWorker module. That is, after step 8, the Wasm module sends the encoded audio data to the WebWorker module using the PostMeggage message transmission method, the WebWorker module decodes the encoded audio data to obtain initial audio data, and the WebWorker module renders the initial audio data to obtain and play the audio signal.

[0088] Corresponding to the multimedia playback method of the present application, the present application provides: providing a browser including a main thread module, a hard decryption module, and a web assembly Wasm module; The main thread module obtains an encapsulated video bitstream to be played in a browser; the Wasm module parses the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; The hard decoding module calls an interface provided to a hardware decoder, inputs the encoded video data into the hardware decoder and hard decodes it to obtain initial video data, renders the initial video data to obtain a video screen, and displays the video screen.

[0089] In one possible embodiment, the hard decoding module may further include: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0090] In one possible embodiment, the initial video data is bitmap data, and the hard decoding module stores the initial video data in a preset bitmap linked list; The hard decoding module renders the initial video data to obtain a video screen, extracting bitmap data from the bitmap linked list; and rendering the extracted bitmap data to obtain a video screen.

[0091] In one possible embodiment, the hard decoding module may further include: The main thread or the background process invokes an interface provided to the browser to render the initial video data and obtain a video screen.

[0092] In one possible embodiment, the hard decoding module calls an interface provided to a hardware decoder, and inputs the encoded video data to the hardware decoder running in a background process to hard-decode the encoded video data to obtain initial video data, The method includes calling an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder executed in a background process by a post message PostMessage, and hard-decoding the encoded video data to obtain initial video data.

[0093] In one possible embodiment, the Wasm module parses the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream, the parsing comprising: The method includes invoking a demultiplexer DEMUX module in the Wasm module to parse the encapsulated video bitstream to obtain raw video data, and storing the raw video data in a preset raw data linked list as encoded video data in the encapsulated video bitstream.

[0094] In one possible embodiment, the main thread module further includes: obtaining an encapsulated audio bitstream to be played in the browser; The Wasm module further parses the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream, decodes the encoded audio data to obtain initial audio data, renders the initial audio data to obtain an audio signal, and plays the audio signal.

[0095] In one possible embodiment, the initial audio data is Pulse Code Modulation (PCM) data, and the Wasm module further stores the initial audio data in a preset PCM linked list; The Wasm module renders the initial audio data to obtain an audio signal, extracting PCM data from the PCM link list; and rendering the extracted PCM data to obtain an audio signal.

[0096] In one possible embodiment, the main thread module further includes: obtaining an encapsulated audio bitstream to be played in the browser; the Wasm module further parses the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream; The hard decoding module further invokes an interface provided to a hardware decoder, inputs the encoded audio data into the hardware decoder and hard-decodes it to obtain initial audio data, renders the initial audio data to obtain an audio signal, and plays back the audio signal.

[0097] As shown in FIG. 4, the embodiment of the present application: a memory 401 storing a computer program; The present invention further provides an electronic device including a processor 402 that, when executing a program stored in the memory 401, realizes the following steps, wherein the steps include: Obtaining an encapsulated video bitstream to be played in a browser, the browser having integrated therein a hardware decoder and a web assembly Wasm module; calling the Wasm module to parse the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; calling an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder for hard decoding to obtain initial video data; rendering the initial video data to obtain a video screen; and displaying said video screen.

[0098] The electronic device may further include a communication bus and / or a communication interface, and the processor 402, the communication interface, and the memory 401 communicate with each other via the communication bus.

[0099] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus is divided into an address bus, a data bus, a control bus, etc. In the figure, for convenience of illustration, it is shown with one thick line, but this does not mean that there is only one bus or one type of bus.

[0100] The communication interface is used for communication between the electronic device and other devices.

[0101] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one magnetic disk memory. Preferably, the memory may also be a storage device remote from the at least one processor.

[0102] The processors described above may be general-purpose processors including a central processing unit (CPU), a network processor (NP), etc., or may be general-purpose processors including a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware assembly.

[0103] In a further embodiment of the present application, there is further provided a computer-readable recording medium having a computer program recorded thereon, the computer program implementing the steps of any of the above multimedia playback methods when executed by a processor.

[0104] In a further embodiment of the present application, there is further provided a computer program product including commands, which, when executed on a computer, cause the computer to implement any of the multimedia playback methods in the above embodiments.

[0105] The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer commands. When the computer program commands are loaded into a computer and executed, the process or function described in the embodiments of the present application is realized in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other rewritable device. The computer commands may be stored on a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer commands may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, radio, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can read or write, or may be a data storage device, including a server, data center, etc., that integrates one or more available media. The available media may be magnetic media (eg, floppy disks, hard disks, tapes), optical media (eg, DVDs), solid state drives, etc.

[0106] In this context, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another and do not require or imply any actual relationship or order between those entities or operations. Furthermore, the terms "comprise," "include," "includes," or any other variant thereof, mean a non-exclusive "comprise." Thus, a process, method, article, or device comprising a set of elements does not include only those elements, but also other elements not expressly listed, or elements inherent in such process, method, article, or device. Absent further limitations, an element limited by a phrase "comprising..." does not exclude the presence of other identical elements in a process, method, article, or device that includes the recited element.

[0107] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments may be cross-referenced. The emphasis in each embodiment is on the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so they are only briefly described. For related points, please refer to the description of the method embodiments.

[0108] The above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. Obtaining an encapsulated video bitstream to be played in a browser, the browser having integrated therein a hardware decoder and a web assembly Wasm module; calling the Wasm module to parse the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; calling an interface provided to the hardware decoder, inputting the encoded video data into the hardware decoder for hard decoding to obtain initial video data; rendering the initial video data to obtain a video screen; and displaying the video screen. Multimedia playback methods.

2. Invoking an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder and hard-decoding the encoded video data to obtain initial video data, calling an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder executed in a background process and hard-decoding the data to obtain initial video data, wherein the background process is a thread independent from a main thread for executing the browser. The multimedia playback method of claim 1.

3. Rendering the initial video data to obtain a video screen includes: The method further comprises: invoking an interface provided to the browser by the main thread or the background process to render the initial video data and obtain a video screen. The multimedia playback method of claim 2.

4. the initial video data is bitmap data, and the multimedia playback method includes: further comprising storing the initial video data in a preset bitmap linked list; Rendering the initial video data to obtain a video screen includes: extracting bitmap data from the bitmap linked list; and rendering the extracted bitmap data to obtain a video screen. The multimedia playback method of claim 2.

5. Invoking an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder running in a background process and hard-decoding the encoded video data to obtain initial video data, and calling an interface provided to the hardware decoder, inputting the encoded video data to the hardware decoder executed in a background process by a post message PostMessage, and hard-decoding the encoded video data to obtain initial video data. A multimedia playback method according to any one of claims 2 to 4.

6. Invoking the Wasm module to parse the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream includes: Invoking a demultiplexer DEMUX module in the Wasm module to parse the encapsulated video bitstream to obtain raw video data; storing the video raw data in a preset raw data linked list as encoded video data in the encapsulated video bitstream. The multimedia playback method of claim 1.

7. obtaining an encapsulated audio bitstream to be played on the browser; calling the Wasm module to parse the encapsulated audio bitstream to obtain encoded audio data in the encapsulated audio bitstream; decoding the encoded audio data to obtain initial audio data; rendering the initial audio data to obtain an audio signal; and playing the audio signal. The multimedia playback method of claim 1.

8. The initial audio data is pulse coded modulation (PCM) data, and the multimedia playback method includes: further comprising storing the initial audio data in a preset PCM linked list; Rendering the initial audio data to obtain an audio signal includes: extracting PCM data from the PCM linked list; and rendering the extracted PCM data to obtain an audio signal.

8. The multimedia playback method of claim 7.

9. Decoding the encoded audio data to obtain initial audio data includes: calling the Wasm module to decode the encoded audio data to obtain initial audio data; or calling an interface provided to the hardware decoder, inputting the encoded audio data into the hardware decoder and hard-decoding it to obtain initial audio data; 8. The multimedia playback method of claim 7.

10. a main thread module, a hard decryption module, and a web assembly Wasm module; The main thread module obtains an encapsulated video bitstream to be played in a browser; the Wasm module parses the encapsulated video bitstream to obtain encoded video data in the encapsulated video bitstream; the hard decoding module calls an interface provided to a hardware decoder, inputs the encoded video data into the hardware decoder and hard-decodes it to obtain initial video data, renders the initial video data to obtain a video screen, and displays the video screen. browser.

Images