Audio and video decoding method and system of browser

Abstract

The invention provides an audio and video decoding method and system of a browser, and the method comprises the steps: when an HTTP server pulls code stream data from a code stream database and sends the code stream data to a WebSocket server, a client receives the code stream data sent by the WebSocket server; the client sends the code stream data to the WASM module; the WASM module transcodes the code stream data to obtain transcoded code stream data; after the WASM module is loaded on the browser, code stream data obtained after transcoding is played; according to the method, the audio and video coding format of the code stream format which is not supported by the browser is transcoded into the proper coding format, and then the audio and video coding format is sent to the browser for playing, so that the browser does not need to be upgraded, and the adaptability of the browser is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an audio and video decoding method and system for a browser. Background Art

[0002] With the development of the Internet, audio and video have become an important part of network content. Users expect to be able to watch audio and video content directly in the browser without relying on external players or applications. However, the current mainstream browsers have limited support for bitstream formats. Some browsers only support the H.264 format, some have limited support for the H.265 format, or only support a few fixed audio formats, such as only supporting AAC audio.

[0003] When the user's audio uses non-AAC audio, the browser cannot play it correctly and subjectively no sound can be heard. Mainstream browsers are developed by large companies, and it is difficult for ordinary users to ask large companies to provide patches or software upgrades. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a browser audio and video decoding method and system, which can transcode the audio and video encoding format of the bitstream format that the browser does not support, and then give it to the browser for playback after transcoding into a suitable encoding format without upgrading the browser itself, thereby improving the adaptability of the browser.

[0005] In a first aspect, an embodiment of the present invention provides an audio and video decoding method for a browser, the method comprising:

[0006] When the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server, the client receives the code stream data sent by the WebSocket server;

[0007] The client sends the code stream data to the WASM module;

[0008] The WASM module transcodes the code stream data to obtain transcoded code stream data;

[0009] After the WASM module is loaded on the browser, the transcoded stream data is played.

[0010] Furthermore, the WASM module transcodes the code stream data to obtain transcoded code stream data, including:

[0011] Determine whether the code stream data is in a code stream format supported by the browser;

[0012] If not, the code stream data is decoded to obtain original audio PCM data;

[0013] Encode the original audio PCM data to obtain an ES stream;

[0014] The ES stream consists of multiple slices.

[0015] Further, determining whether the code stream data is in a code stream format supported by the browser further includes:

[0016] If yes, the code stream data is transparently transmitted without being processed.

[0017] Furthermore, the WASM module transcodes the code stream data to obtain transcoded code stream data, including:

[0018] When the code stream data is video data in H.265 video format, the WASM module converts the video data in H.265 video format into video data in H.264 video format.

[0019] Further, after the WASM module is loaded on the browser, the transcoded stream data is played, including:

[0020] After the WASM module is loaded on the browser, the transcoded stream data is pushed to the MSE API;

[0021] The transcoded code stream data is played through the MSE API.

[0022] Furthermore, the code stream data includes audio data, video data and audio and video data.

[0023] In a second aspect, an embodiment of the present invention provides an audio and video decoding system for a browser, the system comprising:

[0024] The code stream data receiving module is used for, when the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server, the client receives the code stream data sent by the WebSocket server;

[0025] A code stream data sending module, used for the client to send the code stream data to the WASM module;

[0026] A transcoding module, used for the WASM module to transcode the code stream data to obtain transcoded code stream data;

[0027] The loading module is used to play the transcoded stream data after loading the WASM module on the browser.

[0028] Furthermore, the transcoding module is specifically used for:

[0029] Determine whether the code stream data is in a code stream format supported by the browser;

[0030] If not, the code stream data is decoded to obtain original audio PCM data;

[0031] Encode the original audio PCM data to obtain an ES stream;

[0032] The ES stream consists of multiple slices.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method described above when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method as described above.

[0035] The embodiment of the present invention provides a method and system for audio and video decoding of a browser, including: when an HTTP server pulls code stream data from a code stream database and sends the code stream data to a WebSocket server, a client receives the code stream data sent by the WebSocket server; the client sends the code stream data to a WASM module; the WASM module transcodes the code stream data to obtain transcoded code stream data; after the WASM module is loaded on the browser, the transcoded code stream data is played; the audio and video encoding format of the code stream format not supported by the browser is transcoded into a suitable encoding format and then given to the browser for playback without upgrading the browser itself, thereby improving the adaptability of the browser.

[0036] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of the audio and video decoding method of a browser provided in the first embodiment of the present invention;

[0040] Figure 2 A schematic diagram of an audio and video decoding system of a browser provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Video playback has strict requirements on the format. There are two types of MP4 codecs: H.264, also known as MPEG-4 AVC (Advanced Video Coding); and H.265, also known as HEVC (High Efficiency Video Coding). Chrome browser has limited support for H.265 MP4 format, and there may be a situation where there is only sound but no picture during playback. In addition, if the video is not correctly transcoded or stored, and the server is not configured with the correct access policy and CORS support, the video may not be played on the Web side (Chrome browser).

[0043] The audio formats that Chrome browser does not support include / ogg;codecs=opus. The audio formats supported by Chrome browser include MP3, AAC, WAV and WebM. For unsupported audio formats, this application transcodes the audio format that the browser does not support into an audio format that the browser can support, such as converting the Opus format into the AAC format, and then sends it to the browser for playback.

[0044] G.711 is an audio compression standard defined by ITU-T (International Telecommunication Union Telecommunication Standardization Sector), mainly used for voice signal coding in telephone communications. In some video conferencing systems that require high-quality voice transmission, G.711 may still be one of the preferred coding standards.

[0045] However, current browsers do not support audio formats such as G.711U, G.711A, and G.729, so this application uses format conversion to enable audio and video to be played normally in browsers that do not originally support this format.

[0046] To facilitate understanding of this embodiment, the embodiment of the present invention is described in detail below.

[0047] Embodiment 1:

[0048] Figure 1 This is a flow chart of the audio and video decoding method of the browser provided in the first embodiment of the present invention.

[0049] Reference Figure 1 , the method comprises the following steps:

[0050] Step S101, when the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server, the client receives the code stream data sent by the WebSocket server;

[0051] Step S102, the client sends the code stream data to the WASM module;

[0052] Specifically, the client creates a WebSocket connection and sends a request to the server. After the client listens to the WebSocket response, it interacts with the server. The specific implementation process is: after pulling the code stream data from a code stream database, the code stream data is provided to the HTTP server, and the HTTP server pushes the code stream data to the WebSocket server. After the client listens to the code stream data, it transmits the code stream data to the WASM module.

[0053] Step S103, the WASM module transcodes the code stream data to obtain transcoded code stream data;

[0054] Specifically, a WASM module is established, and the WASM module is used to process the decoding and encoding (ie, transcoding) of video or audio.

[0055] C++ provides a rich library and tools to handle audio and video encoding and decoding. Among them, the code stream format is transcoded. Call the methods in the C++ library to convert the code stream encoding format, including audio encoding and decoding, video encoding and decoding, streaming media processing, etc. The code calls the methods in the method library to convert the H.265 video format to the H.264 video format, decode the G.711U, G.711A, G.729 and other audio formats that are not supported by the browser, and then encode them after decoding into the AAC audio format, so that the browser can play the sound normally. You can also use Emscripten or other tools to compile C++ code into WASM.

[0056] Introduce the compiled WASM file into the code, and the browser will have a corresponding loading module to load the WASM file. The introduced WASM file is equivalent to a supplement and enhancement to the browser decoding format. First, initialize the WASM module, call the initialization method of the WASM module, and prepare the decoding capability; secondly, call the decoding method to decode the code stream data received from the server backend, and compile the decoding. For example, when there is video stream data on the web page, the video is decoded according to its decoding parameters, and then repackaged and encoded after decoding.

[0057] Step S104, after the WASM module is loaded on the browser, the transcoded stream data is played.

[0058] In this embodiment, the audio and video encoding format that is not originally supported by the browser is decoded, and then encoded again after decoding. The method (i.e., transcoding) in which the encoded code stream can be played on the browser can support the video formats of H.264 and H.265, and the code streams of AAC, G.711a, G.711u and G.729 audio formats for playback in the browser. At the same time, this method can be adaptively adjusted according to the browser's support for the audio and video format. The audio and video streams in the format supported by the browser itself are directly transmitted, and the unsupported formats are transcoded, thereby expanding the adaptability of the browser.

[0059] Further, step S103 includes the following steps:

[0060] Step S201, determining whether the code stream data is in a code stream format supported by the browser; if not, executing step S202; if yes, executing step S203;

[0061] Step S202, decoding the code stream data to obtain original audio PCM data;

[0062] Step S203, transparently transmit the code stream data without processing;

[0063] Step S204, encode the original audio PCM data to obtain an ES stream; wherein the ES stream is a plurality of slices, and the slices are output continuously, that is, the video tag plays a complete video.

[0064] Specifically, the code stream data is obtained (for example, through WebSocket, File API or other means); the code stream data is passed to the WASM module for filtering. The WASM module transmits the audio formats that can be decoded by the browser without processing, and decodes the formats that cannot be decoded, such as G.711A / U, G.729, etc., decodes the original audio PCM data, and then re-encodes it according to the format supported by the browser, such as encoding it into AAC audio, so that the browser can play the correct sound, thereby solving the problem that the browser does not support certain audio formats. Among them, the code stream data includes audio data, video data and audio and video data.

[0065] Further, step S103 includes the following steps:

[0066] When the code stream data is video data in H.265 video format, the WASM module converts the video data in H.265 video format into video data in H.264 video format.

[0067] Further, step S104 includes the following steps:

[0068] Step S301, after loading the WASM module on the browser, push the transcoded stream data to the MSE API;

[0069] Step S302: Play the transcoded stream data through the MSE API.

[0070] Specifically, the transcoded bitstream data is converted into <audio>or <video>The format played in the element. Push the transcoded stream data to the MediaSource Extensions (MSE) API, and continuously push the stream to simulate the same effect as playing the stream in the video tag. The browser decompresses the video byte stream data and restores it to the original video frame data to play the stream.

[0071] This application solves the problem that special code stream data cannot be played in the browser, so that the browser can transcode the code stream data locally in real time and convert it into a media format supported by the current browser and device.

[0072] This application provides a method for running code written in multiple languages ​​on the Web at near-native speeds, eliminating format compatibility issues and dynamically adjusting the quality and resolution of the code stream based on the user's network conditions, thereby achieving a smoother and more stable playback experience.

[0073] This application writes code in C++, and after compilation (this file generally ends with WASM), the assembly instructions are obtained, and then the file is loaded into the Web container through the JavaScript related API. WebAssembly is the Assembly running in the Web container.

[0074] This application completes the data interaction channel between Browser and Play Control through the encapsulated protocol.

[0075] This application converts audio formats such as G.711A, G.771U and G.729 into AAC format for playback through compiled WASM files.

[0076] Embodiment 2:

[0077] Figure 2 A schematic diagram of an audio and video decoding system of a browser provided in Embodiment 2 of the present invention.

[0078] Reference Figure 2 , the system comprises:

[0079] The code stream data receiving module is used for the client to receive the code stream data sent by the WebSocket server after the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server;

[0080] The code stream data sending module is used by the client to send the code stream data to the WASM module;

[0081] The transcoding module is used by the WASM module to transcode the code stream data to obtain the transcoded code stream data;

[0082] The loading module is used to play the transcoded stream data after the WASM module is loaded on the browser.

[0083] Furthermore, the transcoding module is specifically used for:

[0084] Determine whether the code stream data is in a code stream format supported by the browser;

[0085] If not, decode the code stream data to obtain the original audio PCM data;

[0086] Encode the original audio PCM data to obtain ES stream;

[0087] Among them, the ES stream is divided into multiple slices.

[0088] This application is applied to online video players, which can greatly improve the performance of web-side video playback, reduce CPU load, and improve user experience.

[0089] Using WASM technology, the decoding speed is close to native, significantly better than pure JavaScript implementation.

[0090] Cross-platform compatibility: Supports multiple browsers including Chrome, Firefox, Safari, and Edge.

[0091] The project is active, constantly optimizing performance and adding new features to keep pace with the latest video encoding standards, making technology evolution and version management easier. There is no need to upgrade the browser itself, which improves the adaptability of the browser.

[0092] The embodiment of the present invention provides a method and system for audio and video decoding of a browser, including: when an HTTP server pulls code stream data from a code stream database and sends the code stream data to a WebSocket server, a client receives the code stream data sent by the WebSocket server; the client sends the code stream data to a WASM module; the WASM module transcodes the code stream data to obtain transcoded code stream data; after the WASM module is loaded on the browser, the transcoded code stream data is played; the audio and video encoding format of the code stream format not supported by the browser is transcoded into a suitable encoding format and then given to the browser for playback without upgrading the browser itself, thereby improving the adaptability of the browser.

[0093] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the audio and video decoding method of the browser provided in the above embodiment are implemented.

[0094] An embodiment of the present invention further provides a computer-readable medium having a non-volatile program code executable by a processor, wherein a computer program is stored on the computer-readable medium, and when the computer program is executed by the processor, the steps of the audio and video decoding method of the browser of the above embodiment are executed.

[0095] The computer program product provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment and will not be repeated here.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0098] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0099] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0100] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.< / video> < / audio>

Claims

1. A method for audio and video decoding of a browser, characterized in that: The method comprises: When the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server, the client receives the code stream data sent by the WebSocket server; The client sends the code stream data to the WASM module; The WASM module transcodes the code stream data to obtain transcoded code stream data; After the WASM module is loaded on the browser, the transcoded stream data is played.

2. The method for audio and video decoding of a browser according to claim 1, characterized in that: The WASM module transcodes the code stream data to obtain transcoded code stream data, including: Determine whether the code stream data is in a code stream format supported by the browser; If not, the code stream data is decoded to obtain original audio PCM data; Encode the original audio PCM data to obtain an ES stream; The ES stream consists of multiple slices.

3. The method for audio and video decoding of a browser according to claim 2, characterized in that: Determining whether the code stream data is in a code stream format supported by the browser further includes: If yes, the code stream data is transparently transmitted without being processed.

4. The method for audio and video decoding of a browser according to claim 1, characterized in that: The WASM module transcodes the code stream data to obtain transcoded code stream data, including: When the code stream data is video data in H.265 video format, the WASM module converts the video data in H.265 video format into video data in H.264 video format.

5. The method for audio and video decoding of a browser according to claim 1, characterized in that: After the WASM module is loaded on the browser, the transcoded stream data is played, including: After the WASM module is loaded on the browser, the transcoded stream data is pushed to the MSE API; The transcoded code stream data is played through the MSE API.

6. The method for audio and video decoding of a browser according to claim 1, characterized in that: The code stream data includes audio data, video data and audio and video data.

7. A browser audio and video decoding system, characterized in that: The system comprises: The code stream data receiving module is used for, when the HTTP server pulls the code stream data from the code stream database and sends the code stream data to the WebSocket server, the client receives the code stream data sent by the WebSocket server; A code stream data sending module, used for the client to send the code stream data to the WASM module; A transcoding module, used for the WASM module to transcode the code stream data to obtain transcoded code stream data; The loading module is used to play the transcoded stream data after loading the WASM module on the browser.

8. The audio and video decoding system of the browser according to claim 7, characterized in that: The transcoding module is specifically used for: Determine whether the code stream data is in a code stream format supported by the browser; If not, the code stream data is decoded to obtain original audio PCM data; Encode the original audio PCM data to obtain an ES stream; The ES stream consists of multiple slices.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 6.

Images