Video transcoding system, method and electronic device

Abstract

The present application provides a video transcoding system, method and electronic device. In one example, the video transcoding system includes: a web page layer and a web page assembly layer; the web page layer is used to send the received audio and video code stream to the web page assembly layer; the web page assembly layer is used to parse the audio and video code stream; the web page assembly layer is also used to perform audio post-processing on the audio data; the web page layer is also used to perform video post-processing on the video data; the web page assembly layer is also used to encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data. The video transcoding system can reduce the occupation of streaming media service resources.

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a video transcoding system, method and electronic device. Background Art

[0002] In order to achieve copyright protection of video data or avoid leakage of key video information, there is a need to superimpose intelligent information (such as watermarks or mosaics) during video transmission or playback.

[0003] Currently, intelligent information superposition of video data is mainly performed through streaming media servers, and the processing of a large amount of video data will occupy a large amount of streaming media service resources. Summary of the invention

[0004] In view of this, the present application provides a video transcoding system, method and electronic device.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a video transcoding system is provided, which is deployed in a browser. The video transcoding system includes a web page layer and a web page assembly WebAssembly layer; wherein:

[0007] The Web layer is used to receive audio and video code streams and send the audio and video code streams to the WebAssembly layer;

[0008] The WebAssembly layer is used to parse the audio and video code stream to obtain the audio code stream and the video code stream in the audio and video code stream;

[0009] The WebAssembly layer is further used to perform audio post-processing on the audio data to obtain processed audio data; wherein the audio data is obtained by performing audio decoding on the audio code stream;

[0010] The Web layer is further used to perform video post-processing on the video data to obtain processed video data; wherein the video post-processing at least includes intelligent information superposition; the video data is obtained by performing video decoding on the video code stream;

[0011] The WebAssembly layer is also used to encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data; wherein the encoded audio data is obtained by encoding the processed audio data, and the encoded video data is obtained by encoding the processed video data.

[0012] According to a second aspect of an embodiment of the present application, a video transcoding method is provided, which is applied to a browser, and the method includes:

[0013] Receiving an audio and video code stream, and parsing the audio and video code stream to obtain an audio code stream and a video code stream in the audio and video code stream;

[0014] Decoding the audio code stream to obtain audio data, and performing audio post-processing on the audio data;

[0015] Decoding the video code stream to obtain video data, and performing video post-processing on the video data; wherein the video post-processing at least includes intelligent information superposition;

[0016] encoding the processed audio data to obtain encoded audio data; and encoding the processed video data to obtain encoded video data;

[0017] The encoded audio data and the encoded video data are encapsulated and packaged, and the obtained audio and video data are saved.

[0018] According to a fifth aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the method described in the first aspect above.

[0019] The video transcoding system of the embodiment of the present application builds a video transcoding system based on the Web layer and the WebAssembly layer in the browser. The audio and video code streams received by the Web layer can be parsed by the WebAssembly layer to obtain the audio code stream and the video code stream in the audio and video code stream; the obtained audio code stream can be audio-decoded to obtain audio data, and the audio can be post-processed by the WebAssembly layer; the video code stream can be video-decoded to obtain video data, and the video can be post-processed by the Web layer; further, the processed audio data can be encoded to obtain encoded audio data, and the processed video data can be encoded to obtain encoded video data, and the WebAssembly layer can encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data. By realizing intelligent information superposition of video data on the terminal browser side, the occupation of streaming media service resources is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a structural schematic diagram of a video transcoding system shown in an exemplary embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of a "watermarked" video transcoding framework shown in an exemplary embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a process for implementing "watermarked" video transcoding as shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a video post-processing module shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is a flowchart of a video transcoding method shown in an exemplary embodiment of the present application;

[0025] Figure 6 is a structural schematic diagram of a video transcoding device shown in an exemplary embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application are briefly explained below.

[0028] 1. WebAssembly: A special assembly language that runs in the browser. It can also be used as target code, compiled from high-level languages ​​such as C / C++ and Rust.

[0029] 2. Web layer: usually refers to the browser layer, also known as the front end, developed using JavaScript language.

[0030] 3. WebAssembly layer: Use the Emscripten compiler to compile the functional module code written in C language into a WebAssembly library.

[0031] 4. Parsing: The encapsulated code stream needs to be parsed during playback.

[0032] 5. Decoding: The encoded bit stream needs to be decoded during playback.

[0033] 6.YUV: The data stream after parsing and decoding of video data is YUV data.

[0034] 7.PCM: The data stream after parsing and decoding of audio data is PCM data.

[0035] 8. Rendering: The decoded YUV / PCM data must be rendered before it can be played.

[0036] 9. WebCodecs: A new feature of Chrome V94. The WebCodecs API makes it possible to use existing media components and codecs in the browser, and can be hardware accelerated. It supports audio and video encoding and decoding, capturing raw video frames, image decoding, and other functions.

[0037] 10. WebGL: A 3D graphics standard that allows JavaScript and OpenGL ES to be combined. In the browser, the Canvas in HTML5 interacts with the DOM and provides hardware 3D accelerated rendering for the Canvas.

[0038] 11. Watermark: When the video is playing, a watermark (text, image, etc.) is superimposed on the video screen. The watermark can be set to italic, full screen, specified position, color, etc.

[0039] 12. Fisheye correction: Correct the severely distorted edges of the fisheye code stream and display it as a flat or 3D image.

[0040] 13. Intelligent information: The private data in the encapsulated code stream is decoded and rendered to form intelligent information, which mainly includes picture frames, writing and overlay images on the screen.

[0041] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0042] See also Figure 1 , is a schematic diagram of the structure of a video transcoding system provided in an embodiment of the present application, wherein the video transcoding system can be deployed in a browser, such as Figure 1 As shown, the video transcoding system may include: a web page layer and a web page assembly WebAssembly layer; wherein:

[0043] The Web layer is used to receive audio and video code streams and send the audio and video code streams to the WebAssembly layer;

[0044] The WebAssembly layer is used to parse the audio and video code stream to obtain the audio code stream and the video code stream in the audio and video code stream;

[0045] The WebAssembly layer is further used to perform audio post-processing on the audio data to obtain processed audio data; wherein the audio data is obtained by audio decoding of the audio code stream;

[0046] The Web layer is further used to perform video post-processing on the video data to obtain processed video data; wherein the video post-processing at least includes intelligent information superposition; the video data is obtained by performing video decoding on the video code stream;

[0047] The WebAssembly layer is also used to encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data; wherein the encoded audio data is obtained by encoding the processed audio data, and the encoded video data is obtained by encoding the processed video data.

[0048] In an embodiment of the present application, in order to reduce the occupation of streaming media service resources by intelligent information superposition of video data, intelligent information superposition of video data can be implemented through a browser on the terminal side, and the video encoding of the superimposed intelligent information can be packaged and saved in the form of a video recording.

[0049] In order to achieve the above purpose, a video transcoding system can be built based on the Web (web page) layer and the WebAssembly (web page assembly) layer in the browser.

[0050] The Web layer can be used to receive audio and video streams (which can be called raw streams), such as receiving audio and video streams sent by front-end monitoring devices, and sending the audio and video streams to the WebAssembly layer.

[0051] The WebAssembly layer can parse the received audio and video code streams to obtain the audio code stream and video code stream in the audio and video code streams.

[0052] The WebAssembly layer can perform audio post-processing on the audio data obtained by audio decoding of the audio code stream.

[0053] Exemplarily, audio post-processing may include, but is not limited to, one or more of noise reduction, resampling, and equalization.

[0054] Among them, noise reduction refers to reducing or eliminating noise in an audio signal to improve the quality of the audio signal.

[0055] Exemplarily, noise may be random background interference, which may come from equipment, environment, etc.

[0056] Resampling refers to the process of generating a new signal by changing the sampling rate of an audio signal to adapt to different processing environments or requirements.

[0057] Equalization is the process of adjusting the intensity of an audio signal at different frequencies to improve the sound quality.

[0058] For example, in order to optimize the decoding performance of video decoding, the Web may use hardware decoding to perform video decoding, for example, decoding the video code stream through a GPU (Graphics Processing Unit).

[0059] Exemplarily, for the video data obtained by decoding the video code stream, video post-processing can be performed on it through the Web.

[0060] Exemplarily, video post-processing may at least include intelligent information overlay.

[0061] Exemplarily, the above-mentioned intelligent information may include but is not limited to one or more of watermark, mosaic and other intelligent information.

[0062] Among them, the above-mentioned other intelligent information may include one or more of picture frames, writing and overlay images in the video picture.

[0063] In one example, video post-processing may also include, but is not limited to, 2D fisheye correction or 3D image processing.

[0064] Exemplarily, the 3D image processing includes one or more of the following:

[0065] 3D fisheye correction, eagle eye expansion, and VR (Virtual Reality).

[0066] Among them, Hawkeye expansion refers to flattening complex three-dimensional scenes or objects for easy observation and analysis.

[0067] VR refers to the use of computer-generated environments to immerse users in a simulated three-dimensional world.

[0068] In an embodiment of the present application, the processed audio data may be encoded to obtain encoded audio data; and the processed video data may be encoded to obtain encoded video data.

[0069] The WebAssembly layer can encapsulate and package the encoded audio data and the encoded video data to obtain video data, and save the obtained video data.

[0070] Exemplarily, in order to improve the security of the video data, the video data may be encrypted and the encrypted video data may be saved.

[0071] It can be seen that in the embodiment of the present application, by constructing a video transcoding system based on the Web layer and the WebAssembly layer in the browser, the audio and video code streams received by the Web layer can be parsed by the WebAssembly layer to obtain the audio code stream and the video code stream in the audio and video code stream; for the obtained audio code stream, audio decoding can be performed on it to obtain audio data, and the audio can be post-processed by using the WebAssembly layer; for the video code stream, video decoding can be performed on it to obtain video data, and the video can be post-processed by the Web layer; then, the processed audio data can be encoded to obtain encoded audio data, and the processed video data can be encoded to obtain encoded video data, and the WebAssembly layer can encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data, and reduce the occupation of streaming media service resources by realizing intelligent information superposition of video data on the terminal browser side.

[0072] In some embodiments, the web layer performs video post-processing on the obtained video data, which may include:

[0073] In the process of rendering the video data, the video data is converted into pixel data, the pixel data is passed to the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is sampled by the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is determined, and the color value is assigned to the pixel position corresponding to the picture according to the color value; and

[0074] The intelligent information to be superimposed is converted into RGB dot matrix data, the RGB dot matrix data is passed to the overlapping fragment shader, the color value of the corresponding coordinate in the texture of the RGB dot matrix data is sampled by the overlapping fragment shader, the color value of the corresponding coordinate in the texture of the RGB dot matrix data is determined, and the color value is assigned to the corresponding pixel position of the screen according to the color value.

[0075] Exemplarily, in the process of rendering the video data on the screen, on the one hand, the video data can be converted into pixel data, the pixel data can be passed to the fragment shader, and the fragment shader can be used to sample the color values ​​of the corresponding coordinates in the texture of the pixel data, determine the color values ​​of the corresponding coordinates in the texture of the pixel data, and assign color values ​​to the pixel positions corresponding to the screen based on the color values.

[0076] The above method can determine the color value of each pixel position in the picture, and assign a corresponding color value to each pixel position.

[0077] On the other hand, for the intelligent information to be superimposed (such as watermarks), it can be converted into RGB dot matrix data, and the obtained RGB dot matrix data can be passed to the overlay fragment shader. The overlay fragment shader is used to sample the color values ​​of the corresponding coordinates in the texture of the RGB dot matrix data, determine the color values ​​of the corresponding coordinates in the texture of the RGB dot matrix data, and assign color values ​​to the corresponding pixel positions on the screen based on the color values.

[0078] The above method can determine the color value to be superimposed at the pixel position where the intelligent information is to be superimposed in the picture, and superimpose the corresponding color value for each pixel position.

[0079] Through the above-mentioned picture rendering processing and intelligent information superposition, a picture superimposed with intelligent information can be obtained.

[0080] In some embodiments, the WebAssembly layer is further used to perform soft decoding on the audio code stream to obtain audio data;

[0081] The Web layer is also used to perform hard decoding on the video code stream to obtain video data;

[0082] The Web layer is further used to hard-code the processed audio data to obtain encoded audio data; and to hard-code the processed video data to obtain encoded video data.

[0083] For example, considering that the video / audio hard decoding / hard encoding has better performance but poor compatibility, the video / audio soft decoding / soft encoding has better compatibility (applicable to more scenarios) but poor performance, in order to meet the needs of different services, the audio and video encoding and decoding can be realized by combining software and hardware.

[0084] Considering that the Web layer can call the GPU through the browser interface, hard decoding / hard coding can be implemented.

[0085] In addition, the Web layer can also call audio and video codec software through the browser interface to implement soft decoding / soft encoding. However, when using the Web layer to implement soft decoding, the functions that can be called need to be determined based on the functions of the software called through the browser interface.

[0086] The WebAssembly layer can implement audio and video encoding and decoding (soft encoding / soft decoding) through software programs, and its functions can be more complete.

[0087] Based on this, the audio code stream can be decoded by the WebAssembly layer using soft decoding to obtain audio data and improve compatibility.

[0088] For the video code stream, the Web layer can use hard decoding to decode it to obtain video data, thereby improving decoding performance.

[0089] For the processed audio data and video data obtained in the above manner, in order to unify the audio data and the video data, the Web layer may encode them in a hard-coded manner to obtain the encoded audio data and the encoded video data.

[0090] It can be recognized that in the embodiment of the present application, the above-mentioned audio and video encoding and decoding method is only a preferred implementation scheme shown in the embodiment of the present application, but it does not limit the protection scope of the present application, that is, the embodiment of the present application can also use other methods to implement audio and video encoding and decoding. For example, for audio code streams, the Web layer can also decode them using hard decoding, and its specific implementation is not repeated here. In some embodiments, the above-mentioned Web layer encodes the processed video data to obtain the encoded video data, which may include:

[0091] In the process of rendering the video data, the rendering state is read;

[0092] When the picture rendering state is not in the empty state, the processed video data is obtained, and the processed video data is encoded to obtain the encoded video data.

[0093] For example, considering that in the process of rendering the video data through the GPU, when the screen rendering is completed, it will enter the blanking state. When the blanking is completed, the GPU will clear the rendered data. Therefore, when the screen rendering is completed and is in a non-blank state, the rendered data is read.

[0094] Accordingly, in the process of rendering the video data, the Web layer can monitor the rendering status, and when the rendering is completed and the status is not empty, the processed video data can be obtained. For example, the processed video data (video data with superimposed intelligent information) can be read from the GPU cache, and the processed video data can be encoded to obtain the encoded video data.

[0095] Exemplarily, during the video data rendering process, the first frame of screen data is rendered, and when the rendering is completed, the screen pixel data is read, and when the reading is completed, the rendering state is cleared, and then the next frame of screen data is rendered.

[0096] The second frame of screen data is rendered. When the rendering is completed, the screen pixel data is read. When the reading is completed, the rendering state is cleared. After clearing, the next frame of screen data is rendered.

[0097] This cycle is repeated to render multiple frames of image data, and multiple frames are connected together to form a video.

[0098] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0099] In this embodiment, a combination of browser WebCodecs and WebAssembly is used to achieve the purpose of transcoding a video with a watermark (taking the superimposed intelligent information as a superimposed watermark as an example) on the browser side.

[0100] See also Figure 2 , is a schematic diagram of a “watermarked” video transcoding framework provided in an embodiment of the present application, such as Figure 2 As shown, the "watermarked" video transcoding framework may include: a Web layer and WebAssembly. Among them:

[0101] In addition to receiving audio and video data sent by the device, the Web layer is also responsible for video hard decoding, video post-processing, and audio and video hard encoding.

[0102] The WebAssembly layer is responsible for parsing audio and video, decoding and post-processing audio, encapsulating and packaging the encoded data, and encrypting the obtained video data.

[0103] based on Figure 2 The "watermarked" video transcoding framework shown in the figure, and the implementation process of "watermarked" video transcoding can be found in Figure 3 shown.

[0104] Combine the following Figure 2 and Figure 3 , some implementation details of the "watermarked" video transcoding implementation process provided in the embodiment of the present application are explained.

[0105] 1. Video processing process.

[0106] 1.1. When the Web layer receives the original video stream, it passes the original video stream data to the WebAssembly layer;

[0107] 1.2. When the WebAssembly layer receives the original video code stream, it parses the original video code stream, and the parsed video code stream (which can be called raw video data) is called back to the Web layer, which uses WebCodecs for hard decoding;

[0108] 1.3. For the video data obtained after hard decoding, the Web layer can use the video post-processing module to perform video post-processing.

[0109] Exemplarily, video post-processing may include “watermark information superposition”, “fisheye correction” post-processing, etc. in WebGL rendering.

[0110] In one example, the schematic diagram of the video post-processing module can be as follows: Figure 4 As shown, the video post-processing module can render the video data and perform watermark superposition, fisheye correction and other 3D image processing (such as Hawkeye expansion or VR, etc.) during the rendering process.

[0111] 1.4. The Web layer obtains the video data containing the “watermark information” and encodes it using WebCodecs.

[0112] 1.5. The Web layer packages and saves the encoded video data and the encoded audio data together.

[0113] 2. Audio processing process.

[0114] 2.1. When the Web layer receives the original audio code stream, it passes the original audio code stream data to the WebAssembly layer.

[0115] 2.2. The WebAssembly layer receives the original audio code stream and parses it. The parsed audio raw data is sent for decoding to obtain PCM (Pulse Code Modulation) data (i.e. the above audio data).

[0116] 2.3. When the PCM data is obtained, the WebAssembly layer can use the audio post-processing module to perform audio post-processing on the PCM data.

[0117] Exemplarily, audio post-processing mainly includes noise reduction, resampling, etc. When the post-processing is completed, the processed PCM data is transmitted to the Web layer.

[0118] 2.4. When the Web layer receives the processed PCM data, it uses WebCodecs to encode the processed PCM data.

[0119] 2.5. The Web layer packages and saves the encoded audio data and the encoded video data together.

[0120] The key implementations in the above implementation process are described below.

[0121] 3.1. When the Web layer receives audio and video data (i.e., the above-mentioned video original code stream and audio original code stream), it is sent to the WebAssembly layer for audio and video parsing. The parsed video data is passed to the Web layer for decoding. After the WebAssembly layer completes decoding and audio post-processing of the parsed audio data, the post-processed audio data is passed to the Web layer.

[0122] 3.2. After the video data is decoded at the Web layer, it enters the video post-processing module and uses WebGL rendering to perform watermark overlay and fisheye correction post-processing.

[0123] Exemplarily, the process of "adding watermark" by rendering technology is as follows:

[0124] 3.2.1. Screen rendering: 1) Convert video data in YUV format (a color space used for video encoding and transmission) into RGB format (i.e., pixel data); 2) Pass the pixel data to the fragment shader through the texture unit; 3) Use the fragment shader to sample the color value of the corresponding coordinate in the texture of the pixel data, determine the color value of the corresponding coordinate in the texture of the pixel data, and assign a color value to the corresponding pixel position of the screen based on the color value.

[0125] 3.2.2. Adding watermark: 1) Convert the font text to be superimposed with watermark into RGB dot matrix data; 2) Pass the RGB dot matrix data to the overlapping fragment shader through the texture unit; 3) Use the color value of the corresponding coordinate in the texture of the overlapping fragment shader RGB dot matrix data, and assign a color value to the corresponding pixel position of the screen based on the color value.

[0126] It can be seen that by assigning the picture color and the watermark color to the same fragment and finally displaying it, the effect of superimposing the watermark in the video picture is achieved.

[0127] 3.3. In the Web layer, the video picture data "with watermark / after fisheye correction" in rendering is captured, that is, the RGB pixel data (with watermark / after fisheye correction) is read from the graphics processing unit (GPU) and video encoding is performed.

[0128] 3.4. The post-processed audio data is fed into the audio encoding module in the Web layer.

[0129] 3.5. Send the encoded video data and audio data to the WebAssembly packaging library for packaging and packaging, and finally obtain the video transcoding data with watermark / fisheye correction.

[0130] like Figure 5 FIG. 1 is a flow chart of a video transcoding method provided in an embodiment of the present application, wherein the video transcoding method can be applied to a browser, such as Figure 5 As shown, the video transcoding method may include the following steps:

[0131] Step S500: receiving an audio and video code stream, and parsing the audio and video code stream to obtain an audio code stream and a video code stream in the audio and video code stream.

[0132] In the embodiment of the present application, the browser can parse the received audio and video streams to obtain the audio stream and video stream in the audio and video streams.

[0133] In an example, receiving an audio and video code stream, parsing the audio and video code stream, and obtaining an audio code stream and a video code stream in the audio and video code stream may include:

[0134] The audio and video code stream is received through the Web layer and sent to the WebAssembly layer, which parses the audio and video code stream to obtain the audio code stream and video code stream in the audio and video code stream.

[0135] Exemplarily, a video transcoding system may be constructed based on the Web layer and the WebAssembly layer in the browser.

[0136] The browser can receive audio and video streams (which can be called original streams) through the Web layer, such as receiving audio and video streams sent by front-end monitoring devices, and send the audio and video streams to the WebAssembly layer.

[0137] The WebAssembly layer can parse the received audio and video code streams to obtain the audio code stream and video code stream in the audio and video code streams.

[0138] Step S510: decode the audio code stream to obtain audio data, and perform audio post-processing on the audio data.

[0139] In an embodiment of the present application, the browser can decode the acquired audio code stream to obtain audio data and perform audio post-processing on the audio data.

[0140] Exemplarily, audio post-processing may include, but is not limited to, one or more of noise reduction, resampling, and equalization.

[0141] In an example, decoding the audio code stream to obtain audio data and performing audio post-processing on the audio data may include:

[0142] The audio code stream is soft-decoded through the WebAssembly layer to obtain audio data, and then the audio data is post-processed.

[0143] Exemplarily, for an audio bitstream, the browser may perform audio decoding on it through the WebAssembly layer (such as soft decoding, i.e., decoding the audio bitstream through a software program), obtain decoded audio data, and perform audio post-processing on the obtained audio data.

[0144] Step S520: decode the video code stream to obtain video data, and perform video post-processing on the video data; wherein the video post-processing at least includes intelligent information superposition.

[0145] In the embodiment of the present application, the browser can decode the acquired video stream to obtain video data and perform video post-processing on the video data.

[0146] Exemplarily, video post-processing may at least include intelligent information overlay.

[0147] Exemplarily, the above-mentioned intelligent information may include but is not limited to one or more of watermark, mosaic and other intelligent information.

[0148] In one example, video post-processing may also include, but is not limited to, 3D image processing.

[0149] In an example, decoding the video code stream to obtain video data and performing video post-processing on the video data may include:

[0150] The video code stream is sent to the Web layer through the WebAssembly layer, and the Web layer hard-decodes the video code stream to obtain video data, and performs video post-processing on the obtained video data.

[0151] For example, for a video stream, the browser may send it to the Web layer through the WebAssembly layer, and the Web layer may perform video decoding (such as hard decoding, that is, decoding the video stream through a GPU).

[0152] The Web layer can decode the video code stream sent by the WebAssembly layer to obtain decoded video data, and perform video post-processing on the obtained video data.

[0153] Step S530: encode the processed audio data to obtain encoded audio data; and encode the processed video data to obtain encoded video data.

[0154] In an embodiment of the present application, when the processed audio data and the processed video data are obtained in the above manner, the browser can encode the processed audio data to obtain encoded audio data; and encode the processed video data to obtain encoded video data.

[0155] In one example, encoding the processed audio data to obtain the encoded audio data; and encoding the processed video data to obtain the encoded video data may include:

[0156] The audio data processed by the Web layer is hard-coded to obtain encoded audio data; and the processed video data is hard-coded to obtain encoded video data.

[0157] Exemplarily, the browser may encode (e.g., hard code) the processed audio data through the Web layer to obtain encoded audio data; and encode (e.g., hard code) the processed video data to obtain encoded video data, and send the encoded audio data and the encoded video data to the WebAssembly layer.

[0158] Step S540: encapsulate and package the encoded audio data and the encoded video data, and save the obtained audio and video data.

[0159] In an embodiment of the present application, when the encoded audio data and the encoded video data are obtained in the above manner, the browser can encapsulate and package the encoded audio data and the encoded video data, and save the obtained audio and video data.

[0160] In an example, encapsulating and packaging the encoded audio data and the encoded video data, and saving the obtained audio and video data, may include:

[0161] The encoded audio data and the encoded video data are sent to the WebAssembly layer through the Web layer, and the WebAssembly layer encapsulates and packages the encoded audio data and the encoded video data, and saves the obtained audio and video data.

[0162] Exemplarily, the browser can send the encoded audio data and the encoded video data to the WebAssembly layer at the Web layer, and the WebAssembly layer can encapsulate and package the encoded audio data and the encoded video data, and save the obtained audio and video data.

[0163] In some embodiments, the above-mentioned video post-processing of the obtained video data may include:

[0164] In the process of rendering the video data, the video data is converted into pixel data, the pixel data is passed to the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is sampled by the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is determined, and the color value is assigned to the pixel position corresponding to the screen according to the color value; and

[0165] The intelligent information to be superimposed is converted into RGB dot matrix data, the RGB dot matrix data is passed to the overlapping fragment shader, the color value of the corresponding coordinate in the texture of the RGB dot matrix data is sampled by the overlapping fragment shader, the color value of the corresponding coordinate in the texture of the RGB dot matrix data is determined, and the color value is assigned to the corresponding pixel position of the screen according to the color value.

[0166] In some embodiments, encoding the processed video data to obtain the encoded video data may include:

[0167] In the process of rendering the video data, the rendering state is read;

[0168] When the picture rendering state is not in the empty state, the processed video data is obtained, and the processed video data is encoded to obtain the encoded video data.

[0169] For example, considering that in the process of rendering the video data through the GPU, when the screen rendering is completed, it will enter the blanking state. When the blanking is completed, the GPU will clear the rendered data. Therefore, when the screen rendering is completed and is in a non-blank state, the rendered data is read.

[0170] Accordingly, during the process of rendering the video data, the browser can monitor the rendering status, and when the rendering is completed and the status is not empty, the processed video data can be obtained. For example, the processed video data (video data with superimposed intelligent information) can be read from the GPU cache, and the processed video data can be encoded to obtain the encoded video data.

[0171] like Figure 6 FIG. 1 is a flow chart of a video transcoding device provided in an embodiment of the present application, wherein the video transcoding device can be deployed in a browser, such as Figure 6 As shown, the video transcoding device may include:

[0172] An audio and video parsing unit, used for receiving an audio and video code stream, and parsing the audio and video code stream to obtain an audio code stream and a video code stream in the audio and video code stream;

[0173] An audio processing unit, used for decoding the audio code stream to obtain audio data, and performing audio post-processing on the audio data;

[0174] A video processing unit, configured to decode the video code stream to obtain video data, and perform video post-processing on the video data; wherein the video post-processing at least includes intelligent information superposition;

[0175] An audio encoding unit, used to encode the processed audio data to obtain encoded audio data;

[0176] A video encoding unit, used for encoding the processed video data to obtain encoded video data;

[0177] The encapsulation unit is used to encapsulate and package the encoded audio data and the encoded video data, and save the obtained audio and video data.

[0178] In some embodiments, the browser includes a web page Web layer and a web page assembly WebAssembly layer;

[0179] The audio and video parsing unit receives an audio and video code stream, and parses the audio and video code stream to obtain an audio code stream and a video code stream in the audio and video code stream, including:

[0180] The audio and video code stream is received through the Web layer and sent to the WebAssembly layer, and the WebAssembly layer parses the audio and video code stream to obtain the audio code stream and the video code stream in the audio and video code stream;

[0181] The audio processing unit decodes the audio code stream to obtain audio data, and performs audio post-processing on the audio data, including:

[0182] Performing soft decoding on the audio code stream through the WebAssembly layer to obtain audio data, and performing audio post-processing on the audio data;

[0183] The video processing unit decodes the video code stream to obtain video data, and performs video post-processing on the video data, including:

[0184] The video code stream is sent to the Web layer through the WebAssembly layer, and the Web layer hard-decodes the video code stream to obtain video data, and performs video post-processing on the obtained video data;

[0185] The audio encoding unit encodes the processed audio data to obtain encoded audio data, including:

[0186] Hard-coding the processed audio data through the Web layer to obtain encoded audio data;

[0187] The video processing unit encodes the processed video data to obtain encoded video data, including:

[0188] Hard-coding the processed video data through the Web layer to obtain encoded video data;

[0189] The encapsulation unit encapsulates and packages the encoded audio data and the encoded video data, and saves the obtained audio and video data, including:

[0190] The encoded audio data and the encoded video data are sent to the WebAssembly layer through the Web layer, and the WebAssembly layer encapsulates and packages the encoded audio data and the encoded video data, and saves the obtained video data.

[0191] In some embodiments, the video processing unit performs video post-processing on the obtained video data, including:

[0192] In the process of rendering the video data, the video data is converted into pixel data, the pixel data is passed to a fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is sampled by the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is determined, and the color value is assigned to the pixel position corresponding to the screen according to the color value; and

[0193] The intelligent information to be superimposed is converted into RGB dot matrix data, and the RGB dot matrix data is passed to the overlapping fragment shader. The color value of the corresponding coordinate in the texture of the RGB dot matrix data is sampled by the overlapping fragment shader to determine the color value of the corresponding coordinate in the texture of the RGB dot matrix data, and a color value is assigned to the pixel position corresponding to the screen according to the color value.

[0194] In some embodiments, the video encoding unit encodes the processed video data to obtain encoded video data, including:

[0195] In the process of rendering the video data, reading the rendering state;

[0196] When the picture rendering state is not in the clear state, the processed video data is obtained, and the processed video data is encoded to obtain the encoded video data.

[0197] In some embodiments, the video post-processing further includes: 2D fisheye correction or 3D image processing;

[0198] The 3D image processing includes one or more of the following:

[0199] 3D fisheye correction, eagle eye expansion, and virtual reality VR;

[0200] and / or,

[0201] The audio post-processing includes one or more of the following:

[0202] Noise reduction, resampling, and equalization;

[0203] and / or,

[0204] The intelligent information includes one or more of watermark, mosaic and other intelligent information;

[0205] The other intelligent information includes one or more of picture frames, writing and overlay images in the video picture.

[0206] See also Figure 7 , is a schematic diagram of the hardware structure of an encoding end device provided in an embodiment of the present application. The encoding end device may include a processor 701 and a machine-readable storage medium 702 storing machine-executable instructions. The processor 701 and the machine-readable storage medium 702 may communicate via a system bus 703. In addition, by reading and executing the machine-executable instructions corresponding to the filtering control logic in the machine-readable storage medium 702, the processor 701 may execute the filtering method of the encoding end device described above.

[0207] The machine-readable storage medium 702 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0208] In some embodiments, a machine-readable storage medium is further provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the filtering method of the encoding end device described above is implemented. For example, the machine-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

1. A video transcoding system, characterized in that: Deployed in a browser, the video transcoding system includes a web layer and a web assembly WebAssembly layer; wherein: The Web layer is used to receive audio and video code streams and send the audio and video code streams to the WebAssembly layer; The WebAssembly layer is used to parse the audio and video code stream to obtain the audio code stream and the video code stream in the audio and video code stream; The WebAssembly layer is further used to perform audio post-processing on the audio data to obtain processed audio data; wherein the audio data is obtained by performing audio decoding on the audio code stream; The Web layer is further used to perform video post-processing on the video data to obtain processed video data; wherein the video post-processing at least includes intelligent information superposition; the video data is obtained by performing video decoding on the video code stream; The WebAssembly layer is further used to encapsulate and package the encoded audio data and the encoded video data, and save the obtained video data; wherein the encoded audio data is obtained by encoding the processed audio data, and the encoded video data is obtained by encoding the processed video data; The WebAssembly layer is further used to perform soft decoding on the audio code stream to obtain audio data; The Web layer is also used to perform hard decoding on the video code stream to obtain video data; The Web layer is further used to hard-code the processed audio data to obtain encoded audio data; and hard-code the processed video data to obtain encoded video data; The Web layer performs video post-processing on the obtained video data, including: In the process of rendering the video data, the video data is converted into pixel data, the pixel data is passed to a fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is sampled by the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is determined, and the color value is assigned to the pixel position corresponding to the screen according to the color value; and Converting the intelligent information to be superimposed into RGB dot matrix data, passing the RGB dot matrix data to a reduplicated fragment shader, using the reduplicated fragment shader to sample the color value of the corresponding coordinate in the texture of the RGB dot matrix data, determining the color value of the corresponding coordinate in the texture of the RGB dot matrix data, and assigning a color value to the pixel position corresponding to the screen according to the color value; wherein the intelligent information includes a watermark; The Web layer is specifically used to hard-decode the video code stream using WebCodecs to obtain video data; The Web layer is also specifically used to obtain video data superimposed with watermark information, and use WebCodecs to hard-code the video data containing the watermark information.

2. The video transcoding system according to claim 1, characterized in that: The Web layer encodes the processed video data to obtain encoded video data, including: In the process of rendering the video data, reading the rendering state; When the picture rendering state is not in the clear state, the processed video data is obtained, and the processed video data is encoded to obtain the encoded video data.

3. The video transcoding system according to claim 1, characterized in that: The video post-processing also includes: 2D fisheye correction or 3D image processing; The 3D image processing includes one or more of the following: 3D fisheye correction, eagle eye expansion, and virtual reality VR; and / or, The audio post-processing includes one or more of the following: Noise reduction, resampling, and equalization; and / or, Intelligent information also includes mosaic and one or more of other intelligent information; The other intelligent information includes one or more of picture frames, writing and overlay images in the video picture.

4. A video transcoding method, characterized in that: Applied to a browser, the method comprises: Receiving an audio and video code stream, and parsing the audio and video code stream to obtain an audio code stream and a video code stream in the audio and video code stream; Decoding the audio code stream to obtain audio data, and performing audio post-processing on the audio data; Decoding the video code stream to obtain video data, and performing video post-processing on the video data; wherein the video post-processing at least includes intelligent information superposition; encoding the processed audio data to obtain encoded audio data; and encoding the processed video data to obtain encoded video data; Encapsulating and packaging the encoded audio data and the encoded video data, and saving the obtained audio and video data; Wherein, the browser includes a web page Web layer and a web page assembly WebAssembly layer; The step of decoding the audio code stream to obtain audio data and performing audio post-processing on the audio data includes: Performing soft decoding on the audio code stream through the WebAssembly layer to obtain audio data, and performing audio post-processing on the audio data; The decoding of the video code stream to obtain video data and performing video post-processing on the video data includes: The video code stream is sent to the Web layer through the WebAssembly layer, and the Web layer hard-decodes the video code stream to obtain video data, and performs video post-processing on the obtained video data; The step of encoding the processed audio data to obtain the encoded audio data; and encoding the processed video data to obtain the encoded video data comprises: Hard-coding the processed audio data through the Web layer to obtain encoded audio data; and hard-coding the processed video data to obtain encoded video data; The step of performing video post-processing on the obtained video data includes: In the process of rendering the video data, the video data is converted into pixel data, the pixel data is passed to a fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is sampled by the fragment shader, the color value of the corresponding coordinate in the texture of the pixel data is determined, and the color value is assigned to the pixel position corresponding to the screen according to the color value; and Converting the intelligent information to be superimposed into RGB dot matrix data, passing the RGB dot matrix data to a reduplicated fragment shader, using the reduplicated fragment shader to sample the color value of the corresponding coordinate in the texture of the RGB dot matrix data, determining the color value of the corresponding coordinate in the texture of the RGB dot matrix data, and assigning a color value to the pixel position corresponding to the screen according to the color value; wherein the intelligent information includes a watermark; The decoding of the video code stream to obtain video data includes: Decoding the video code stream to obtain video data; The hard encoding of the processed video data comprises: The video data superimposed with the watermark information is obtained, and the video data containing the watermark information is hard-coded using WebCodecs.

5. The method according to claim 4, characterized in that The browser includes a web page Web layer and a web page assembly WebAssembly layer; The receiving audio and video code stream, parsing the audio and video code stream, and obtaining the audio code stream and the video code stream in the audio and video code stream include: The audio and video code stream is received through the Web layer and sent to the WebAssembly layer, and the WebAssembly layer parses the audio and video code stream to obtain the audio code stream and the video code stream in the audio and video code stream; The step of encapsulating and packaging the encoded audio data and the encoded video data, and saving the obtained audio and video data, includes: The encoded audio data and the encoded video data are sent to the WebAssembly layer through the Web layer, and the WebAssembly layer encapsulates and packages the encoded audio data and the encoded video data, and saves the obtained video data.

6. The method according to claim 5, characterized in that The step of encoding the processed video data to obtain encoded video data comprises: In the process of rendering the video data, reading the rendering state; When the picture rendering state is not in the clear state, the processed video data is obtained, and the processed video data is encoded to obtain the encoded video data.

7. The method according to claim 4, characterized in that The video post-processing also includes: 2D fisheye correction or 3D image processing; The 3D image processing includes one or more of the following: 3D fisheye correction, eagle eye expansion, and virtual reality VR; and / or, The audio post-processing includes one or more of the following: Noise reduction, resampling, and equalization; and / or, Intelligent information also includes mosaic and one or more of other intelligent information; The other intelligent information includes one or more of picture frames, writing and overlay images in the video picture.

8. An electronic device, characterized in that: The method comprises a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the method according to any one of claims 4 to 7.

Images