Video synthesis method and device

By using FFmpeg and WebCodecs in the browser to process video scenes in parallel, combined with WebWorker thread pool management, the problems of slow video synthesis speed and insufficient concurrent processing capabilities in the existing technology are solved, and efficient video synthesis and stability improvement are achieved.

CN115955583BActive Publication Date: 2024-12-06BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211633439.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-19
Publication Date
2024-12-06
Estimated Expiration
2042-12-19

AI Technical Summary

Technical Problem

The existing backend-based video editor architecture has performance bottlenecks and process bottlenecks, resulting in slow video synthesis speed and limited concurrent processing capabilities, and it is impossible to process multiple video synthesis requests at the same time.

Method used

By using FFmpeg and WebCodecs in the browser, the video scene is split and processed in parallel. Each frame of the image is rendered into a bmp image, and it is managed by using the WebWorker thread pool. If the browser supports WebCodecs, it will be converted to mp4 video, otherwise it will be converted to ts video, and Synthesize the final video file through FFmpeg.

Benefits of technology

It significantly improves the video synthesis speed and concurrent processing capabilities, avoids server performance bottlenecks and process bottlenecks, and improves the efficiency and stability of video synthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115955583B_ABST
    Figure CN115955583B_ABST
Patent Text Reader

Abstract

The embodiment of the present disclosure discloses a video synthesis method and device. The specific implementation of the method includes: in response to receiving a video synthesis request, determining whether the browser supports WebWorker, wherein the video synthesis request includes audio material and video material; if supported, synthesizing an audio file through FFmpeg based on the audio material, and splitting the video material into multiple parallel video scenes; for each video scene, rendering the scene screen of the canvas corresponding to each frame image; converting the pixel data in each canvas into a bmp picture; if the browser supports WebCodecs, WebCodecs converts the bmp picture into an mp4 video file; and synthesizing the audio file and the mp4 video file into a final video file through FFmpeg. This implementation can solve the performance problem and process bottleneck problem of video rendering synthesis based on the backend architecture.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a video synthesis method and device. Background Art

[0002] In the existing backend-based video editor architecture flow, the editor frontend will generate a canvas in the page based on the user's edited data to simulate video playback. When the user chooses to generate a video, the editor frontend will obtain all the current video scene JSON data, and then pass this data to the backend service. The backend service will start a puppeteer page instance, and then reapply the obtained JSON data to a canvas, and then use the API provided by the canvas to capture each frame. Finally, in the backend service, FFmpeg is used to combine these frame images and audio into a video file, which is uploaded to the operation support system and returned to the frontend.

[0003] There are two problems with the backend-based architecture process. The first is the performance bottleneck problem. Since the video synthesis is performed in the server, the performance of the server machine determines the speed of synthesis, which easily leads to performance bottleneck problems. The second is the process bottleneck problem. Since video frame extraction is achieved by starting the puppeteer instance to use canvas to simulate video playback, and puppeteer has a limit on the number of concurrent instances, the number of concurrent instances cannot exceed ten at the same time. This also means that under this video architecture, only ten video synthesis requests initiated by users can be processed at the same time. Users exceeding the limit can only queue up and wait for the synthesis of the previous users to complete, resulting in a more obvious process bottleneck problem. Summary of the invention

[0004] The embodiments of the present disclosure provide a video synthesis method and device.

[0005] In a first aspect, an embodiment of the present disclosure provides a video synthesis method, comprising: in response to receiving a video synthesis request, determining whether the browser supports WebWorker, wherein the video synthesis request includes audio material and video material; if supported, synthesizing an audio file through FFmpeg based on the audio material, and splitting the video material into multiple parallel video scenes; for each video scene, rendering the scene picture of the canvas corresponding to each frame image; converting pixel data in each canvas into a bmp picture; if the browser supports WebCodecs, converting the bmp picture into an mp4 video file by WebCodecs; and synthesizing the audio file and the mp4 video file into a final video file through FFmpeg.

[0006] In some embodiments, the method further includes: if the browser does not support WebCodecs, converting the bmp picture into multiple ts videos by FFmpeg; and synthesizing the audio file and the multiple ts videos into a final video file by FFmpeg.

[0007] In some embodiments, rendering the scene picture of the canvas corresponding to each frame image includes: converting the video material into a frame atlas through FFmpeg; and taking out the picture of the corresponding frame from the frame atlas to render the scene picture of the canvas.

[0008] In some embodiments, rendering the scene picture of the canvas corresponding to each frame of the image includes: converting all video materials into a bitmap set through WebCodecs; and taking out the bitmap of the corresponding frame from the bitmap set to render the scene picture of the canvas.

[0009] In some embodiments, converting the pixel data in each canvas into a bmp picture includes: converting the pixel data in each canvas into ImageBitmap data through a createImageBitmap method; passing the ImageBitmap data of each canvas into each WebWorker thread; in each WebWorker thread, redrawing the ImageBitmap data into an off-screen canvas, obtaining the corresponding pixel data through getImageData, and converting the pixel data into a bmp picture through an imageDataToArrayBuffer method.

[0010] In some embodiments, multiple threads of a WebWorker are managed through a task queue.

[0011] In some embodiments, functions not required for video synthesis in FFmpeg are removed in advance by compiling a custom FFmpeg tool library.

[0012] In a second aspect, an embodiment of the present disclosure provides a video synthesis device, comprising: a receiving unit, configured to determine whether a browser supports WebWorker in response to receiving a video synthesis request, wherein the video synthesis request includes an audio material and a video material; a splitting unit, configured to, if supported, synthesize an audio file through FFmpeg based on the audio material, and split the video material into multiple parallel video scenes; a rendering unit, configured to render a scene image of a canvas corresponding to each frame image for each video scene; a picture conversion unit, configured to convert pixel data in each canvas into a bmp picture; a video conversion unit, configured to convert the bmp picture into an mp4 video file by WebCodecs if the browser supports WebCodecs; and a synthesis unit, configured to synthesize the audio file and the mp4 video file into a final video file through FFmpeg.

[0013] In some embodiments, the video conversion unit is further configured to: if the browser does not support WebCodecs, convert the bmp picture into multiple ts videos by FFmpeg; and synthesize the audio file and the multiple ts videos into a final video file by FFmpeg.

[0014] In some embodiments, the rendering unit is further configured to: convert the video material into a frame atlas through FFmpeg; and extract pictures of corresponding frames from the frame atlas to render the scene image on the canvas.

[0015] In some embodiments, the rendering unit is further configured to: convert all video materials into a bitmap set through WebCodecs; and extract the bitmap of the corresponding frame from the bitmap set to render the scene screen of the canvas.

[0016] In some embodiments, the image conversion unit is further configured to: convert the pixel data in each canvas into ImageBitmap data through the createImageBitmap method; pass the ImageBitmap data of each canvas into each WebWorker thread; in each WebWorker thread, redraw the ImageBitmap data into an off-screen canvas, obtain the corresponding pixel data through getImageData, and convert the pixel data into a bmp image through an imageDataToArrayBuffer method.

[0017] In some embodiments, multiple threads of a WebWorker are managed through a task queue.

[0018] In some embodiments, functions not required for video synthesis in FFmpeg are removed in advance by compiling a custom FFmpeg tool library.

[0019] In a third aspect, an embodiment of the present disclosure provides an electronic device for video synthesis, comprising: one or more processors; a storage device on which one or more computer programs are stored, and when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0022] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0023] Figure 2 is a flow chart of an embodiment of a video synthesis method according to the present disclosure;

[0024] Figure 3a , 3b is a schematic diagram of an application scenario of the video synthesis method according to the present disclosure;

[0025] Figure 4 is a flow chart of another embodiment of the video synthesis method according to the present disclosure;

[0026] Figure 5 is a structural schematic diagram of an embodiment of a video synthesis device according to the present disclosure;

[0027] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the video synthesis method or video synthesis apparatus of the present disclosure can be applied.

[0031] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as video editors, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0033] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens and support video editing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0034] The server 105 may be a server that provides various services, such as a background video server that provides support for the videos displayed on the terminal devices 101, 102, and 103. The background video server may analyze and process the received data such as the video synthesis request, and feed back the processing results (such as audio material, video material, etc.) to the terminal device.

[0035] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0036] It should be noted that the video synthesis method provided in the embodiment of the present disclosure can be performed by the terminal devices 101, 102, and 103. Accordingly, the video synthesis device can be set in the terminal devices 101, 102, and 103. No specific limitation is made here. If the browser of the terminal device does not support WebWorker, the video synthesis is performed by the server.

[0037] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0038] The technical terms are explained as follows:

[0039] Puppeteer: A tool library that starts a browser on the backend server to simulate page behavior.

[0040] Canvas: The canvas function tag provided by the browser.

[0041] Emscripten: A compilation tool that compiles C / C++ code into Webassembly code.

[0042] FFmpeg: An audio and video processing tool implemented in C language.

[0043] Hard encoding and decoding: A function that uses GPU massive parallelism to accelerate encoding and decoding.

[0044] VideoSeeked: An event callback of Video, which is triggered when the CurrentTime property of the Video tag is set.

[0045] requestAnimationFrame: A timer interface provided by the browser, generally used in canvas animation control.

[0046] toBlob: Converts the pixel data of a Canvas canvas into an immutable, raw data file-like object method.

[0047] WebWorker: A way to run code in another thread of the browser.

[0048] In this technical solution, we will use two technologies, FFmpeg.wasm and WebCodecs. Before explaining the technical solution, you need to know in advance that the synthesis of audio and video is divided into four layers: pixel layer, codec layer, encapsulation format layer and protocol layer.

[0049] In the front-end logic, we mainly need to solve the audio and video encoding process from the pixel layer to the codec layer, as well as the encapsulation process from the audio and video codec layer to the encapsulation format layer. The existing audio and video synthesis technologies FFmpeg.wasm and WebCodecs can realize the audio and video encoding process from the pixel layer to the codec layer, but when used alone, there are problems that do not meet production conditions:

[0050] 1. WebCodecs is a set of APIs provided by browsers, which allows users to access the browser's encoders and decoders to encode and decode videos and audios. During the implementation process, it was found that in terms of audio, WebCodecs currently only supports opus encoding, and does not support the more popular aac encoding. This results in the final generated video being opened in an environment such as the Chrome browser. There will be a problem of no sound, so the method of simply using WebCodecs is not feasible.

[0051] 2. FFmpeg.wasm is a WebAssembly version of FFmpeg that is compiled by a third-party developer using Emscripten to run in a browser. The wasm file size of this version of FFmpeg compiled by Emscripten is very large. Since FFmpeg's Filters are not filtered during compilation, the size of the entire FFmpeg-core.wasm file reaches 24.4MB, which is obviously a file size that is difficult to accept in a production environment.

[0052] If we need to apply the front-end synthesized audio and video to the production environment, we need to ensure that the output solution is small enough and has good enough performance when the audio and video encoding format is correct. Therefore, we first remove most of the unnecessary functions in FFmpeg by compiling the custom FFmpeg tool library, such as disabling the CPU detection function at runtime through --disable-runtime-cpudetect, disabling all components such as encoders and decoders through --disable-everything, and then manually turning on the compilation options you need through configurations such as --enable-decoder and --enable-encoder. Only the core codecs and encapsulators and decapsulators are left, which minimizes the size of FFmpeg after compilation.

[0053] The streamlined FFmpeg.wasm can be used in the production environment. We use the streamlined FFmpeg.wasm to encode the audio data and video data separately, and then mix them into a video file to complete the audio and video synthesis function.

[0054] Continue to refer Figure 2 , shows a process 200 of an embodiment of a video synthesis method according to the present disclosure. The video synthesis method comprises the following steps:

[0055] Step 201: In response to receiving a video synthesis request, determine whether the browser supports WebWorker.

[0056] In this embodiment, the execution subject of the video synthesis method (eg Figure 1 The terminal device shown in FIG. 1 can receive a video synthesis request input by a user. The video synthesis request includes audio material and video material. The terminal device detects whether the local browser supports WebWorker. The audio material and the video material can be downloaded from the server by the user through the terminal device, or can be an audio file or a video file stored locally in the terminal device.

[0057] Step 202: If supported, synthesize an audio file based on the audio material through FFmpeg, and split the video material into multiple parallel video scenes.

[0058] In this embodiment, if WebWorker can be used, an audio file can be synthesized through FFmpeg. If there is audio in the video material, audio data can also be extracted from the video material, and then integrated with the audio material to synthesize an audio file through FFmpeg. Audio files can also be synthesized based on audio synthesis commands, for example, loop playback, extended audio, etc.

[0059] The video material can be divided into multiple video scenes according to the time length, for example, every 24 frames of images are divided into one video scene.

[0060] The video scenes may also be divided according to the maximum number of WebWorker threads that the terminal device can support. For example, if the maximum number of WebWorker threads is 10, the video material may be divided into 10 video scenes. To simplify processing, the video scenes may be evenly divided according to the number of frames.

[0061] In some optional implementations of the present embodiment, the unnecessary functions of video synthesis in FFmpeg are removed in advance by compiling the custom FFmpeg tool library. If the front-end synthesized audio and video needs to be applied to the production environment, we need to ensure that the output solution volume is small enough and the performance is good enough when the audio and video encoding format is correct. Therefore, first we remove most of the unnecessary functions in FFmpeg by compiling the custom FFmpeg tool library, such as disabling the CPU detection function at runtime through --disable-runtime-cpudetect, disabling all components such as encoder encoder, decoder decoder through --disable-everything, and then manually opening the compilation options needed by oneself through configurations such as --enable-decoder and --enable-encoder. Only the core codec and encapsulation and decapsulation functions are left, which minimizes the volume of FFmpeg after compilation.

[0062] Step 203: For each video scene, render the scene image of the canvas corresponding to each frame of image.

[0063] In this embodiment, each video scene is assigned a WebWorker thread for canvas rendering. In the process based on the backend architecture, a new video tag can be created to load the current video resource, and then the video tag can be rendered into the canvas for playback. When the canvas is captured, we need to render the canvas scene image corresponding to each frame. If there is video material in the current scene, it is necessary to set the currentTime of the video tag corresponding to the video material, and then listen to the sought event callback of the video tag. After the callback is triggered, the canvas rendering operation is performed again. It has been found through experiments that setting the currentTime and waiting for the sought callback takes tens or even hundreds of milliseconds. When the video material is relatively long or more, the speed of the overall video synthesis will also be reduced.

[0064] In some optional implementations of this embodiment, the scene picture of the canvas corresponding to each frame image is rendered, including: converting the video material into a frame atlas through FFmpeg; taking out the picture of the corresponding frame from the frame atlas to render the scene picture of the canvas. Here we use FFmpeg to optimize it, and before each synthesis, execute a predetermined command through FFmpeg.runCmd to convert all video materials into a frame atlas. After that, each rendering of the canvas only needs to take out the picture of the corresponding frame from the frame atlas for rendering, which saves the time of waiting for video seek each time and improves the synthesis speed of the video.

[0065] In some optional implementations of this embodiment, rendering the scene picture of the canvas corresponding to each frame of the image includes: converting all video materials into a bitmap set through WebCodecs; taking the bitmap of the corresponding frame from the bitmap set to render the scene picture of the canvas. A WebCodecs decode method is further provided, which hard-decodes these video resources through a decoder provided by the browser to generate a corresponding bitmap bitmap set. Convert all video materials into a bitmap set through a predetermined command.

[0066] Using WebCodecs decode will generate image resources faster than using FFmpeg and is a better choice.

[0067] The method using FFmpeg reduces loading time by 65% ​​compared to the Video seeked method, while the method using WebCodecsdecode reduces loading time by 80% compared to the method using FFmpeg.

[0068] Step 204, convert the pixel data in each canvas into a bmp picture.

[0069] In this embodiment, in the canvas frame capture operation, each time the canvas completes rendering of the current frame, the pixel data in the canvas can be converted into blob data through the toBlob method.

[0070] Step 205: If the browser supports WebCodecs, WebCodecs converts the bmp picture into an mp4 video file.

[0071] In this embodiment, we introduce the new feature of WebCodecs for the synthesized video clips. As mentioned above, WebCodecs has some defects in audio encoding format, but if it is only used in video encoding processing, it will have a good effect in improving performance due to its hard-coded processing. We encapsulate an isWebCodecsSupported method, which will judge the current environment. If the current environment supports the use of WebCodecs, we will use WebCodecs to hard-code the video.

[0072] In some optional implementations of this embodiment, if the browser does not support WebCodecs, FFmpeg converts the bmp image into multiple ts videos; FFmpeg synthesizes the audio file and the multiple ts videos into a final video file. There is a problem with early video transcoding to MP4 format. When the video file is large, it is more difficult. Currently, it is considered to output the video as multiple .ts video clips and a .ts file list in m3u8 format. The following ffmpeg command can be used for conversion:

[0073] ffmpeg-i Wildlife.wmv-codec:v libx264-codec:a mp3-map 0-fssegment-segment_format mpegts-segment_list. / m3u8 / index.m3u8-segment_time 10. / m3u8 / '%03d.ts'

[0074] The method of FFmpeg synthesizing an audio file and multiple ts videos into a video is a prior art, so it will not be described in detail.

[0075] Step 206, synthesize the audio file and the mp4 video file into a final video file through FFmpeg.

[0076] In this embodiment, the method of synthesizing the audio file and the mp4 video file by FFmpeg is a prior art, so it is not described in detail.

[0077] In some optional implementations of the present embodiment, multiple threads of WebWorker are managed through task queues. In terms of memory optimization, since WebWorker is frequently used in this modification, a new worker thread will be started each time the FFmpeg command is executed and the canvas pixel data is assembled and integrated, and different browsers have restrictions on the maximum number of worker threads, so we provide a queue management to limit the creation tasks of the worker thread. When our preset worker instance is online, we will put the tasks created by the worker thread on hold, and only when the running worker is destroyed will these put-on tasks be re-executed, thereby achieving dynamic control of the creation and destruction of the worker thread. When performing canvas frame capture operations, since multiple tasks are executed concurrently and each task consumes more memory, similar task queue management is also used here to avoid the occurrence of memory overflow.

[0078] The method provided by the above embodiment of the present disclosure adjusts the compilation configuration of FFmpeg to streamline the size of the compiled FFmpeg lib tool library, implements the video synthesis function in the form of FFmpeg+Webassembly, embraces new features, combines the official WebCodecs api to process the synthesis of scene video files, and optimizes performance and memory management, thereby significantly improving the speed and stability of audio and video rendering and synthesis, such as Figure 3b shown.

[0079] Continue to see Figure 3a , Figure 3a FIG. 1 is a schematic diagram of an application scenario of the video synthesis method according to this embodiment. Figure 3a In the application scenario, the user triggers a video synthesis request through the video editing software on the terminal device. The video editing software performs a pre-process to determine whether the local browser can use WebWorker. If not, the video synthesis request is sent to the back-end server for video synthesis. If WebWorker can be used locally, the front-end terminal device performs video synthesis. The user selects audio material and video material, and enters an audio synthesis command (for example, looping audio clip 1 five times). The terminal device performs audio synthesis and video synthesis in parallel. Figure 3a The left side shows the audio synthesis process, and the right side shows the video synthesis process. The terminal device obtains the audio file and the audio synthesis command. The background and narration audio are extracted from the audio file. The audio data can also be extracted from the video material, and then all the audio data and commands are integrated, and the integrated audio is synthesized using FFmpeg. Video synthesis is performed in parallel with audio synthesis, such as Figure 3a As shown on the right side of . First, extract the video scene frame, split the video material into multiple video scenes, then obtain the scene canvas at different frame rates in parallel, and convert them into bitmap data. The bitmap data corresponding to each video scene is passed to a worker to be converted into a bmp picture. Then the bmp pictures are synthesized into videos. If the browser supports WebCodecs, the bmp pictures are converted into mp4 video files. If not, FFmpeg is used to convert the bmp pictures into multiple ts videos. Finally, FFmpeg is used to synthesize the audio and video to obtain the final video file.

[0080] Further references Figure 4 , which shows a process 400 of another embodiment of a video synthesis method. The process 400 of the video synthesis method includes the following steps:

[0081] Step 401: In response to receiving a video synthesis request, determine whether the browser supports WebWorker.

[0082] Step 402: If supported, synthesize an audio file based on the audio material through FFmpeg, and split the video material into multiple parallel video scenes.

[0083] Step 403: For each video scene, render the scene image of the canvas corresponding to each frame of image.

[0084] Steps 401-403 are substantially the same as steps 201-203, and thus are not described in detail herein.

[0085] Step 404: Convert the pixel data in each canvas into ImageBitmap data through the createImageBitmap method.

[0086] In this embodiment, the canvas.toBlob() operation is not used to convert the pixel data in the canvas into blob data, because this step is an asynchronous operation and takes a long time. When the number of frames is too large, it will block the execution of the main thread. Therefore, this article converts the pixel data into ImageBitmap data, and createImageBitmap is a very fast operation.

[0087] Step 405: transfer the ImageBitmap data of each canvas to each WebWorker thread.

[0088] In this embodiment, each WebWorker thread can be executed in parallel, is not affected by too many frames, and does not block the execution of the main thread.

[0089] Step 406, in each WebWorker thread, redraw the ImageBitmap data to an off-screen canvas, obtain the corresponding pixel data through getImageData, and convert the pixel data into a bmp image through an imageDataToArrayBuffer method.

[0090] In this embodiment, the pixel data is redrawn in each WebWorker thread in parallel and then converted into a bmp picture.

[0091] Step 407: If the browser supports WebCodecs, WebCodecs converts the bmp picture into an mp4 video file.

[0092] Step 407 is substantially the same as step 205 and thus will not be described in detail.

[0093] Step 408, synthesize the audio file and the mp4 video file into a final video file through FFmpeg.

[0094] Step 408 is substantially the same as step 206 and thus will not be described in detail.

[0095] Although the method of converting canvas pixel data into a bmp image will lengthen the overall process, since createImageBitmap is a very fast operation and subsequent operations are performed in the worker thread, it will not block the main thread of the page, giving users a smoother user experience.

[0096] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video synthesis device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 5As shown, the video synthesis device 500 of this embodiment includes: a receiving unit 501, a splitting unit 502, a rendering unit 503, a picture conversion unit 504, a video conversion unit 505 and a synthesis unit 506. The receiving unit 501 is configured to determine whether the browser supports WebWorker in response to receiving a video synthesis request, wherein the video synthesis request includes audio material and video material; the splitting unit 502 is configured to synthesize an audio file based on the audio material through FFmpeg if supported, and split the video material into multiple parallel video scenes; the rendering unit 503 is configured to render the scene screen of the canvas corresponding to each frame image for each video scene; the picture conversion unit 504 is configured to convert the pixel data in each canvas into a bmp picture; the video conversion unit 505 is configured to convert the bmp picture into an mp4 video file by WebCodecs if the browser supports WebCodecs; the synthesis unit 506 is configured to synthesize the audio file and the mp4 video file into a final video file through FFmpeg.

[0098] In some optional implementations of this embodiment, the video conversion unit 505 is further configured to: if the browser does not support WebCodecs, FFmpeg converts the bmp picture into multiple ts video segments; and FFmpeg synthesizes the audio file and the multiple ts video segments into a final video file.

[0099] In some optional implementations of this embodiment, the rendering unit 503 is further configured to: convert the video material into a frame atlas through FFmpeg; and extract pictures of corresponding frames from the frame atlas to render the scene screen of the canvas.

[0100] In some optional implementations of this embodiment, the rendering unit 503 is further configured to: convert all video materials into a bitmap set through WebCodecs; and extract the bitmap of the corresponding frame from the bitmap set to render the scene screen of the canvas.

[0101] In some optional implementations of this embodiment, the image conversion unit 504 is further configured to: convert the pixel data in each canvas into ImageBitmap data through the createImageBitmap method; pass the ImageBitmap data of each canvas into each WebWorker thread; in each WebWorker thread, redraw the ImageBitmap data into an off-screen canvas, obtain the corresponding pixel data through getImageData, and convert the pixel data into a bmp image through an imageDataToArrayBuffer method.

[0102] In some optional implementations of this embodiment, multiple threads of WebWorker are managed through a task queue.

[0103] In some optional implementations of this embodiment, the functions not required for video synthesis in FFmpeg are removed in advance by compiling a custom FFmpeg tool library.

[0104] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0105] An electronic device for video synthesis includes: one or more processors; a storage device on which one or more computer programs are stored, and when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method described in process 200 or 400.

[0106] A computer-readable medium stores a computer program, wherein the computer program implements the method described in process 200 or 400 when executed by a processor.

[0107] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0108] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a video synthesis method. For example, in some embodiments, the video synthesis method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the video synthesis method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the video synthesis method in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0116] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video synthesis method, comprising: In response to receiving a video synthesis request, determining whether the browser supports WebWorker, wherein the video synthesis request includes audio material and video material; If supported, synthesize an audio file based on the audio material through FFmpeg, and split the video material into multiple parallel video scenes according to the maximum number of WebWorker threads that the terminal device can support; For each video scene, rendering the scene picture of the canvas corresponding to each frame image includes: converting the video material into a frame atlas through FFmpeg; taking out the picture of the corresponding frame from the frame atlas to render the scene picture of the canvas; Convert the pixel data in each canvas into a bmp picture; If the browser supports WebCodecs, WebCodecs converts the bmp picture into an mp4 video file; The audio file and the mp4 video file are synthesized into a final video file through FFmpeg.

2. The method according to claim 1, wherein: The method further comprises: If the browser does not support WebCodecs, FFmpeg will convert the bmp picture into multiple ts videos; The audio file and the multiple ts videos are synthesized into a final video file through FFmpeg.

3. The method according to claim 1, wherein: The rendering of the scene picture of the canvas corresponding to each frame of the image includes: Convert all video materials into bitmap sets through WebCodecs; The bitmap of the corresponding frame is taken out from the bitmap set to render the scene image of the canvas.

4. The method according to claim 1, wherein: The step of converting pixel data in each canvas into a bmp picture includes: Convert the pixel data in each canvas into ImageBitmap data through the createImageBitmap method; Pass the ImageBitmap data of each canvas to each WebWorker thread; In each WebWorker thread, the ImageBitmap data is redrawn to an off-screen canvas, the corresponding pixel data is obtained through getImageData, and the pixel data is converted into a bmp image through an imageDataToArrayBuffer method.

5. The method according to claim 1, wherein: WebWorker's multiple threads are managed through task queues.

6. The method according to claim 1, wherein: Pre-compile the custom FFmpeg tool library to remove unnecessary functions of video synthesis in FFmpeg.

7. A video synthesis device, comprising: A receiving unit is configured to determine whether the browser supports WebWorker in response to receiving a video synthesis request, wherein the video synthesis request includes audio material and video material; a splitting unit, configured to, if supported, synthesize an audio file through FFmpeg based on the audio material, and split the video material into a plurality of parallel video scenes according to a maximum number of threads of the WebWorker that can be supported by the terminal device; The rendering unit is configured to render the scene picture of the canvas corresponding to each frame image for each video scene, including: converting the video material into a frame atlas through FFmpeg; taking out the picture of the corresponding frame from the frame atlas to render the scene picture of the canvas; A picture conversion unit, configured to convert pixel data in each canvas into a bmp picture; The video conversion unit is configured to convert the bmp picture into an mp4 video file by WebCodecs if the browser supports WebCodecs; The synthesis unit is configured to synthesize the audio file and the mp4 video file into a final video file through FFmpeg.

8. An electronic device for video synthesis, comprising: one or more processors; a storage device having one or more computer programs stored thereon, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.