Video editing method, device, computer equipment and storage medium

Abstract

The present application relates to a video editing method, device, computer equipment, storage medium and computer program product. The method includes: obtaining a real-time video stream; transmitting the real-time video stream to a cloud server; receiving audio and video data, which is obtained by encoding the real-time video stream by the cloud server; responding to the editing operation, obtaining the editing material and audio and video clips corresponding to the editing operation, and sending the editing material and audio and video clips to the cloud server; receiving the target editing video, which is generated by the cloud server according to the editing material and the audio and video clips. The synthesis queue is edited and synthesized on the audio and video data according to the synthesis queue, and the target editing video is obtained and pushed to the terminal. The method directly obtains the real-time video stream shot by the user, transmits the real-time video stream to the cloud server, and encodes and edits the real-time video stream through the cloud server, thereby improving the real-time performance of video transmission and editing, and improving the efficiency of video editing.

Description

Technical Field

[0001] The present application relates to the technical field of cloud computing resource scheduling, and in particular to a video editing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the advent of the 5G information age, information transmission and interactive experience have brought about technological innovation. Short videos refer to short scene-based videos of tens of seconds to several minutes that have been popular in recent years; short video editing refers to compressing the length of the short videos shot through editing tools and platforms, splicing clips, adding stickers, text, background music, filters, and transition animations to enrich the presentation of short videos.

[0003] Traditional video processing software has many functions, high requirements on equipment performance, complex editing process, cumbersome interaction, and the editing process cannot be interactive in real time, resulting in low video editing efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a video editing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of video editing in response to the above technical problems.

[0005] In a first aspect, the present application provides a video editing method. The method comprises:

[0006] Get live video stream;

[0007] Transmit live video streams to cloud servers;

[0008] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0009] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0010] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0011] In one embodiment, transmitting the real-time video stream to the cloud server includes: transmitting the real-time video stream to the cloud server through a real-time transmission service thread.

[0012] In one of the embodiments, before transmitting the real-time video stream to the cloud server, it also includes: obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds the preset transmission bit rate, actively dropping frames of the real-time video stream.

[0013] In a second aspect, the present application also provides a video editing method. The method comprises:

[0014] Receive real-time video stream transmitted by the terminal;

[0015] Encode the real-time video stream, obtain and push audio and video data;

[0016] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0017] Generate a synthesis queue based on editing materials and audio and video clips;

[0018] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0019] In one of the embodiments, audio and video data are edited and synthesized according to a synthesis queue to obtain and push a target edited video to a terminal, including: reading the synthesis queue in sequence according to a time chain; editing and synthesizing the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain edited clips; assembling the edited clips to obtain and push the target edited video to the terminal.

[0020] In one of the embodiments, the audio and video clips corresponding to the synthesis queue are edited and synthesized according to the synthesis queue to obtain the edited clip, which includes: taking the first key frame in the audio and video clip as the starting point, editing and synthesizing the audio and video clips according to the editing materials in the synthesis queue through the editing synthesis service thread to obtain the edited clip.

[0021] In a third aspect, the present application also provides a video editing device. The device comprises:

[0022] Video stream acquisition module, used to acquire real-time video stream;

[0023] The terminal transmission module is used to transmit the real-time video stream to the cloud server;

[0024] The terminal receiving module is used to receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0025] The terminal response module is used to respond to the editing operation, obtain the editing material and the audio and video clips corresponding to the editing operation, and send the editing material and the audio and video clips to the cloud server;

[0026] The target clip video receiving module is used to receive the target clip video. The target clip video is generated by the cloud server based on the clip material and the audio and video clips. The audio and video data are edited and synthesized according to the synthesis queue.

[0027] In one of the embodiments, the terminal transmission module is also used to transmit the real-time video stream to the cloud server through the real-time transmission service thread.

[0028] In one of the embodiments, the video editing device further includes an active frame dropping module for obtaining a video bit rate of a real-time video stream; when the video bit rate exceeds a preset transmission bit rate, active frame dropping is performed on the real-time video stream.

[0029] In a fourth aspect, the present application also provides a video editing device. The device comprises:

[0030] A cloud receiving module is used to receive real-time video streams transmitted by terminals;

[0031] The encoding module is used to encode the real-time video stream and obtain and push the audio and video data;

[0032] The clip receiving module is used to receive clip materials and audio and video clips. The clip operation and the audio and video clips are obtained by the terminal responding to the clip operation and acquiring the clip materials and the audio and video clips corresponding to the clip operation;

[0033] A synthesis queue generation module is used to generate a synthesis queue according to the editing materials and audio and video clips;

[0034] The editing and synthesis module is used to edit and synthesize the audio and video data according to the synthesis queue, obtain and push the target edited video to the terminal.

[0035] In one of the embodiments, the editing and synthesis module is also used to read the synthesis queue in sequence according to the time chain; edit and synthesize the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain editing clips; assemble the editing clips to obtain and push the target editing video to the terminal.

[0036] In one of the embodiments, the editing and synthesis module is further used to take the first key frame in the audio and video clip as the starting point, and to edit and synthesize the audio and video clip according to the editing materials in the synthesis queue through the editing and synthesis service thread to obtain a clip.

[0037] In a fifth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0038] Get live video stream;

[0039] Transmit live video streams to cloud servers;

[0040] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0041] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0042] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0043] In a sixth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0044] Receive real-time video stream transmitted by the terminal;

[0045] Encode the real-time video stream, obtain and push audio and video data;

[0046] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0047] Generate a synthesis queue based on editing materials and audio and video clips;

[0048] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0049] In a seventh aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0050] Get live video stream;

[0051] Transmit live video streams to cloud servers;

[0052] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0053] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0054] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0055] In an eighth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0056] Receive real-time video stream transmitted by the terminal;

[0057] Encode the real-time video stream, obtain and push audio and video data;

[0058] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0059] Generate a synthesis queue based on editing materials and audio and video clips;

[0060] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0061] In a ninth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0062] Get live video stream;

[0063] Transmit live video streams to cloud servers;

[0064] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0065] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0066] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0067] In a tenth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0068] Receive real-time video stream transmitted by the terminal;

[0069] Encode the real-time video stream, obtain and push audio and video data;

[0070] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0071] Generate a synthesis queue based on editing materials and audio and video clips;

[0072] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0073] The above-mentioned video editing method, device, computer equipment, storage medium and computer program product obtain real-time video stream; transmit the real-time video stream to a cloud server; receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server; respond to the editing operation, obtain the editing material and audio and video clips corresponding to the editing operation, and send the editing material and audio and video clips to the cloud server; receive the target editing video, which is obtained by the cloud server generating a synthesis queue based on the editing material and the audio and video clips, and editing and synthesizing the audio and video data according to the synthesis queue. This method directly obtains the real-time video stream shot by the user through the terminal, transmits the real-time video stream to the cloud server, and encodes and edits the real-time video stream through the cloud server, which reduces the dependence on device performance, improves the real-time performance of video transmission and editing, and further improves the efficiency of video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A diagram showing an application environment of a video editing method in an embodiment;

[0075] Figure 2 A schematic diagram of a video editing method according to an embodiment of the present invention;

[0076] Figure 3 is a flow chart of a video editing method in another embodiment;

[0077] Figure 4 A schematic diagram of a process of real-time video stream acquisition and encoding in one embodiment;

[0078] Figure 5 A schematic diagram of the composition of a video frame in one embodiment;

[0079] Figure 6 A schematic diagram of a clip synthesis process in one embodiment;

[0080] Figure 7 A schematic diagram of a core framework unit in one embodiment;

[0081] Figure 8 is a structural block diagram of a video editing device in one embodiment;

[0082] Fig. 9 is a structural block diagram of a video editing device in another embodiment;

[0083] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0085] Nowadays, the era of text and image communication has gradually lost its communication advantages, and short video communication can better capture the user's attention. For some small scenarios that want to use short videos for marketing, user activation, etc., there is no need to integrate the terminal short video SDK but can meet the needs of high-precision editing of short videos, adding stickers, adding background music, adding filters, transition animations, etc.

[0086] Traditional video editing methods often suffer from high requirements for device performance, difficulty in using large-scale software, or inability to meet frame-level video processing accuracy requirements. At the same time, with the rapid development of short video applications in the mobile era in recent years, short video apps such as "TikTok" and "Kuaishou" have become emerging mobile applications with hundreds of millions of users. However, the development of such applications requires the integration of a third-party short video editing SDK, which has high development costs, high SDK fees, and requires multi-terminal integration.

[0087] This application is designed to solve this problem and provides a lightweight, flexible, cross-platform, high-precision and multi-functional cloud-based cross-platform editing method.

[0088] Similarly, the traditional cloud editing method requires uploading the filmed video to the cloud platform, and then sending editing instructions to the platform. This editing method is not real-time enough and the interaction is cumbersome. The details of the delay in the upload, interactive synthesis, download, etc. This application uses WebRTC communication technology to eliminate the upload link, which can solve the cumbersome and inefficient problems of traditional cloud editing.

[0089] Cloud editing refers to placing the video editing platform tool on a cloud server, and using the server to perform editing operations such as compressing the video, splicing clips, adding stickers, adding text, adding background music, adding filters, adding transition animations, etc.

[0090] WebRTC, the name comes from the abbreviation of Web Real-Time Communication, is an API that supports web browsers to conduct real-time voice or video conversations. It was open sourced on June 1, 2011 and was included in the W3C recommended standard of the World Wide Web Consortium with the support of Google, Mozilla, and Opera. WebRTC allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary, and to achieve the transmission of video streams and / or audio streams or any other data. The standards included in WebRTC make it possible for users to create peer-to-peer data sharing and conference calls without installing any plug-ins or third-party software.

[0091] The video editing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the cloud server 104 through the network. The data storage system can store the data that the cloud server 104 needs to process. The data storage system can be integrated on the cloud server 104, or it can be placed on the cloud or other network servers. The terminal obtains the real-time video stream, transmits the real-time video stream to the cloud server 104, and the cloud server 104 encodes the real-time video stream to obtain audio and video data. The terminal responds to the user's editing operation, obtains the editing material and audio and video clips corresponding to the editing operation, and sends the editing material and audio and video clips to the cloud server 104. The cloud server generates a synthesis queue based on the editing material and the audio and video clips, and edits and synthesizes the audio and video data according to the synthesis queue, and sends it to the terminal 102. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0092] In one embodiment, Figure 2 As shown, a video editing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0093] Step 202: Acquire real-time video stream.

[0094] Specifically, the terminal obtains the real-time video stream collected by the terminal camera through the video stream acquisition interface. Further, the terminal obtains the real-time video stream collected by the terminal camera through the getUserMedia API interface method of WebRTC.

[0095] Step 204: Transmit the real-time video stream to the cloud server.

[0096] Specifically, the terminal transmits the real-time video stream collected by the camera to the cloud server through a two-way data transmission channel. The two-way data transmission channel is a transmission channel established between the terminal and the cloud server through the UDP protocol.

[0097] Step 206, receiving audio and video data.

[0098] Among them, the audio and video data is obtained by encoding the real-time video stream by the cloud server.

[0099] Specifically, the terminal receives audio and video data transmitted by the cloud server through a two-way data transmission channel, and pushes key frames in the audio and video data to a display interface for display, so that the user can preview the audio and video in real time.

[0100] Step 208, in response to the editing operation, obtaining the editing material and the audio and video clips corresponding to the editing operation, and sending the editing material and the audio and video clips to the cloud server.

[0101] Among them, the editing material library provides a variety of editing materials, which is the rich editing content function of cloud editing. For example, the filter function and rotation provided by FFMpeg; or some gif animations, light effect animations, or background music; each material includes the material id and material type, synthesis operation action instructions, etc. The subsequent synthesis queue will parse them one by one and call the corresponding synthesis method.

[0102] Specifically, in response to the user's editing operation on the audio and video data on the terminal display interface, the editing materials, synthesis operation action instructions and corresponding audio and video clips corresponding to the editing operation are obtained, and the editing materials, synthesis operation action instructions and audio and video clips are transmitted to the cloud server through a two-way data transmission channel.

[0103] Step 210: Receive a target video clip.

[0104] The target clip video is obtained by generating a synthesis queue by the cloud server based on the clipping materials and audio and video clips, and editing and synthesizing the audio and video data according to the synthesis queue.

[0105] Specifically, when the cloud server completes the audio and video editing, it transmits the target edited video to the terminal through a two-way data transmission channel. After the terminal receives the target edited video, it pushes it to the display interface so that the user can browse the target edited video.

[0106] In the above-mentioned video editing method, the above-mentioned video editing method, device, computer equipment, storage medium and computer program product obtain real-time video stream; transmit the real-time video stream to the cloud server; receive audio and video data, the audio and video data is obtained by the cloud server encoding the real-time video stream; respond to the editing operation, obtain the editing material and audio and video clips corresponding to the editing operation, and send the editing material and audio and video clips to the cloud server; receive the target editing video, the target editing video is generated by the cloud server according to the editing material and the audio and video clip The synthesis queue is obtained by editing and synthesizing the audio and video data according to the synthesis queue. This method directly obtains the real-time video stream shot by the user through the terminal, transmits the real-time video stream to the cloud server, and encodes and edits the real-time video stream through the cloud server, which reduces the dependence on device performance, improves the real-time performance of video transmission and editing, and further improves the efficiency of video editing.

[0107] In an optional embodiment, transmitting the real-time video stream to the cloud server includes: transmitting the real-time video stream to the cloud server through a real-time transmission service thread.

[0108] Among them, the real-time transmission service thread provides transmission services based on the WebRTC communication service protocol.

[0109] Specifically, after the terminal obtains the implemented video stream, it establishes a connection with the cloud server through the RTCPeerConnection method, and obtains the two-way data transmission channel RTCDataChannel and downlink information. The downlink information is the handshake information when the terminal establishes a connection with the cloud server. According to the two-way data transmission channel and downlink information, the real-time video stream is transmitted to the cloud server based on the RTP and RTCP protocols.

[0110] RTP and RTCP protocols are the data packet formats transmitted by WebRTC on the Internet. They are defined by the WebRTC specification, including message formats and transmission rules. Among them, the RTP protocol is the basic protocol for streaming media transmission on the Internet, which specifies the standard data packet format for transmitting audio and video on the Internet. The RTP protocol itself only guarantees the transmission of real-time data, while the RTCP protocol is responsible for ensuring the transmission quality of streaming media and providing services such as flow control and congestion control. During the RTP session, the terminal and the cloud server periodically send RTCP messages to each other. The message contains statistical information such as data sending and receiving between the terminal and the cloud server, and the terminal and the cloud server can dynamically control the quality of streaming media transmission based on this.

[0111] In an optional embodiment, before transmitting the real-time video stream to the cloud server, the method further includes: obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds the preset transmission bit rate, actively dropping frames of the real-time video stream.

[0112] The video bit rate is the number of bits of data transmitted per unit time during data transmission, and the general unit is kbps, or kilobits per second. In layman's terms, it is the sampling rate. The higher the sampling rate per unit time, the higher the accuracy, and the closer the processed file is to the original file.

[0113] Specifically, the present application designs a frame drop logic in the WebRTC communication service protocol to track the video bit rate of the current real-time video stream. When the current video bit rate exceeds the preset transmission bit rate of the encoder, the terminal actively drops frames for the real-time video stream.

[0114] When the cumulative bit rate of real-time video stream transmission exceeds the preset cumulative transmission bit rate of the encoder, the terminal actively drops frames of the real-time video stream.

[0115] BufferFullnessSkip=Σ(CodedBitsPerFrame–TargetBitsPerFrame)

[0116] The BufferFullnessSkip frame drop logic checks the encoding of each frame. TargetBitsPerFrame indicates the preset bit rate for encoding each frame, and CodedBitsPerFrame indicates the current transmission bit rate for each frame. When the cumulative bit rate of the real-time video stream transmission exceeds the preset cumulative transmission bit rate (that is, the buffer is full), the terminal actively drops frames for the real-time video stream.

[0117] When the maximum bit rate of real-time video stream transmission exceeds the preset maximum transmission bit rate of the encoder, the terminal actively drops frames of the real-time video stream.

[0118] BufferMaxBRFullness=Σ(CodedBitsPerFrame–MaxBitsPerFrame)

[0119] The BufferMaxBRFullness frame loss logic checks each frame of video, and MaxBitsPerFrame indicates the preset maximum transmission bit rate.

[0120] The encoded audio and video data includes three types of frames: I-frame, P-frame, and B-frame. Active frame loss is for P-frames in real-time video streams, while I-frames will not be lost. I-frames are key frames, which can be understood as images. P-frames need to refer to the previous I-frame or B-frame to form a complete picture. Similarly, B-frames also need to refer to the previous I-frame or P-frame or the following P-frame to form a complete picture.

[0121] Active frame loss can occur not only during the video stream transmission process, but also during the encoding process, when the cloud server processes the real-time video stream, when the decoder loses frames, etc. Active frame loss can be sent during video stream acquisition and transmission, or when it is sent back after editing and synthesis.

[0122] In this embodiment, active P frame loss can remove non-important frames without affecting the quality, with low bit error rate, less packet loss, improved transmission efficiency and transmission success rate, and can also improve synthesis efficiency and high-precision editing effects.

[0123] In one embodiment, Figure 3 As shown, a video editing method is provided, which is applied to Figure 1 The following steps are used to illustrate the cloud server in the example:

[0124] Step 302: receiving a real-time video stream transmitted by a terminal.

[0125] Specifically, the cloud server receives the real-time video stream transmitted by the terminal through a two-way data transmission channel established with the terminal.

[0126] Step 304: Encode the real-time video stream to obtain and push audio and video data.

[0127] Specifically, Figure 4 As shown in the figure, the cloud server performs VP8 encoding on the video stream in the real-time video stream and Opus encoding on the audio in the real-time video stream to obtain the encoded audio and video data. The encoded audio and video data includes three frame types: I frame, P frame, and B frame. Figure 5 As shown in the figure, GOP (Group of Pictures) is a group of similar pictures consisting of an I frame and multiple P frames or B frames. The video frame GOP strategy affects the encoding quality. A GOP is a group of continuous pictures. A group is a data stream after a segment of image encoding, starting with an I frame and ending with the next I frame.

[0128] I frame: Intra-coded frame, which is a complete frame. P frame: Forward predictive coded frame, which is an incomplete frame, generated by referring to the previous I frame or P frame. B frame: Bidirectional predictive interpolation coded frame, generated by referring to the previous and next image frames. B frame depends on the most recent I frame or P frame before it and the most recent P frame after it. Audio and video decoders can directly decode I frames, but P frames and B frames must rely on I frames, or the previous and next P or B frames to be decoded.

[0129] Step 306, receiving editing materials and audio and video clips.

[0130] The editing operation and the audio and video clips are obtained by the terminal responding to the editing operation and acquiring the editing materials and the audio and video clips corresponding to the editing operation.

[0131] Specifically, the cloud server receives the editing materials, synthesis operation action instructions and audio and video clips sent by the terminal through a two-way data transmission channel.

[0132] Step 308: Generate a synthesis queue based on the editing materials and the audio and video clips.

[0133] Among them, the synthesis processing queue is a series of instruction set methods issued to the editing synthesis service thread.

[0134] Specifically, each instruction starts with a key I frame and the processing time is n I frame intervals, that is, the length of the audio and video clip. Each instruction header is the type of clip material to be processed, and the instruction payload is the processing object, such as video frames, audio tracks, transition filters, sticker animations, etc. The cloud server generates a synthesis queue for each audio and video clip through the editing and synthesis service thread according to the editing materials of each audio and video clip and the synthesis operation action instructions.

[0135] Step 310, the audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0136] Specifically, the cloud server clips each synthesis queue through the editing and synthesis service thread, synthesizes the edited synthesis queues, obtains and pushes the target clip video to the terminal.

[0137] In an optional embodiment, if Figure 6 As shown, the audio and video data are edited and synthesized according to the synthesis queue to obtain and push the target edited video to the terminal, including: reading the synthesis queue in sequence according to the time chain; editing and synthesizing the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain edited clips; assembling the edited clips to obtain and push the target edited video to the terminal.

[0138] Specifically, the time chain of the clip is obtained according to the timestamp (PTS (Presentation TimeStamp), the timestamp used for rendering) of each audio and video clip, and each synthesis queue is read in turn according to the time chain; the audio and video clips corresponding to the synthesis queue are edited and synthesized through the editing and synthesis service thread to obtain clips; the clips are assembled according to the time chain, and the target clip video is obtained and pushed to the terminal.

[0139] Clips (i.e. video frames) are displayed according to the PTS timestamp. DTS (Decoding Time Stamp) decoding timestamp is used for video decoding.

[0140] The terminal can also continuously respond to the user's editing operations, generate a cache queue based on new editing materials and audio and video clips, edit and synthesize the audio and video data according to the cache queue, and obtain and push the target editing video to the terminal.

[0141] In this embodiment, the user's multiple dynamic adjustments to the audio and video are saved by caching the intermediate state, and the target edited video can be directly edited and generated according to the cache queue, thereby improving the cloud editing efficiency.

[0142] In an optional embodiment, the audio and video clips corresponding to the synthesis queue are edited and synthesized according to the synthesis queue to obtain the edited clip, which includes: taking the first key frame in the audio and video clip as the starting point, editing and synthesizing the audio and video clips according to the editing materials in the synthesis queue through the editing synthesis service thread to obtain the edited clip.

[0143] Specifically, starting from the first key frame in the audio and video clip, the editing and synthesis service thread (i.e. Figure 6 The synthesis service in the synthesis queue) edits and synthesizes each frame in the audio and video clips according to the clip materials in the synthesis queue, so that each frame queue clip is encoded into a clip by FFMpeg.

[0144] Furthermore, FFMpeg is the core technology for processing video editing and synthesis. The editing and synthesis service thread first reads the queues of multiple video contents that need to be merged, the mixed audio contents, and the selected editing material contents according to the time chain (i.e., the synthesis queue), and then starts to merge the video content, audio content, material content, etc. respectively. The merging is implemented by encoding the video content and audio content based on FFmpeg, merging the encoded video and audio (edited fragments) into an integral media file (i.e., the target editing video), and finally outputting the merged target editing video to the outside. It can be output in mp4 format or in other formats, which is not limited to this embodiment.

[0145] In this embodiment, video editing and synthesis are realized on the cloud server side, which effectively transfers the performance pressure of the editing operation on the device to the cloud; the efficient communication service also reduces the system processing delay. The multi-threaded transmission of real-time video stream, editing materials, audio and video clips, and target editing videos is carried out through the WebRTC transmission protocol, which can be specifically realized through the MessageQueue of WebRTC.

[0146] For example, to facilitate understanding of the video editing method proposed in the present application, the video editing method provided in the embodiment of the present application is described in detail using the video editing process of interaction between the terminal and the cloud server, specifically including:

[0147] (1) The terminal obtains the real-time video stream and transmits it to the cloud server.

[0148] (2) The cloud server receives the real-time video stream transmitted by the terminal, encodes the real-time video stream, and obtains and pushes the audio and video data.

[0149] (3) The terminal receives audio and video data.

[0150] (4) The terminal responds to the editing operation, obtains the editing material and the audio and video clips corresponding to the editing operation, and sends the editing material and the audio and video clips to the cloud server.

[0151] (5) The cloud server receives editing materials and audio and video clips; generates a synthesis queue based on the editing materials and audio and video clips; reads the synthesis queue in sequence according to the time chain; takes the first key frame in the audio and video clip as the starting point, and uses the editing synthesis service thread to edit and synthesize the audio and video clips according to the editing materials in the synthesis queue to obtain an editing clip; assembles the editing clips to obtain and push the target editing video to the terminal.

[0152] (6) The terminal receives the target clip video.

[0153] like Figure 7 As shown, this application is divided into three core processing device units:

[0154] (1) Front-end interaction layer (implemented by the terminal): WebRTC audio and video acquisition, video frame preview (key frame preview of audio and video data), acquisition of editing operations (editing interaction instructions), etc.

[0155] (2) Cloud editing logic layer (implemented by cloud servers): editing material management, timeline management (timestamp of synthesis queue), cache intermediate state (cache queue storage), synthesis queue, etc.

[0156] (3) Cloud editing processing layer (implemented by cloud server): WebRTC communication service (real-time transmission service thread), editing synthesis (editing synthesis service thread), editing output, etc.

[0157] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0158] Based on the same inventive concept, the embodiment of the present application also provides a video editing device for implementing the video editing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more video editing device embodiments provided below can refer to the limitations of the video editing method above, and will not be repeated here.

[0159] In one embodiment, Figure 8 As shown, a video clipping device is provided, which is applied to a terminal, and includes: a video stream acquisition module 802, a terminal transmission module 804, a terminal receiving module 806, a terminal response module 808 and a target clip video receiving module 810, wherein:

[0160] The video stream acquisition module 802 is used to acquire the real-time video stream;

[0161] The terminal transmission module 804 is used to transmit the real-time video stream to the cloud server.

[0162] The terminal receiving module 806 is used to receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server.

[0163] The terminal response module 808 is used to respond to the editing operation, obtain the editing materials and audio and video clips corresponding to the editing operation, and send the editing materials and audio and video clips to the cloud server.

[0164] The target clipping video receiving module 810 is used to receive the target clipping video. The target clipping video is generated by the cloud server according to the clipping material and the audio and video clips. The synthesis queue is used to edit and synthesize the audio and video data according to the synthesis queue.

[0165] In an optional embodiment, the terminal transmission module 804 is also used to transmit the real-time video stream to the cloud server through the real-time transmission service thread.

[0166] In an optional embodiment, the video editing device further includes an active frame dropping module for obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds a preset transmission bit rate, the real-time video stream is actively dropped.

[0167] In one embodiment, Fig. 9 As shown, a video editing device is provided, which is applied to a cloud server, including: a cloud receiving module 902, an encoding module 904, a clip receiving module 906, a synthesis queue generating module 908 and a clip synthesis module 910, wherein:

[0168] The cloud receiving module 902 is used to receive the real-time video stream transmitted by the terminal;

[0169] The encoding module 904 is used to encode the real-time video stream to obtain and push audio and video data;

[0170] The clip receiving module 906 is used to receive clip materials and audio and video clips. The clip operation and the audio and video clips are obtained by the terminal responding to the clip operation and acquiring the clip materials and audio and video clips corresponding to the clip operation;

[0171] A synthesis queue generation module 908, used to generate a synthesis queue according to the editing materials and the audio and video clips;

[0172] The editing and synthesis module 910 is used to edit and synthesize the audio and video data according to the synthesis queue, obtain and push the target edited video to the terminal.

[0173] In one of the embodiments, the editing and synthesis module 910 is also used to read the synthesis queue in sequence according to the time chain; edit and synthesize the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain editing clips; assemble the editing clips to obtain and push the target editing video to the terminal.

[0174] In one of the embodiments, the editing and synthesis module 910 is further used to take the first key frame in the audio and video clip as the starting point, and to edit and synthesize the audio and video clip according to the editing materials in the synthesis queue through the editing and synthesis service thread to obtain a clip.

[0175] Each module in the above video editing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0176] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video editing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the computer device housing, or an external keyboard, touch pad or mouse, etc.

[0177] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0178] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0179] Get live video stream;

[0180] Transmit live video streams to cloud servers;

[0181] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0182] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0183] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0184] In one embodiment, when the processor executes the computer program, the following steps are also implemented: transmitting the real-time video stream to the cloud server includes: transmitting the real-time video stream to the cloud server through a real-time transmission service thread.

[0185] In one embodiment, the processor also implements the following steps when executing the computer program: before transmitting the real-time video stream to the cloud server, it also includes: obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds the preset transmission bit rate, actively dropping frames of the real-time video stream.

[0186] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0187] Receive real-time video stream transmitted by the terminal;

[0188] Encode the real-time video stream, obtain and push audio and video data;

[0189] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0190] Generate a synthesis queue based on editing materials and audio and video clips;

[0191] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0192] In one embodiment, the processor also implements the following steps when executing the computer program: editing and synthesizing the audio and video data according to the synthesis queue, and obtaining and pushing the target edited video to the terminal, including: reading the synthesis queue in sequence according to the time chain; editing and synthesizing the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain edited clips; assembling the edited clips to obtain and push the target edited video to the terminal.

[0193] In one embodiment, when the processor executes the computer program, the following steps are also implemented: according to the synthesis queue, the audio and video clips corresponding to the synthesis queue are edited and synthesized to obtain the edited clip, including: taking the first key frame in the audio and video clip as the starting point, through the editing and synthesis service thread, the audio and video clips are edited and synthesized according to the editing materials in the synthesis queue to obtain the edited clip.

[0194] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0195] Get live video stream;

[0196] Transmit live video streams to cloud servers;

[0197] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0198] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0199] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0200] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: transmitting the real-time video stream to the cloud server includes: transmitting the real-time video stream to the cloud server through a real-time transmission service thread.

[0201] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: before transmitting the real-time video stream to the cloud server, it also includes: obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds the preset transmission bit rate, actively dropping frames of the real-time video stream.

[0202] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0203] Receive real-time video stream transmitted by the terminal;

[0204] Encode the real-time video stream, obtain and push audio and video data;

[0205] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0206] Generate a synthesis queue based on editing materials and audio and video clips;

[0207] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0208] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: editing and synthesizing the audio and video data according to the synthesis queue, obtaining and pushing the target edited video to the terminal, including: reading the synthesis queue in sequence according to the time chain; editing and synthesizing the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain edited clips; assembling the edited clips to obtain and push the target edited video to the terminal.

[0209] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the audio and video clips corresponding to the synthesis queue are edited and synthesized according to the synthesis queue to obtain the edited clip, including: taking the first key frame in the audio and video clip as the starting point, editing and synthesizing the audio and video clips according to the editing materials in the synthesis queue through the editing synthesis service thread to obtain the edited clip.

[0210] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0211] Get live video stream;

[0212] Transmit live video streams to cloud servers;

[0213] Receive audio and video data, which is obtained by encoding the real-time video stream by the cloud server;

[0214] In response to the editing operation, the editing material and the audio and video clips corresponding to the editing operation are obtained, and the editing material and the audio and video clips are sent to the cloud server;

[0215] The target video clip is received. The cloud server generates a synthesis queue based on the clip material and the audio and video clips, and the audio and video data are edited and synthesized according to the synthesis queue.

[0216] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: transmitting the real-time video stream to the cloud server includes: transmitting the real-time video stream to the cloud server through a real-time transmission service thread.

[0217] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: before transmitting the real-time video stream to the cloud server, it also includes: obtaining the video bit rate of the real-time video stream; when the video bit rate exceeds the preset transmission bit rate, actively dropping frames of the real-time video stream.

[0218] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0219] Receive real-time video stream transmitted by the terminal;

[0220] Encode the real-time video stream, obtain and push audio and video data;

[0221] Receiving clipping materials and audio and video clips, where the clipping operation and the audio and video clips are obtained by the terminal responding to the clipping operation and acquiring the clipping materials and the audio and video clips corresponding to the clipping operation;

[0222] Generate a synthesis queue based on editing materials and audio and video clips;

[0223] The audio and video data are edited and synthesized according to the synthesis queue, and the target edited video is obtained and pushed to the terminal.

[0224] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: editing and synthesizing the audio and video data according to the synthesis queue, obtaining and pushing the target edited video to the terminal, including: reading the synthesis queue in sequence according to the time chain; editing and synthesizing the audio and video clips corresponding to the synthesis queue according to the synthesis queue to obtain edited clips; assembling the edited clips to obtain and push the target edited video to the terminal.

[0225] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the audio and video clips corresponding to the synthesis queue are edited and synthesized according to the synthesis queue to obtain the edited clip, including: taking the first key frame in the audio and video clip as the starting point, editing and synthesizing the audio and video clips according to the editing materials in the synthesis queue through the editing synthesis service thread to obtain the edited clip.

[0226] It should be noted that the video data involved in this application (including but not limited to data used for editing, data for storage, data for display, etc.) are all data authorized by the user or fully authorized by all parties.

[0227] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0228] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0229] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video editing method, characterized in that: Applied to a terminal, the method comprises: Get live video stream; Transmitting the real-time video stream to a cloud server; Receiving audio and video data, where the audio and video data is obtained by encoding the real-time video stream by the cloud server; In response to the editing operation, obtaining the editing material and the audio and video clips corresponding to the editing operation, and sending the editing material and the audio and video clips to the cloud server; Receive a target video clip, wherein the target video clip is obtained by assembling clip segments according to a time chain by the cloud server, wherein the clip segments are read from a synthesis queue in sequence according to a time chain of clips obtained by the cloud server according to the timestamp of each audio and video clip, and the audio and video clips corresponding to the synthesis queue are edited and synthesized by a clip synthesis service thread starting from the first key frame in the audio and video clip, and the synthesis queue is generated by the cloud server according to the clip materials, synthesis operation action instructions and the audio and video clips.

2. The method according to claim 1, characterized in that The transmitting the real-time video stream to the cloud server comprises: The real-time video stream is transmitted to the cloud server through a real-time transmission service thread.

3. The method according to claim 1, characterized in that Before transmitting the real-time video stream to the cloud server, the method further includes: Obtaining the video bit rate of the real-time video stream; When the video bit rate exceeds the preset transmission bit rate, the real-time video stream is actively dropped.

4. A video editing method, characterized in that: Applied to a cloud server, the method comprises: Receive real-time video stream transmitted by the terminal; Encode the real-time video stream to obtain and push audio and video data; Receiving clipping material and audio and video clips, wherein the clipping material and the audio and video clips are obtained by the terminal responding to a clipping operation and acquiring the clipping material and the audio and video clips corresponding to the clipping operation; A synthesis queue is generated according to the editing material, synthesis operation action instructions and the audio and video clips; the synthesis queue is read in sequence according to the time chain of the clips obtained according to the timestamp of each audio and video clip, and the audio and video clips corresponding to the synthesis queue are edited and synthesized through the editing synthesis service thread with the first key frame in the audio and video clip as the starting point to obtain a clip clip; the clip clips are assembled according to the time chain to obtain a target clip video, and the target clip video is pushed to the terminal.

5. A video editing device, characterized in that: Applied to a terminal, the device comprises: Video stream acquisition module, used to acquire real-time video stream; A terminal transmission module, used for transmitting the real-time video stream to a cloud server; A terminal receiving module, used for receiving audio and video data, wherein the audio and video data is obtained by encoding the real-time video stream by the cloud server; The terminal response module is used to respond to the editing operation, obtain the editing material and the audio and video clips corresponding to the editing operation, and send the editing material and the audio and video clips to the cloud server; A target clip video receiving module is used to receive a target clip video, wherein the target clip video is obtained by the cloud server assembling clip segments according to a time chain, wherein the clip segments are read from a synthesis queue in sequence according to a time chain of clips obtained by the cloud server according to the timestamp of each audio and video clip, and the audio and video clips corresponding to the synthesis queue are edited and synthesized by a clip synthesis service thread starting from the first key frame in the audio and video clip, and the synthesis queue is generated by the cloud server according to the clip materials, synthesis operation action instructions and the audio and video clips.

6. The device according to claim 5, characterized in that The terminal transmission module is also used to transmit the real-time video stream to the cloud server through a real-time transmission service thread.

7. The device according to claim 5, characterized in that The video editing device further includes a frame dropping module, which is used for: Obtaining the video bit rate of the real-time video stream; When the video bit rate exceeds the preset transmission bit rate, the real-time video stream is actively dropped.

8. A video editing device, characterized in that: Applied to a cloud server, the device comprises: A cloud receiving module is used to receive real-time video streams transmitted by terminals; An encoding module, used to encode the real-time video stream, obtain and push audio and video data; The clip receiving module is used to receive clip materials and audio and video clips, wherein the clip materials and the audio and video clips are obtained by the terminal response module responding to the clip operation and acquiring the clip materials and the audio and video clips corresponding to the clip operation; A synthesis queue generation module, used to generate a synthesis queue according to the editing material and the audio and video clips; The editing and synthesis module is used to read the synthesis queue in sequence according to the time chain of the clips obtained according to the timestamp of each audio and video clip, and take the first key frame in the audio and video clip as the starting point to edit and synthesize the audio and video clips corresponding to the synthesis queue through the editing and synthesis service thread to obtain a clip; assemble the clips according to the time chain to obtain a target clip video, and push the target clip video to the terminal.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images