Video processing method, device, equipment and storage medium

Abstract

The present application provides a video processing method, device, electronic device and computer-readable storage medium; the method comprises: obtaining a video file to be edited and at least one special effect file for adding special effects to the video file to be edited; performing tree-like editing processing based on the video file to be edited and at least one special effect file to obtain a tree-like rendering structure including the video file and the special effect file; performing rendering processing on the tree-like rendering structure based on the video layer included in the video file to be edited and at least one special effect file to obtain a rendered video file; performing audio synthesis processing based on the rendered video file to obtain a target video file with special effects added. Through the present application, the flexibility and efficiency of adding special effects can be improved.

Description

Technical Field

[0001] The present application relates to computer application technology, and in particular to a video processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, computer animation has been widely used in game production, animation production, animation special effects, etc. For example, after the user shoots a video, the corresponding processing device of the video social platform parses each video frame, and the user can specify the video frame to add the special effect, and then render it after adding the special effect to obtain the video with the special effect added.

[0003] However, the video social platform in the related art relies on the platform interface, which requires users to perform a lot of manual operations, increasing the complexity of adding animation effects. Summary of the invention

[0004] The embodiments of the present application provide a video processing method, device, electronic device and computer-readable storage medium, which can improve the flexibility and efficiency of adding special effects.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a video processing method, including:

[0007] Acquire a video file to be edited and at least one special effect file for adding special effects to the video file to be edited;

[0008] Performing tree-like editing processing based on the video file to be edited and the at least one special effect file to obtain a tree-like rendering structure including the video file and the special effect file;

[0009] Rendering the tree-like rendering structure based on the video file to be edited and the video layer included in the at least one special effect file to obtain a rendered video file;

[0010] Audio synthesis processing is performed based on the rendered video file to obtain a target video file with special effects added.

[0011] In the above technical solution, the audio synthesis processing is performed based on the rendered video file to obtain a target video file with special effects added, including:

[0012] Segmenting the audio file corresponding to the special effect file and the audio file corresponding to the video file to be edited to obtain a plurality of audio segments;

[0013] Dividing the multiple audio clips into corresponding audio tracks to obtain multiple audio tracks to which the audio clips are added, wherein the audio clips between the audio tracks do not overlap;

[0014] The multiple audio tracks to which the audio clips are added are merged to obtain a target video file to which special effects are added.

[0015] In the above technical solution, before dividing the multiple audio clips into corresponding audio tracks, the method further includes:

[0016] Decoding the multiple audio clips by decontainering to obtain the decoded audio clips;

[0017] Resampling the decoded audio segment to obtain the resampled audio segment;

[0018] Audio adjustment processing is performed on the resampled audio segment to obtain the audio segment for division processing.

[0019] In the above technical solution, before obtaining the video file to be edited and at least one special effect file for adding special effects to the video file to be edited, the method further includes:

[0020] Displaying multiple candidate video files in a human-computer interaction interface;

[0021] In response to a selection operation on the candidate video file, taking a selected portion of the candidate video file as the video file to be edited;

[0022] Acquire at least one special effect file for adding special effects to the video file to be edited.

[0023] In the above technical solution, the step of obtaining at least one special effects file for adding special effects to the video file to be edited includes:

[0024] Displaying a plurality of candidate special effect files in the human-computer interaction interface;

[0025] In response to a selection operation on the candidate special effect file, the selected portion of the candidate special effect file is used as at least one special effect file for adding special effects to the video file to be edited.

[0026] In the above technical solution, before displaying a plurality of candidate special effect files in the human-computer interaction interface, the method further includes:

[0027] The special effect files in the special effect file library are matched based on the video content of the video file to be edited, and the matched special effect files are used as the candidate special effect files.

[0028] In the above technical solution, before displaying a plurality of candidate special effect files in the human-computer interaction interface, the method further includes:

[0029] Get the frequency of selecting special effect files in the special effect file library;

[0030] When the selected frequency is greater than the selection threshold, the special effect file is used as the candidate special effect file.

[0031] The present application provides a video processing device, including:

[0032] An acquisition module, used for acquiring a video file to be edited and at least one special effect file for adding special effects to the video file to be edited;

[0033] An editing module, configured to perform tree-like editing processing based on the video file to be edited and the at least one special effect file, to obtain a tree-like rendering structure including the video file and the special effect file;

[0034] A rendering module, configured to render the tree-like rendering structure based on the video file to be edited and the video layer included in the at least one special effect file, to obtain a rendered video file;

[0035] The synthesis module is used to perform audio synthesis processing based on the rendered video file to obtain a target video file with animation special effects added.

[0036] In the above technical solution, the editing module is further used to divide the video file to be edited based on the at least one special effect file to obtain a plurality of video clips;

[0037] The multiple video clips and the at least one special effect file are combined to obtain a tree-like rendering structure including the video files and the special effect files.

[0038] In the above technical solution, the editing module is also used to obtain the time information of each special effect file, wherein the time information includes the starting time and the duration;

[0039] Based on the starting time and duration of each special effect file, the video segment corresponding to each special effect file and the video segment corresponding to the video filling in the video file to be edited are determined, wherein the video filling indicates that the special effect is not added to the video file to be edited.

[0040] In the above technical solution, the editing module is further used to perform the following processing on any of the at least one special effect file:

[0041] Determine the starting time of any of the special effect files as the starting time of the video segment corresponding to any of the special effect files in the video file to be edited;

[0042] Determine the duration of any of the special effect files as the duration of a video segment in the video file to be edited that corresponds to any of the special effect files;

[0043] Based on the starting time and duration of any of the special effects files, determine the ending time of any of the special effects files, and use the ending time of any of the special effects files as the starting duration of the video segment filled with the corresponding video in the video file to be edited;

[0044] The time interval between the end time of any of the special effect files and the start time of an adjacent special effect file is used as the duration of the video segment filled with the corresponding video in the video file to be edited.

[0045] In the above technical solution, the editing module is further used to determine the rendering size of the video file to be edited based on the header file and the end file in the special effect file;

[0046] Determine a rendering root node based on the rendering size of the video file to be edited;

[0047] Taking each of the video clips and each of the special effects files as a rendering leaf node;

[0048] Based on the rendering root node and the rendering leaf nodes, a tree rendering structure including the video file and the special effect file is determined.

[0049] In the above technical solution, the editing module is also used to create an add layer interface based on the rendering root node;

[0050] Through the adding layer interface, the rendering leaf nodes are added sequentially on the rendering root node according to the time information of the rendering leaf nodes, so as to obtain a tree-like rendering structure including the video file and the special effect file.

[0051] In the above technical solution, the rendering module is further used to traverse the tree-like rendering structure to obtain rendering leaf nodes of the tree-like rendering structure;

[0052] When the rendering leaf node represents a video segment of the video file, rendering processing is performed on the video segment of the video file to obtain a rendered video corresponding to the video file;

[0053] When the rendering leaf node represents the special effect file, rendering is performed on the video layer included in the special effect file to obtain a rendered video corresponding to the special effect file;

[0054] The rendered video corresponding to the video file and the rendered video corresponding to the special effect file are combined and processed to obtain a rendered video file.

[0055] In the above technical solution, the rendering module is further used to perform format conversion processing on the video segment of the video file to obtain a video layer corresponding to the video segment;

[0056] Animation rendering is performed based on the video layer corresponding to the video clip to obtain a rendered video corresponding to the video file.

[0057] In the above technical solution, the rendering module is also used to obtain a target video segment corresponding to the special effect file in the video file to be edited;

[0058] Filling the target video clip into the placeholder image of the video layer included in the special effect file to obtain a video layer filled with the target video clip;

[0059] Animation rendering is performed on the video layer filled with the target video clip to obtain a rendered video corresponding to the special effects file.

[0060] In the above technical solution, the rendering module is also used to obtain a target video segment corresponding to the special effect file in the video file to be edited;

[0061] Filling the target video clip into the corresponding layer to obtain a layer filled with the target video clip;

[0062] Filling the video layer included in the special effect file into the layer filled with the target video clip to obtain a layer filled with the special effect file;

[0063] Animation rendering is performed on the layer filled with the special effect file to obtain a rendering video corresponding to the special effect file.

[0064] In the above technical solution, before rendering the tree-like rendering structure based on the video file to be edited and the video layer included in the at least one special effect file to obtain the rendered video file, the device further includes:

[0065] The decoding module is used to call a decoding interface to decode the at least one special effect file and the video file to be edited, so as to obtain the special effect file and the video file to be edited for the rendering process.

[0066] In the above technical solution, the decoding module is further used to call the system decoding interface to decode the at least one special effect file and the video file to be edited when the system supports hardware decoding, so as to obtain the special effect file and the video file to be edited for the rendering process;

[0067] When the system does not support hardware decoding, the decoding interface built into the software development kit is called to decode the at least one special effect file and the video file to be edited to obtain the special effect file and the video file to be edited for the rendering process.

[0068] In the above technical solution, the synthesis module is further used to segment the audio file corresponding to the special effect file and the audio file corresponding to the video file to be edited to obtain multiple audio clips;

[0069] Dividing the multiple audio clips into corresponding audio tracks to obtain multiple audio tracks to which the audio clips are added, wherein the audio clips between the audio tracks do not overlap;

[0070] The multiple audio tracks to which the audio clips are added are merged to obtain a target video file to which special effects are added.

[0071] In the above technical solution, the synthesis module is further used to decode the multiple audio clips by decontainering to obtain the decoded audio clips;

[0072] Resampling the decoded audio segment to obtain the resampled audio segment;

[0073] Audio adjustment processing is performed on the resampled audio segment to obtain the audio segment for division processing.

[0074] In the above technical solution, the acquisition module is also used to display multiple candidate video files in the human-computer interaction interface;

[0075] In response to a selection operation on the candidate video file, taking a selected portion of the candidate video file as the video file to be edited;

[0076] Acquire at least one special effect file for adding special effects to the video file to be edited.

[0077] In the above technical solution, the acquisition module is also used to display multiple candidate special effect files in the human-computer interaction interface;

[0078] In response to a selection operation on the candidate special effect file, the selected portion of the candidate special effect file is used as at least one special effect file for adding special effects to the video file to be edited.

[0079] In the above technical solution, the acquisition module is also used to match the special effect files in the special effect file library based on the video content of the video file to be edited, and use the matched special effect files as the candidate special effect files.

[0080] In the above technical solution, the acquisition module is also used to obtain the frequency of the special effect files in the special effect file library being selected;

[0081] When the selected frequency is greater than the selection threshold, the special effect file is used as the candidate special effect file.

[0082] An embodiment of the present application provides an electronic device for video processing, the electronic device comprising:

[0083] A memory for storing executable instructions;

[0084] The processor is used to implement the video processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0085] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the video processing method provided in the embodiment of the present application.

[0086] The embodiments of the present application have the following beneficial effects:

[0087] By performing tree-like editing processing on the video files to be edited and the special effects files, a tree-like rendering structure for rendering is obtained, so that the special effects files are automatically organized through the tree-like rendering structure to improve the efficiency of adding special effects; the tree-like rendering structure is rendered based on the video layers included in the special effects files, so that the platform interface is decoupled through the video layers, and cross-platform applications can be realized to improve the flexibility of adding special effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a breakdown diagram of the video post-editing template provided by the related technology;

[0089] Figure 2A It is a schematic diagram of adding special effects to video images provided by related technologies;

[0090] Figure 2B It is a schematic diagram of adding special effects to video images provided by related technologies;

[0091] Figure 3is a schematic diagram of an application scenario of a distributed file system 10 provided in an embodiment of the present application;

[0092] Figure 4 is a structural schematic diagram of an electronic device for video processing provided in an embodiment of the present application;

[0093] Figure 5A-Figure 5C It is a flowchart of a video processing method provided in an embodiment of the present application;

[0094] Figure 6 is a schematic diagram of video division provided in an embodiment of the present application;

[0095] Figure 7 It is a schematic diagram of a post-editing template of a video publisher provided in an embodiment of the present application;

[0096] Figure 8 is a schematic diagram of a video file with special effects applied provided by an embodiment of the present application;

[0097] Fig. 9 It is a flowchart of a PAG-based video post-editing template rendering solution provided in an embodiment of the present application;

[0098] Fig.10 It is a schematic diagram of material division provided in an embodiment of the present application;

[0099] Fig.11 is a schematic diagram of a tree rendering structure provided by an embodiment of the present application;

[0100] Fig.12 is a schematic diagram of a large sticker provided in an embodiment of the present application;

[0101] Fig.13 is a schematic diagram of a small sticker provided in an embodiment of the present application;

[0102] Fig.14 is a schematic diagram of audio track division provided in an embodiment of the present application;

[0103] Fig.15 It is a schematic diagram of the audio mixing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0105] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0106] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0107] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0108] 1) Client: An application running in a terminal device to provide various services, such as a video player client, a game client, etc.

[0109] 2) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0110] 3) Non-linear special effects production software (AE, Adobe After Effects): a graphic video processing software, a layer-type post-production software that can efficiently and accurately create countless eye-catching dynamic graphics and stunning visual effects, covering text effects, particle effects, lighting effects, simulation effects, color grading techniques and advanced special effects in film and television special effects production.

[0111] 4) Special effects file: After the designer designs the animation special effects through the graphic video processing software, the designer exports the animation format file through the export plug-in. For example, after the designer designs the animation through AE, the designer exports the AE project file to the animation format file.

[0112] 5) Portable Animated Graphics (PAG): The name of a sticker animation implementation solution. The format of the animation file obtained based on the PAG solution is the PAG format. The PAG animation file uses a dynamic bit storage technology with an extremely high compression rate. Images, sounds, videos and other resources can be directly integrated into a single file. In the sticker animation solution based on PAG, designers can use AE software to design the animation effects required by the product, and then use the PAGExporter export plug-in of the PAG solution to read the animation feature data in the AE project file. According to specific needs, you can choose one of the vector export, bitmap sequence frame export, and video sequence frame export methods to export a PAG format binary file. When displayed on the client, the client can decode the PAG binary file used through the PAG SDK, render it through the rendering module, and then present it on the Android platform, iOS platform, and web side respectively.

[0113] Compared with Airbnb's open source Lottie animation implementation, the sticker animation solution based on PAG significantly reduces development costs. In terms of describing the same animation content, the vector solution has smaller files, supports richer animation effects, supports rendering multi-level cache drawing, supports text editing and replacement of image content, and the bitmap sequence frame solution and video sequence frame solution support AE unlimited special effects and are more powerful. The bitmap sequence frame solution is mainly used on the web side, and the video sequence frame solution focuses on the mobile side. In the PAG data structure, a PAGFile contains multiple compositions (Composition), which can be divided into three categories: VectorComposition, BitmapComposition and VideoComposition, which correspond to the three export methods of PAG files: vector export, bitmap sequence frame export and video sequence frame export.

[0114] Among them, vector export is the restoration of the AE animation layer structure. In the specific representation of VectorComposition, it can include composition attributes (CompositionAttributes) and layer attributes. CompositionAttributes is used to represent the basic attributes of the composition, such as ID, width, height, duration, frame rate, background color, etc. The layer information contained in the layer attributes corresponds to the AE layer structure one by one. For example, a composition supports virtual objects (Null Object), solid color layers (SolidLayer), text layers (TextLayer), shape layers (ShapeLayer), image layers (ImageLayer), pre-composite layers (PreComposeLayer), etc. Taking the solid color layer (SolidLayer) as an example, it can be further subdivided to include the width, height, color and layer attributes of the layer. The layer attributes include layer duration, start time, stretching parameters, mask (Mask), layer effects (Effects), layer style (LayerStyle), transformation (Transform), etc. The transformation (Transform) can be further subdivided into anchor point, position, x / y direction position, scaling factor, rotation factor, transparency, and the layer structure in the AE animation.

[0115] Bitmap sequence frame export is to capture each frame of AE animation into a picture. In the specific representation of Bitmap Composition, it includes CompositionAttributes and BitmapSequence arrays. BitmapSequence includes the width (width), height (height), frame rate (frameRate), whether it is a key frame (isKeyFrame), position information (x, y) of the bitmap and binary data (ByteData) of the bitmap.

[0116] The video sequence frame compresses the captured images into video format based on the bitmap sequence frame. The specific representation of VideoComposition includes CompositionAttributes, whether it contains transparent video (hasAlpha) and VideoSequence array. VideoSequence includes the width, height, frame rate, whether it is a key frame and binary data of the bitmap.

[0117] 6) Video layer: In essence, it is an image layer. The difference is that ordinary image layers only need to be replaced once and take effect during the entire rendering process, while video layers need to replace image information frame by frame during the rendering process. Adding animation effects to videos can also be understood as adding video content to sticker animations. When adding video content, you need to know at which time point and position in the video to add the video content. The video layer serves this purpose, that is, the video layer is a layer in the animation file that is used to carry the image content in the video file.

[0118] 7) Video decontainer: Video decontainer is used to decapsulate video files, or called demultiplexing (demux). In fact, each video file contains multiple tracks. The image track is the picture you see, the audio track is the sound you hear, and the subtitle track is the subtitles displayed. Since video files need to be transmitted, the resources of different tracks must be gathered together and then transmitted to the destination through the network or other means. The process of multiplexing (mux) is to combine the resources of multiple tracks into a container. Therefore, after obtaining the video file, it is necessary to demultiplex, that is, to decompose the multiple tracks in the video file.

[0119] 8) Decoder: During video transmission, the video data will be encoded through encoding to reduce the size of the data packet. Therefore, after the actual use of video decoding, the video decoder is also required to decode the image track data.

[0120] 9) FFmpeg (Fast Forward Mpeg): It is an open source computer program that can be used to record, convert digital audio and video, and convert them into streams. FFmpeg has built-in video decontainer and decoder, so video decontainer and video decoder can be implemented through FFmpeg.

[0121] The video processing method is based on the video timeline and is driven by the video decoder on the platform. Figure 1 As shown in the figure, a video post-editing template consists of three filler materials, with transition effects added to the materials. In addition to the opening and ending special effects, there are also special effects such as speed change and look-up table (LUT) filter. At the same time, background music needs to be added when the video is finally synthesized. The specific rendering process is as follows:

[0122] 1) Drive video decoding through the video decoder on the platform side, using this as the main timeline. The Android side uses MediaCodec to implement video decoding, and the iOS side uses VideoToolbox to implement video decoding.

[0123] 2) Special effects processing and filling After the video is decoded, you need to add special effects to the video screen. There are two ways: Method 1, such as Figure 2A As shown, if the size of the special effect screen is small, the special effect screen is directly rendered on the surface (Surface) drawn by the video screen; Method 2, such as Figure 2B As shown, the video screen is used as input to add special effects. At this time, it is necessary to decode the video at a specific time point, render it through special effects rendering, obtain the special effects screen content and then render it. In this process, there will be a lot of filling business logic.

[0124] In the above method, the embodiment of the present application further provides a video processing method, device, electronic device and computer-readable storage medium, which can improve the flexibility and efficiency of adding special effects.

[0125] The video processing method provided in the embodiment of the present application can be implemented independently by a terminal or a server; it can also be implemented jointly by a terminal and a server. For example, the terminal independently undertakes the video processing method described below, or the terminal sends a request for adding a special effects file to the server, and the server executes the video processing method according to the received request for adding the special effects file, and renders the tree-like rendering structure based on the video layer included in the special effects file and the video file to be edited, so as to realize the function of adding the special effects file to the video file to be edited.

[0126] The electronic device for video processing provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart TV, smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0127] See also Figure 3 , Figure 3 It is a schematic diagram of an application scenario of the distributed file system 10 provided in an embodiment of the present application. The terminal 200 is connected to the server 100 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0128] A terminal (running a client, such as a video client, a short video client, etc.) can be used to obtain a request to add a special effect file. For example, when a user opens a short video client running on the terminal, selects a video to be edited from the album, and selects at least one special effect template (special effect file) from a special effect template set, the terminal automatically obtains a request to add a special effect file (including the video file to be edited and at least one special effect file).

[0129] In some embodiments, a video processing plug-in may be implanted in the client running in the terminal 200 to implement the video processing method locally on the client. After the terminal 200 obtains the request to add a special effect file, the video processing plug-in is called to perform tree-like editing processing based on the video file to be edited and at least one special effect file to obtain a tree-like rendering structure, and the tree-like rendering structure is rendered based on the video layer included in the special effect file and the video file to be edited to implement the function of adding a special effect file to the video file to be edited. The target video file with the special effect added can also be applied across platforms to respond to the request to add the special effect file.

[0130] In some embodiments, after the terminal 200 obtains a request to add a special effects file, the video processing interface of the server 100 is called. The server 100 performs tree-like editing processing based on the video file to be edited and at least one special effects file to obtain a tree-like rendering structure, and renders the tree-like rendering structure based on the video layer included in the special effects file and the video file to be edited to obtain a target video file with special effects added, and sends the target video file with special effects added to the terminal 200 to implement the function of adding a special effects file to the video file to be edited. The target video file with special effects added can also be applied across platforms to respond to the request to add a special effects file.

[0131] In some embodiments, multiple servers may form a blockchain, and server 100 is a node on the blockchain. An information connection may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connection.

[0132] Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include the underlying blockchain platform, the platform product service layer, and the application service layer. Among them, the data related to the graph data processing method provided in the embodiment of the present application (such as the logic of graph data processing and the image presented after aggregation) can be stored on the blockchain.

[0133] The structure of the electronic device for video processing provided by the embodiment of the present application is described below. Figure 4 , Figure 4 is a structural diagram of an electronic device 500 for video processing provided in an embodiment of the present application, and takes the electronic device 500 as a terminal as an example. Figure 4 The electronic device 500 for video processing shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 4 Various buses are labeled as bus system 540 .

[0134] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0135] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0136] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0137] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0138] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0139] In some embodiments, the video processing device provided in the embodiments of the present application can be implemented in software. The video processing device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including applications, software, software modules, scripts or codes.

[0140] Figure 4 A video processing device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, and includes a series of modules, including an acquisition module 5551, an editing module 5552, a rendering module 5553, a synthesis module 5554, and a decoding module 5555. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0141] As mentioned above, the video processing method provided in the embodiments of the present application can be implemented by various types of electronic devices, such as terminals. Figure 5A , Figure 5A is a flow chart of a video processing method provided in an embodiment of the present application, combined with Figure 5A The steps shown are explained.

[0142] In the following steps, the video file to be edited (ie, the video file to be edited) may be a locally stored video (eg, a video stored in an album) or a currently shot video (eg, a video currently shot by a camera).

[0143] In step 101, a video file to be edited and at least one special effect file for adding special effects to the video file to be edited are obtained.

[0144] As an example of obtaining video files and special effects files, when the user opens the video editing client running on the terminal, selects the video to be edited from the album, and selects at least one special effects file from the special effects template set, the terminal automatically obtains an addition request for the special effects file (including the video file to be edited and at least one special effects file), and sends the addition request for the special effects file to the server. The server parses the addition request for the special effects file to obtain the video file to be edited and at least one special effects file.

[0145] In some embodiments, before obtaining the video file to be edited and at least one special effects file for adding special effects to the video file to be edited, the method also includes: displaying multiple candidate video files in a human-computer interaction interface; in response to a selection operation on a candidate video file, using the selected part of the candidate video file as the video file to be edited; and obtaining at least one special effects file for adding special effects to the video file to be edited.

[0146] For example, multiple candidate video files are first displayed in the human-computer interaction interface. The multiple candidate video files can be video files in an album (for adding special effects). Through the user's selection operation on the candidate video files displayed in the human-computer interaction interface, the candidate video files selected by the user are determined as the video files to be edited, and based on the video files to be edited, at least one special effects file for adding special effects to the video files to be edited is determined.

[0147] In some embodiments, obtaining at least one special effects file for adding special effects to a video file to be edited includes: displaying a plurality of candidate special effects files in a human-computer interaction interface; and in response to a selection operation on a candidate special effects file, using a selected portion of the candidate special effects file as at least one special effects file for adding special effects to the video file to be edited.

[0148] For example, after determining the video file to be edited, the special effects files in the special effects file library are matched based on the video content of the video file to be edited through artificial intelligence technology, and the matched special effects files are used as candidate special effects files to display multiple candidate special effects files in the human-computer interaction interface; or the frequency of special effects files in the special effects file library being selected by sample users (users in the client other than the current user) or the current user is obtained. When the frequency of selection is greater than the selection threshold (set according to actual needs), the special effects file is used as a candidate special effects file, and multiple candidate special effects files are displayed in the human-computer interaction interface.

[0149] As an implementation method, after multiple candidate special effects files are determined through artificial intelligence technology or selected frequency, the styles of the multiple candidate special effects files can be determined to display a fixed number of special effects files of different styles in the human-computer interaction interface. For example, 10 candidate special effects files are determined, including 3 anime-style special effects files, 3 ancient-style special effects files, and 4 modern-style special effects files, then 1 anime-style special effects file, 1 ancient-style special effects file, and 1 modern-style special effects file are displayed in the human-computer interaction interface.

[0150] As an implementation method, after multiple candidate special effects files are determined through artificial intelligence technology or selected frequency, the popularity of the multiple candidate special effects files can be determined, and based on the popularity of the multiple candidate special effects files, the multiple candidate special effects files are sorted in descending order, and some of the special effects files that are ranked first in descending order are displayed in the human-computer interaction interface.

[0151] In step 102, tree-like editing processing is performed based on the video file to be edited and at least one special effect file to obtain a tree-like rendering structure including the video file and the special effect file.

[0152] For example, after obtaining the video file to be edited and at least one special effects file, the video file to be edited and at least one special effects file are subjected to tree-like editing processing through the PAG tree-like editing function to form rendering leaf nodes corresponding to the video file and the special effects file, and a tree-like rendering structure is constructed based on the rendering leaf nodes, so as to automatically organize the special effects files through the tree-like rendering structure to improve the efficiency of adding special effects, so as to facilitate subsequent animation rendering based on the tree-like rendering structure to obtain a rendered video file.

[0153] See also Figure 5B , Figure 5B is an optional flow chart of the video processing method provided in the embodiment of the present application. Figure 5B Show Figure 5A Step 102 can also be implemented through steps 1021-1022: in step 1021, the video file to be edited is divided based on at least one special effect file to obtain multiple video clips; in step 1022, the multiple video clips and at least one special effect file are combined to obtain a tree-like rendering structure including video files and special effect files.

[0154] For example, the specific division processing process is as follows: obtain the time information of each special effects file, wherein the time information includes the starting time and duration; based on the starting time and duration of each special effects file, determine the video segment corresponding to each special effects file in the video file to be edited and the video segment corresponding to the video filling, wherein the video segment corresponding to the video filling is the video segment without adding special effects in the video file to be edited, and the video filling indicates that no special effects are added in the video file to be edited.

[0155] Continuing with the above example, based on the starting moment and duration of each special effects file, the video segment corresponding to each special effects file and the video segment filled in the video file to be edited are determined, including: performing the following processing for any special effects file in at least one special effects file: determining the starting moment of any special effects file as the starting moment of the video segment corresponding to any special effects file in the video file to be edited; determining the duration of any special effects file as the duration of the video segment corresponding to any special effects file in the video file to be edited; determining the ending moment of any special effects file based on the starting moment and duration of any special effects file, and taking the ending moment of any special effects file as the starting duration of the video segment filled in the video file to be edited; taking the time interval between the ending moment of any special effects file and the starting moment of an adjacent special effects file as the duration of the video segment filled in the video file to be edited.

[0156] For example, Figure 6 As shown, for special effect 3, the starting time of special effect 3 is taken as the starting time of video clip 1 (the video clip corresponding to special effect 3), and the duration of special effect 3 is taken as the duration of video clip 1. Based on the starting time and duration of special effect 3, the ending time of special effect 3 is determined, and the ending time of special effect 3 is taken as the starting duration of video clip 2 (the video clip filled with the corresponding video in the video file to be edited, that is, the video clip without special effects added in the video file to be edited), and the time interval between the ending time of special effect 3 and the starting time of special effect 4 is taken as the duration of video clip 2.

[0157] For example, the specific combination processing process is as follows: based on the header file and the tail file in the special effects file, determine the rendering size of the video file to be edited; based on the rendering size of the video file to be edited, determine the rendering root node; use each video clip and each special effects file as a rendering leaf node; based on the rendering root node and the rendering leaf nodes, determine the tree rendering structure including the video file and the special effects file.

[0158] Continuing with the above example, Figure 6 As shown, when special effect 1 is the opening special effect and special effect 4 is the ending special effect, the number of video clips between special effect 1 and special effect 4 is used as the rendering size of the video file to be edited, and the rendering root node (Root PAGComposition) is determined based on the rendering size of the video file to be edited. An add layer interface (addLayer interface) is created based on the rendering root node. Through the add layer interface, rendering leaf nodes are added sequentially to the rendering root node according to the time information of the rendering leaf nodes, so as to obtain a tree-like rendering structure including video files and special effect files.

[0159] In step 103, the tree-like rendering structure is rendered based on the video file to be edited and the video layer included in at least one special effect file to obtain a rendered video file.

[0160] For example, after determining the tree-like rendering structure, the tree-like rendering structure is rendered based on the PAG video layer included in the special effects file and the video file to be edited to obtain a rendered video file. Since the PAG video layer can realize cross-platform decoding of the video, the video file obtained after rendering through the PAG video layer can be applied across platforms.

[0161] In some embodiments, a tree-like rendering structure is rendered based on a video file to be edited and a video layer included in at least one special effects file to obtain a rendered video file, including: traversing the tree-like rendering structure to obtain rendering leaf nodes of the tree-like rendering structure; when a rendering leaf node represents a video segment of a video file, rendering the video segment of the video file to obtain a rendered video of the corresponding video file; when a rendering leaf node represents a special effects file, rendering the video layer included in the special effects file to obtain a rendered video of the corresponding special effects file; and combining the rendered video of the corresponding video file and the rendered video of the corresponding special effects file to obtain a rendered video file.

[0162] For example, since the rendering leaf node in the tree-like rendering structure may be a special effects file or a video clip of a video file to be edited, when the rendering leaf node is a video clip of a video file to be edited, the video clip of the video file is rendered to obtain a rendered video of the corresponding video file; when the rendering leaf node is a special effects file, the video layer included in the special effects file is rendered to obtain a rendered video of the corresponding special effects file; finally, the two rendered videos are combined according to the time information of the rendering leaf node to obtain a rendered video file, so that the audio file can be subsequently synthesized based on the rendered video file.

[0163] Continuing with the above example, the video segments of the video file are rendered to obtain a rendered video of the corresponding video file, including: format conversion of the video segments of the video file to obtain video layers of the corresponding video segments; animation rendering based on the video layers of the corresponding video segments to obtain a rendered video of the corresponding video file.

[0164] Continuing with the above example, there are two ways to render the video layer included in the special effects file, one is the large sticker method, and the other is the small sticker method.

[0165] Among them, the processing process of the large sticker method is as follows: obtain the target video clip corresponding to the special effects file in the video file to be edited; fill the target video clip into the placeholder image of the video layer included in the special effects file to obtain a video layer filled with the target video clip; perform animation rendering processing on the video layer filled with the target video clip to obtain a rendered video corresponding to the special effects file.

[0166] For example, a time segment (target video segment) is captured from a video to construct a PAGMovie, which is filled into a placeholder image of a video layer to obtain a video layer filled with the target video segment, and then animation rendering is performed on the video layer filled with the target video segment to obtain a rendered video corresponding to the special effects file.

[0167] Among them, the processing process of the small sticker method is as follows: obtain the target video clip corresponding to the special effects file in the video file to be edited; fill the target video clip into the corresponding layer to obtain a layer filled with the target video clip; fill the video layer included in the special effects file into the layer filled with the target video clip to obtain a layer filled with the special effects file; perform animation rendering processing on the layer filled with the special effects file to obtain a rendered video corresponding to the special effects file.

[0168] For example, a time segment (target video segment) is captured from a video to construct a PAGMovie, which is then filled with the corresponding constructed layer (PAGImageLayer), the constructed PAGImageLayer is added to the Root PAGComposition, and then the special effects file is added. Finally, the layer filled with the special effects file is animated and rendered to obtain a rendered video of the corresponding special effects file.

[0169] In some embodiments, a tree-like rendering structure is rendered based on the video layers included in the video file to be edited and at least one special effects file. Before obtaining the rendered video file, a decoding interface is called to decode at least one special effects file and the video file to be edited to obtain the special effects file for rendering and the video file to be edited.

[0170] For example, when the system supports hardware decoding, the system decoding interface is called to decode at least one special effects file and the video file to be edited, so as to obtain the special effects file for rendering and the video file to be edited; when the system does not support hardware decoding, the built-in decoding interface of the software development kit is called to decode at least one special effects file and the video file to be edited, so as to obtain the special effects file for rendering and the video file to be edited.

[0171] Continuing with the above examples, some system platforms can support hardware decoding. These system platforms provide hardware decoding interfaces, such as the interface VideoToolbox on iOS or Mac, and the interface MediaCodec on Android. Hardware decoding can fully utilize the image processor for decoding, and the decoding efficiency and time required are both low. Therefore, when the system platform can support hardware decoding, the decoding interface of the system platform can be called first. Therefore, in actual applications, it can be determined whether the current system supports hardware decoding. When it is determined that the current system supports hardware decoding, the system decoding interface is called to decode the video using hardware decoding; when it is determined that the current system does not support hardware decoding, such as the Linux system, the SDK built-in decoding interface can be called to decode the video using software decoding.

[0172] In step 104, audio synthesis processing is performed based on the rendered video file to obtain a target video file with special effects added.

[0173] For example, when the special effect file includes an audio file (sound effect), audio synthesis processing is performed based on the special effect file including the audio file and the rendered video file to obtain a target video file merged with the sound effect.

[0174] See also Figure 5C , Figure 5C is an optional flow chart of the video processing method provided in the embodiment of the present application. Figure 5C Show Figure 5A Step 104 can also be implemented through steps 1041-1042: in step 1041, the audio file corresponding to the special effect file and the audio file corresponding to the video file to be edited are segmented to obtain multiple audio clips; in step 1042, the multiple audio clips are divided into corresponding audio tracks to obtain multiple audio tracks with added audio clips, wherein the audio clips between the audio tracks do not overlap; in step 1043, the multiple audio tracks with added audio clips are merged to obtain the target video file with added special effects.

[0175] For example, the audio file corresponding to the video file to be edited includes the original sound of the video file to be edited and the background music added to the video file to be edited. Fig.14As shown, the audio file corresponding to the special effect file and the audio file corresponding to the video file to be edited are segmented to obtain multiple audio clips, such as audio clip 1, audio clip 2, audio clip 3 and audio clip 4, and audio clip 1 and audio clip 2 are divided into the corresponding audio track 1, and audio clip 3 and audio clip 4 are divided into the corresponding audio track 2 to obtain audio track 1 and audio track 2 with added audio clips, wherein the audio clips between audio track 1 with added audio clips and audio track 2 with added audio clips do not overlap, and audio track 1 and audio track 2 with added audio clips are merged to obtain the target video file with added special effects.

[0176] In some embodiments, before dividing the multiple audio clips into corresponding audio tracks, the method also includes: decoding the multiple audio clips by decontainering to obtain decoded audio clips; resampling the decoded audio clips to obtain resampled audio clips; and performing audio adjustment processing on the resampled audio clips to obtain audio clips for division processing.

[0177] For example, after completing the division of the audio tracks, since the formats of different audio files are different, a single audio segment is decoded by the container, the audio segment is resampled and converted into a unified format, and then the audio speed or volume is adjusted for the audio segment.

[0178] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0179] In related technologies, the video post-editing template rendering implementation solution is based on the video timeline and driven by the video decoder on the platform. A video post-editing template consists of three filler materials, with transition effects added to the materials. In addition to the opening and ending special effects, there are also special effects such as speed change and LUT filters. At the same time, background music needs to be added when the video is finally synthesized.

[0180] Although the relevant technology can meet the needs of products and users, the applicant found the following problems:

[0181] 1) The video encoding and decoding module relies on the platform interface and cannot be cross-platform. When configuring the template, it relies on mobile preview and does not support desktop preview, resulting in low production efficiency;

[0182] 2) There are a lot of business logics in the rendering process, such as video speed change, video special effects addition, LUT filter effect addition, etc., which need to be implemented separately by the platform.

[0183] In order to solve the above problems, an embodiment of the present application proposes a video processing method, in which the entire rendering chain is controlled by PAG instead of a video decoder, which can solve the problem of video decoding's dependence on the platform end and realize the cross-platform SDK. After the business logic such as adding video special effects is implemented, it can be used on each end. At the same time, desktop preview can also be realized to improve the production efficiency of templates.

[0184] like Figure 7 As shown, the embodiment of the present application can be applied to various video publisher post-editing templates, wherein 701 is a special effect template, and the application is as follows Figure 7 After using the special effects template shown in the figure, you can get the following Figure 8 The video file shown has special effects applied, wherein 801 is a text special effect.

[0185] like Fig. 9 As shown, the video processing method provided by the embodiment of the present application is specifically described below, and its implementation method is as follows:

[0186] (1) PAG video layer and PAG tree editing

[0187] The video processing method proposed in the embodiment of the present application relies on PAG video layers and PAG tree editing.

[0188] In response to the problem that video decoding depends on the platform, the PAG video layer in the embodiment of the present application realizes cross-platform video decoding. The video decontainer is decoded through FFmpeg, the iOS side implements decoding through VideoToolBox, and the Android side implements decoding through MediaCodec. In view of the diversity of Android models, when hardware decoding fails, FFmpeg is started for software decoding. At the same time, time mapping is added to support variable speed control.

[0189] In terms of rendering chain organization, based on the PAG tree editing function, the free combination of multiple PAG files can be achieved, and the PAG rendering tree can add PAG files as rendering leaf nodes.

[0190] (2) User material classification

[0191] like Fig.10 As shown in the figure, the dotted line divides the three user materials into multiple segments for use by the PAG SDK to form the following Fig.11 The tree rendering structure shown.

[0192] like Fig.10As shown, users can pass in multiple video materials, and can add titles, special effects, transitions, speed changes, end effects, LUT filters and other information to the video materials. In terms of specific implementation, the user's materials must first be divided to obtain the minimum rendering node. The video clips are rendered across platforms based on the PAG video layer. The organization of all rendering nodes relies on the PAG tree editing technology, and then a PAG rendering tree (tree rendering structure) is formed for rendering.

[0193] Among them, the entire special effects template is composed of multiple videos and multiple PAG special effects. The user material is divided to obtain the smallest rendering node, that is, the leaf node in the tree rendering structure, and finally a PAG rendering tree is formed. The video clips in the rendering leaf node are organized through the PAG video layer.

[0194] (3) Adding special effects

[0195] Among them, the specific rendering process of adding special effects is as follows:

[0196] 1) Create Root PAGComposition according to the final rendering size, using the interface: static std::shared_ptr <pagcomposition>Make(int width, int height).

[0197] 2) Add the divided rendering nodes in sequence. PAGComposition provides the addLayer interface to support adding rendering nodes and setting the rendering level of the rendering layer.

[0198] There are two situations for adding titles, endings, and special effects to the video screen, namely large stickers and small stickers. In the case of large stickers, the size information of the large stickers is not less than the video size, and the stickers contain placeholders. The video screen content needs to be filled into the placeholders of the stickers, such as Fig.12 As shown, a time segment is captured from the video to construct a PAGMovie, which is filled into the placeholder image of the sticker animation. Regarding the transition effect, the transition effect essentially uses a large sticker solution with two video filling layers. In the case of small stickers, the size of the small sticker is not larger than the video size, and the position information of the small sticker relative to the video screen needs to be provided, and then the sticker and video screen are merged, such as Fig.13 As shown, a time segment is extracted from the video to construct a PAGMovie, which is then filled into the constructed PAGImageLayer. The constructed PAGImageLayer is added to the Root PAGComposition, and then small stickers are added.

[0199] Regarding speed-changing effects, PAG supports setting speed-changing information. In actual use, the required filling time of the placeholder image is obtained, and the filling video time is captured based on the required filling time to build a PAGMovie (video layer) for filling.

[0200] (4) Add special effects with LUT filters

[0201] After adding special effects, you may encounter the need to add LUT filter special effects in the video editing scenario. In the specific implementation, considering the cross-platform, C++ is used to implement the LUT filter function. After the rendering of the special effect video is completed, the texture obtained by the rendering is used as the input of the LUT filter implementation module, and the texture output by the LUT filter implementation module is finally rendered on the screen.

[0202] (5) Audio Effect Synthesis

[0203] In terms of audio processing, the audio input source includes sound effects (existing in special effects), background music and video soundtrack. In terms of data organization, Fig.14 As shown, the audio rendering chain is constructed according to the track and segment structure in AVFoundation to ensure that the audio data on each track does not overlap.

[0204] After completing the division of audio tracks, the process of audio mixing is as follows Fig.15 As shown in the figure, since the formats of different audio files are different, a single audio file (Segment) is decoded by the container, resampled, converted into a unified format, and then the audio speed or volume is adjusted for the audio file. At the audio track level, when mixing audio, in addition to keeping the audio format consistent, the audio length of the sampling interval also needs to be consistent. Therefore, it is necessary to cache the audio data at the audio track level, and finally superimpose and mix the audio sampling data to obtain the audio file of the final video, so as to synthesize the audio file in the video with special effects.

[0205] In summary, the video processing method proposed in the embodiment of the present application has the following beneficial effects: the control of the entire rendering chain is through PAG instead of the video decoding module, and business logic such as adding video special effects is supported. The business logic such as adding video special effects can be used on each end after being implemented once, and desktop preview can be realized, which greatly reduces the workload of template development and improves the production efficiency of templates. For example, after the design is completed on the desktop, rendering on Android, iOS, and Linux can be achieved through SDK without additional development.

[0206] So far, the video processing method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the solution for implementing video processing by cooperating among the various modules in the video processing device 555 provided by the embodiment of the present application.

[0207] An acquisition module 5551 is used to acquire a video file to be edited and at least one special effects file for adding special effects to the video file to be edited; an editing module 5552 is used to perform tree-like editing processing based on the video file to be edited and the at least one special effects file to obtain a tree-like rendering structure including the video file and the special effects file; a rendering module 5553 is used to render the tree-like rendering structure based on the video file to be edited and the video layers included in the at least one special effects file to obtain a rendered video file; a synthesis module 5554 is used to perform audio synthesis processing based on the rendered video file to obtain a target video file with animation special effects added.

[0208] In some embodiments, the editing module 5552 is also used to divide the video file to be edited based on the at least one special effects file to obtain multiple video clips; and combine the multiple video clips and the at least one special effects file to obtain a tree-like rendering structure including the video file and the special effects file.

[0209] In some embodiments, the editing module 5552 is also used to obtain the time information of each of the special effects files, wherein the time information includes a starting time and a duration; based on the starting time and duration of each of the special effects files, determine the video segment corresponding to each of the special effects files and the video segment corresponding to the video fill in the video file to be edited, wherein the video fill indicates that the special effects are not added to the video file to be edited.

[0210] In some embodiments, the editing module 5552 is also used to perform the following processing on any of the at least one special effects file: determine the starting time of any of the special effects files as the starting time of the video segment corresponding to any of the special effects files in the video file to be edited; determine the duration of any of the special effects files as the duration of the video segment corresponding to any of the special effects files in the video file to be edited; based on the starting time and duration of any of the special effects files, determine the ending time of any of the special effects files, and use the ending time of any of the special effects files as the starting duration of the video segment filled with the corresponding video in the video file to be edited; use the time interval between the ending time of any of the special effects files and the starting time of the adjacent special effects file as the duration of the video segment filled with the corresponding video in the video file to be edited.

[0211] In some embodiments, the editing module 5552 is also used to determine the rendering size of the video file to be edited based on the header file and the tail file in the special effects file; determine the rendering root node based on the rendering size of the video file to be edited; use each of the video clips and each of the special effects file as a rendering leaf node; and determine a tree-like rendering structure including the video file and the special effects file based on the rendering root node and the rendering leaf nodes.

[0212] In some embodiments, the editing module 5552 is also used to create an add layer interface based on the rendering root node; through the add layer interface, the rendering leaf nodes are added sequentially on the rendering root node according to the time information of the rendering leaf nodes, so as to obtain a tree-like rendering structure including the video file and the special effects file.

[0213] In some embodiments, the rendering module 5553 is also used to traverse the tree-like rendering structure to obtain rendering leaf nodes of the tree-like rendering structure; when the rendering leaf node represents a video segment of the video file, the video segment of the video file is rendered to obtain a rendered video corresponding to the video file; when the rendering leaf node represents the special effects file, the video layer included in the special effects file is rendered to obtain a rendered video corresponding to the special effects file; the rendered video corresponding to the video file and the rendered video corresponding to the special effects file are combined to obtain a rendered video file.

[0214] In some embodiments, the rendering module 5553 is also used to perform format conversion processing on the video segment of the video file to obtain a video layer corresponding to the video segment; perform animation rendering processing based on the video layer corresponding to the video segment to obtain a rendered video corresponding to the video file.

[0215] In some embodiments, the rendering module 5553 is also used to obtain a target video segment in the video file to be edited that corresponds to the special effects file; fill the target video segment into the placeholder image of the video layer included in the special effects file to obtain a video layer filled with the target video segment; perform animation rendering processing on the video layer filled with the target video segment to obtain a rendered video corresponding to the special effects file.

[0216] In some embodiments, the rendering module 5553 is also used to obtain a target video segment in the video file to be edited that corresponds to the special effects file; fill the target video segment into the corresponding layer to obtain a layer filled with the target video segment; fill the video layer included in the special effects file into the layer filled with the target video segment to obtain a layer filled with the special effects file; perform animation rendering processing on the layer filled with the special effects file to obtain a rendered video corresponding to the special effects file.

[0217] In some embodiments, before rendering the tree-like rendering structure based on the video layer included in the at least one special effects file and the video file to be edited to obtain the rendered video file, the device also includes: a decoding module 5555, which is used to call a decoding interface to decode the at least one special effects file and the video file to be edited to obtain the special effects file and the video file to be edited used for the rendering process.

[0218] In some embodiments, the decoding module 5555 is also used to call the system decoding interface to decode the at least one special effects file and the video file to be edited when the system supports hardware decoding, so as to obtain the special effects file and the video file to be edited for the rendering processing; when the system does not support hardware decoding, call the decoding interface built into the software development kit to decode the at least one special effects file and the video file to be edited, so as to obtain the special effects file and the video file to be edited for the rendering processing.

[0219] In some embodiments, the synthesis module 5554 is also used to segment the audio file corresponding to the special effect file and the audio file corresponding to the video file to be edited to obtain multiple audio clips; divide the multiple audio clips into corresponding audio tracks to obtain multiple audio tracks with the audio clips added, wherein the audio clips between the audio tracks do not overlap; merge the multiple audio tracks with the audio clips added to obtain a target video file with special effects added.

[0220] In some embodiments, the synthesis module 5554 is also used to decode the multiple audio clips through the decontainer to obtain the decoded audio clips; resample the decoded audio clips to obtain the resampled audio clips; perform audio adjustment on the resampled audio clips to obtain the audio clips for division processing.

[0221] In some embodiments, the acquisition module 5551 is also used to display multiple candidate video files in the human-computer interaction interface; in response to a selection operation on the candidate video file, use the selected part of the candidate video file as the video file to be edited; and acquire at least one special effects file for adding special effects to the video file to be edited.

[0222] In some embodiments, the acquisition module 5551 is also used to display multiple candidate special effects files in the human-computer interaction interface; in response to a selection operation on the candidate special effects files, the selected part of the candidate special effects files is used as at least one special effects file for adding special effects to the video file to be edited.

[0223] In some embodiments, the acquisition module 5551 is further used to match the special effect files in the special effect file library based on the video content of the video file to be edited, and use the matched special effect files as the candidate special effect files.

[0224] In some embodiments, the acquisition module 5551 is further used to acquire the frequency of selection of the special effect file in the special effect file library; when the frequency of selection is greater than a selection threshold, the special effect file is used as the candidate special effect file.

[0225] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video processing method described in the embodiment of the present application.

[0226] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the video processing method provided by the present application embodiment, for example, Figure 5A-Figure 5C The video processing method is shown.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0228] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0229] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0230] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0231] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.< / pagcomposition>

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a video file to be edited and at least one special effect file for adding special effects to the video file to be edited; Performing tree-like editing processing based on the video file to be edited and the at least one special effect file to obtain a tree-like rendering structure including the video file and the special effect file; Traversing the tree-like rendering structure to obtain rendering leaf nodes of the tree-like rendering structure; When the rendering leaf node represents a video segment of the video file, rendering processing is performed on the video segment of the video file to obtain a rendered video corresponding to the video file; When the rendering leaf node represents the special effect file, rendering is performed on the video layer included in the special effect file to obtain a rendered video corresponding to the special effect file; Combining and processing the rendered video corresponding to the video file and the rendered video corresponding to the special effect file to obtain a rendered video file; Audio synthesis processing is performed based on the rendered video file to obtain a target video file with special effects added.

2. The method according to claim 1, characterized in that The performing tree-like editing processing based on the video file to be edited and the at least one special effect file to obtain a tree-like rendering structure including the video file and the special effect file includes: Dividing the video file to be edited based on the at least one special effect file to obtain a plurality of video clips; The multiple video clips and the at least one special effect file are combined to obtain a tree-like rendering structure including the video files and the special effect files.

3. The method according to claim 2, characterized in that The step of dividing the video file to be edited based on the at least one special effect file to obtain a plurality of video clips includes: Obtaining time information of each of the special effect files, wherein the time information includes a start time and a duration; Based on the starting time and duration of each special effect file, the video segment corresponding to each special effect file and the video segment corresponding to the video filling in the video file to be edited are determined, wherein the video filling indicates that the special effect is not added to the video file to be edited.

4. The method according to claim 3, characterized in that: The step of determining the video segment corresponding to each of the special effect files and the video segment filled with the corresponding video in the video file to be edited based on the starting time and duration of each of the special effect files comprises: The following processing is performed for any of the at least one special effect file: Determine the starting time of any of the special effect files as the starting time of the video segment corresponding to any of the special effect files in the video file to be edited; Determine the duration of any of the special effect files as the duration of a video segment in the video file to be edited that corresponds to any of the special effect files; Based on the starting time and duration of any of the special effects files, determine the ending time of any of the special effects files, and use the ending time of any of the special effects files as the starting duration of the video segment filled with the corresponding video in the video file to be edited; The time interval between the end time of any of the special effect files and the start time of an adjacent special effect file is used as the duration of the video segment filled with the corresponding video in the video file to be edited.

5. The method according to claim 2, characterized in that: The combining and processing the multiple video clips and the at least one special effect file to obtain a tree-like rendering structure including the video files and the special effect files includes: Determine the rendering size of the video file to be edited based on the header file and the end file in the special effects file; Determine a rendering root node based on the rendering size of the video file to be edited; Taking each of the video clips and each of the special effects files as a rendering leaf node; Based on the rendering root node and the rendering leaf nodes, a tree rendering structure including the video file and the special effect file is determined.

6. The method according to claim 5, characterized in that The determining, based on the rendering root node and the rendering leaf node, a tree rendering structure including the video file and the special effect file comprises: Create an add layer interface based on the rendering root node; Through the adding layer interface, the rendering leaf nodes are sequentially added to the rendering root node according to the time information of the rendering leaf nodes, so as to obtain a tree-like rendering structure including the video file and the special effect file.

7. The method according to claim 1, characterized in that The rendering process of the video segment of the video file to obtain a rendered video corresponding to the video file includes: Performing format conversion processing on the video clip of the video file to obtain a video layer corresponding to the video clip; Animation rendering is performed based on the video layer corresponding to the video clip to obtain a rendered video corresponding to the video file.

8. The method according to claim 1, characterized in that The rendering process is performed on the video layer included in the special effect file to obtain a rendered video corresponding to the special effect file, including: Obtain a target video segment corresponding to the special effects file in the video file to be edited; Filling the target video clip into the placeholder image of the video layer included in the special effect file to obtain a video layer filled with the target video clip; Animation rendering is performed on the video layer filled with the target video clip to obtain a rendered video corresponding to the special effects file.

9. The method according to claim 1, characterized in that: The rendering process is performed on the video layer included in the special effect file to obtain a rendered video corresponding to the special effect file, including: Obtain a target video segment corresponding to the special effects file in the video file to be edited; Filling the target video clip into the corresponding layer to obtain a layer filled with the target video clip; Filling the video layer included in the special effect file into the layer filled with the target video clip to obtain a layer filled with the special effect file; Animation rendering is performed on the layer filled with the special effect file to obtain a rendering video corresponding to the special effect file.

10. The method according to claim 1, characterized in that The method further comprises: A decoding interface is called to decode the at least one special effect file and the video file to be edited to obtain the special effect file and the video file for the rendering process.

11. The method according to claim 10, characterized in that The calling of the decoding interface to decode the at least one special effect file and the video file to be edited to obtain the special effect file and the video file for the rendering process includes: When the system supports hardware decoding, calling a system decoding interface to decode the at least one special effect file and the video file to be edited, to obtain the special effect file and the video file to be edited for the rendering process; When the system does not support hardware decoding, the decoding interface built into the software development kit is called to decode the at least one special effect file and the video file to be edited to obtain the special effect file and the video file for the rendering process.

12. A video processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a video file to be edited and at least one special effect file for adding special effects to the video file to be edited; An editing module, configured to perform tree-like editing processing based on the video file to be edited and the at least one special effect file, to obtain a tree-like rendering structure including the video file and the special effect file; A rendering module is used to traverse the tree-like rendering structure to obtain rendering leaf nodes of the tree-like rendering structure; when the rendering leaf node represents a video segment of the video file, the video segment of the video file is rendered to obtain a rendered video corresponding to the video file; when the rendering leaf node represents the special effect file, the video layer included in the special effect file is rendered to obtain a rendered video corresponding to the special effect file; the rendered video corresponding to the video file and the rendered video corresponding to the special effect file are combined to obtain a rendered video file; The synthesis module is used to perform audio synthesis processing based on the rendered video file to obtain a target video file with animation special effects added.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the video processing method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the video processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the video processing method according to any one of claims 1 to 11 is implemented.

Images