Video editing method and device
By using the point prediction model to identify the target drum beats and perform video editing, the existing card point editing scheme is solved, and the automatic editing and audio-visual appeal is improved.
Patent Information
- Application Number
- CN202411565514.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-11
AI Technical Summary
The existing card point editing scheme relies on the inherent beat or manual intervention of background music, resulting in inefficient editing, rigid and patterned films, lack of rhythmic changes, and affecting the viewing experience.
By obtaining the target audio and initial video, using the pre-trained point prediction model, the drum beat prediction value in the audio frame sequence is determined, the target drum beat is identified, and the initial video is edited based on this information to generate a clip with rhythm changes. .
It realizes automated editing, generates editing into films with rhythm changes, enhances audio-visual appeal, optimizes editing effects and viewing experience.
Smart Images
Figure CN119420992A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a video editing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] With the rapid development of digital multimedia technology, the content and quantity of videos are increasing, and it is more difficult for viewers to obtain useful information. Through video editing (such as card point editing), the content and structure of the video can be optimized to generate a more attractive video.
[0003] However, the current editing solutions rely on the inherent beat of background music or manual intervention, resulting in inefficient editing, stiff and stereotyped films, and a lack of rhythmic changes, which affects the viewing experience.
[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention
[0005] The embodiments of the present application provide a video editing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.
[0006] One aspect of an embodiment of the present application provides a video editing method, the method comprising: Acquire a target audio and an initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames; Inputting the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes a plurality of drum beat prediction values, wherein the plurality of drum beat prediction values correspond to the plurality of audio frames in a one-to-one manner, and the drum beat prediction values are used to indicate whether the corresponding audio frame includes a target drum beat, and the clipping fitness of the target drum beat is greater than a first preset threshold; Determining a target drum beat of the target audio according to the drum beat prediction sequence; Based on the target drum beats and the target audio, the initial video is edited to obtain a target video.
[0007] Optionally, the target drum beats include one or more; correspondingly, determining the target drum beats in the target audio according to the drum beat prediction sequence includes: Sequentially acquiring drum beat prediction values from the drum beat prediction sequence; Determine whether the corresponding audio frame includes the target drum beat according to the obtained drum beat prediction value; In a case where the corresponding audio frame includes a target drum beat, a timestamp of the corresponding audio frame is acquired, where the timestamp is used to represent the target drum beat.
[0008] Optionally, based on the target drum beats and the target audio, the initial video is edited to obtain a target video, including: Performing segment filtering on the initial video to obtain multiple video segments; Based on the target drum beats and the target audio, determine a plurality of audio segments and the duration of each audio segment, wherein the audio segment is determined according to two adjacent target drum beats in the target audio; Determine, according to the duration of each audio segment, a first video segment corresponding to each audio segment from the multiple video segments, wherein the duration of the first video segment is greater than or equal to the duration of the corresponding audio segment; Processing the corresponding first video segment according to the duration of each audio segment to obtain a second video segment corresponding to each audio segment, wherein the duration of the second video segment is equal to the duration of the corresponding audio segment; The second video segments corresponding to the multiple audio segments are spliced together to obtain the target video.
[0009] Optionally, the point prediction model is obtained by the following operations: Get multiple clips into a movie; Extract the audio of each edited video and determine the transition point of each edited video; Based on the multiple audio clips and transition points, multiple training samples are constructed, and model training is performed to obtain the point prediction model.
[0010] Optionally, the clip includes a plurality of video frames, each video frame having a corresponding time stamp; Correspondingly, determine the transition points of each clip into a film, including: Acquire the multiple video frames edited into a film; Sequentially calculating the similarity between two adjacent video frames in the plurality of video frames; Determine the latter of the two adjacent video frames whose similarity is less than a second preset threshold as a transition video frame; Determining the transition point according to the transition video frame; The timestamp corresponding to the transition video frame is used to represent the transition point.
[0011] Optionally, the training sample is obtained by the following operations: Get the audio frame interval; Converting the clipped audio into an audio frame sequence according to the audio frame interval; quantizing the transition points of the clip according to the audio frame interval to obtain a quantized label sequence; The training samples are constructed based on the clipped audio frame sequence and the quantized label sequence.
[0012] Optionally, constructing a plurality of training samples based on the plurality of audio clips and transition points, performing model training, and obtaining the point prediction model includes: Inputting the multiple audio frame sequences edited into a film into a model to obtain the drum beat prediction sequences edited into the multiple films; Calculating a loss function according to the plurality of quantized label sequences clipped into pieces and the drum beat prediction sequence; Based on the calculation result of the loss function, the model is updated through a back propagation algorithm to obtain the point prediction model.
[0013] Another aspect of an embodiment of the present application provides a video editing device, the device comprising: A first determination module is used to obtain a target audio and an initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames; An acquisition module, used for inputting the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes a plurality of drum beat prediction values, wherein the plurality of drum beat prediction values correspond to the plurality of audio frames in a one-to-one manner, and the drum beat prediction values are used to indicate whether the corresponding audio frame includes a target drum beat, and the editing fitness of the target drum beat is greater than a first preset threshold; A second determination module, configured to determine a target drum beat of the target audio according to the drum beat prediction sequence; The editing module is used to edit the initial video based on the target drum beats and the target audio to obtain a target video.
[0014] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.
[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.
[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.
[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: The initial video and the target audio are obtained, and the target audio is processed to obtain an audio frame sequence including multiple audio frames. The audio frame sequence is input into a pre-trained point prediction model to obtain a drum beat prediction sequence including multiple drum beat prediction values. Among them, multiple drum beat prediction values correspond one to one to multiple audio frames, and the drum beat prediction values are used to characterize whether the corresponding audio frames include target drum beats. The target drum beat is a time node in the target audio suitable for card point editing. According to the drum beat prediction sequence, the target drum beat of the target audio can be determined. The initial video is edited according to the target drum beat and the target audio to obtain the target video (i.e., edited into a film). It can be seen that the embodiment of the present application can quickly obtain the time node (target drum beat) suitable for card point editing in the target audio through the point prediction model, thereby realizing automated editing, obtaining an edited film with rhythm changes, enhancing the audiovisual appeal, and effectively optimizing the editing effect and viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0019] Figure 1 The flowchart of the video editing method according to the first embodiment of the present application is schematically shown; Figure 2 The flowchart of obtaining the point prediction model according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S202; Figure 4 The flowchart of constructing training samples according to the first embodiment of the present application is schematically shown; Figure 5 Schematically shows Figure 3 Flow chart of sub-steps of step S304; Figure 6 Schematically shows Figure 1 Flow chart of sub-steps of step S104; Figure 7 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 8 and Fig. 9 This is an example diagram of the application of the video editing method according to the first embodiment of the present application; Fig.10 A block diagram schematically shows a video editing device according to the second embodiment of the present application; and Fig.11 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.
[0023] First, the following terms are explained: Card-point editing: a type of editing technique that makes the transition time of the edited film coincide with the strong beat time of the background music.
[0024] Intelligent video editing: Use algorithms and artificial intelligence technology to automatically analyze video content and edit it according to preset rules.
[0025] Beat: The regular alternation of strong and weak sounds in music, which can correspond to the rhythm of the music.
[0026] Beat tracking: Also known as beat tracking, through beat tracking, computers can identify the regular rhythm points of music, namely beats or downbeats, and thus understand the temporal structure of music.
[0027] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: With the rapid development of digital multimedia technology, the content and quantity of videos are increasing, and it is more difficult for viewers to obtain useful information. Through video editing (such as card point editing), the content and structure of the video can be optimized to generate a more attractive video.
[0028] However, the applicant has learned that the related card point editing scheme still has the following defects: (1) The editor manually marks the transition time points of the selected background music, then selects the video clips and cuts them according to the marked transition time points, and finally concatenates the cut video clips and exports them together with the background music. This editing solution has high labor costs and is time-consuming, and it is difficult to meet the needs of large-scale video editing.
[0029] (2) The drum beats identified by the relevant drum beat tracking algorithm are all the beats of the music. Beats are an inherent attribute of music and are evenly distributed in the time dimension, that is, the time intervals between adjacent drum beats are the same. Performing card-point editing on this basis will result in uniform transition intervals in the edited film, which will be boring and lack rhythmic changes.
[0030] To this end, the embodiment of the present application provides a technical solution for video editing. In this technical solution: (1) A large number of videos with manual card-point editing are collected. The transition timestamps of these videos can be considered as the time points (drum beats) suitable for card-point editing understood by the editor, which include a higher level of understanding of the rhythm of music. The transition timestamps and audio of the videos with manual card-point editing are obtained to form a "background music-drum beats suitable for editing" data pair for training the model. After the training converges, the model can be used to identify drum beats suitable for editing in music, thereby making the output of the intelligent editing algorithm have rhythmic changes, more audiovisual appeal, and closer to the effect of manual editing. (2) By simulating the process of manual card-point editing through an algorithm, and then running the algorithm on a computer to output a large number of editing results, costs can be saved, editing automation can be achieved, and editing with a high audiovisual impact can be obtained. See below for details.
[0031] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.
[0032] Embodiment 1 Figure 1 The flowchart of the video editing method according to the first embodiment of the present application is schematically shown.
[0033] like Figure 1 As shown, the video editing method may include steps S100 to S106, wherein: Step S100: acquiring target audio and initial video, and determining an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames.
[0034] Step S102: input the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes multiple drum beat prediction values, and the multiple drum beat prediction values correspond one-to-one to the multiple audio frames. The drum beat prediction values are used to characterize whether the corresponding audio frames include target drum beats, and the editing adaptability of the target drum beats is greater than a first preset threshold.
[0035] Step S104: determining a target drum beat of the target audio according to the drum beat prediction sequence.
[0036] Step S106, based on the target drum beats and the target audio, the initial video is edited to obtain a target video.
[0037] The video editing method provided in this embodiment obtains the initial video and the target audio, and processes the target audio to obtain an audio frame sequence including multiple audio frames. The audio frame sequence is input into a pre-trained point prediction model to obtain a drum beat prediction sequence including multiple drum beat prediction values. Among them, multiple drum beat prediction values correspond to multiple audio frames one by one, and the drum beat prediction values are used to characterize whether the corresponding audio frame includes the target drum beat. The target drum beat is a time node in the target audio suitable for card point editing. According to the drum beat prediction sequence, the target drum beat of the target audio can be determined. The initial video is edited according to the target drum beat and the target audio to obtain the target video (i.e., edited into a film). It can be seen that the embodiment of the present application can quickly obtain the time node (target drum beat) suitable for card point editing in the target audio through the point prediction model, thereby realizing automated editing, obtaining an edited film with rhythm changes, enhancing the audiovisual appeal, and effectively optimizing the editing effect and viewing experience.
[0038] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.
[0039] Step S100 , obtain target audio and initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes multiple audio frames.
[0040] The target audio may be background music, narration or dialogue, etc., which can be determined according to the actual editing requirements. The initial video, i.e., the video to be edited, may include one or more. The target audio is converted according to a preset audio frame interval to obtain an audio frame sequence x. Exemplarily, the target audio may be divided into multiple smaller segments (i.e., audio frames) along the time dimension, and each segment (audio frame) includes audio data of the target audio within a specific time period. The audio frame sequence is a tensor in which all audio frames are arranged in chronological order.
[0041] Step S102 , input the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, the drum beat prediction sequence includes multiple drum beat prediction values, the multiple drum beat prediction values correspond to the multiple audio frames one by one, the drum beat prediction values are used to characterize whether the corresponding audio frame includes a target drum beat, and the editing adaptability of the target drum beat is greater than a first preset threshold.
[0042] The point prediction model can be pre-trained to identify the time points (target drum beats) in the target audio that are suitable for editing. For example, the audio frame sequence x can be input into the point prediction model to output the drum beat prediction sequence through the point prediction model. . Drum beat prediction sequence The length of the audio frame sequence x can be the same (both are N, where N is a positive integer greater than 0). A plurality of drum beat prediction values may be included, and the plurality of drum beat prediction values (N) correspond one to one with the plurality of audio frames (N). Each drum beat prediction value may be used to indicate whether the corresponding audio frame includes a target drum beat, and the clipping fitness of the target drum beat is greater than a first preset threshold, that is, the target drum beat is a drum beat suitable for clipping.
[0043] In this embodiment, the point prediction model is used to predict whether each audio frame includes the target drum beat, so that the time point in the target audio suitable for editing can be efficiently determined without manual intervention. Unlike related technologies, the point prediction model in the embodiment of the present application does not directly output all the drum beats of the target audio, but after deeply understanding the target audio, selects certain specific drum beats (such as drum beats where harmony, modal particles, etc. appear), and indicates these drum beats through a drum beat prediction sequence for subsequent intelligent editing to optimize the editing effect.
[0044] The point prediction model can be obtained in many ways, such as unsupervised learning, supervised learning, etc. An exemplary solution is provided below.
[0045] In an optional embodiment, if Figure 2 As shown, the point prediction model can be obtained through the following steps: Step S200, obtaining a plurality of clips.
[0046] Step S202, extracting the audio of each edited video and determining the transition point of each edited video.
[0047] Step S204, constructing a plurality of training samples based on the plurality of audio clips and transition points, performing model training, and obtaining the point prediction model.
[0048] The edited video can be a video that is manually edited or a video that is intelligently edited to meet the preset quality standards. The video quality can be evaluated through audience feedback (such as likes, favorites, reposts, and downloads). For any edited video, the time point of the transition (transition point) can be regarded as the drum beat that the editor understands is suitable for editing, or the editing drum beat recognized by the audience, which contains a higher-level semantic understanding of the musical rhythm. Determine the transition point of each edited video, build a data pair (training sample) based on the audio and transition point of each edited video, and perform model training to obtain a point prediction model.
[0049] In this embodiment, a large number of audio clips and transition points are used to construct data pairs for model training. After the training converges, the point prediction model can automatically predict the drum beats suitable for editing in any audio, thereby improving the editing effect and efficiency, so as to achieve automated card point editing that is highly consistent with the rhythm of the music.
[0050] There are many ways to determine the transition point, such as visual analysis, lens language analysis, plot or structure analysis, etc. An exemplary solution is provided below.
[0051] In an optional embodiment, if Figure 3 As shown, step S202 may include: Step S300, obtaining the plurality of video frames edited into a film.
[0052] Step S302 , sequentially calculating the similarity between two adjacent video frames in the plurality of video frames.
[0053] Step S304: Determine the latter of the two adjacent video frames whose similarity is less than a second preset threshold as a transition video frame.
[0054] Step S306, determining the transition point according to the transition video frame; wherein the timestamp corresponding to the transition video frame is used to represent the transition point.
[0055] Exemplarily, the clipped film may include multiple video frames, each of which has a corresponding timestamp. The multiple video frames of the clipped film are obtained, and the timestamps of two adjacent video frames of the clipped film are calculated in sequence ( and ) (Such as cosine similarity ). Determine a general threshold based on experience or actual editing needs (Second preset threshold). The first The flag of whether a video frame is transitioned can be expressed by the following formula: ; 1 indicates transition, and 0 indicates no transition. After all video frames in the edited video are calculated, the timestamp of the transition video frame can be obtained. =frame number / frame rate, the timestamp of the transition video frame is the transition point.
[0056] In this embodiment, the transition video frame is identified by calculating the similarity between frames. The timestamp of the transition video frame is the drum beat suitable for editing, which reduces the time of manually searching for the transition video frame and greatly improves the editing efficiency and editing accuracy.
[0057] The above embodiment exemplarily introduces how to obtain data for constructing training samples. The following will further exemplarily introduce the training sample construction method and the model training process.
[0058] In an optional embodiment, if Figure 4 As shown, the training samples can be constructed by the following steps: Step S400, obtaining the audio frame interval.
[0059] Step S402: convert the clipped audio into an audio frame sequence according to the audio frame interval.
[0060] Step S404: quantize the transition points of the clip according to the audio frame interval to obtain a quantized label sequence.
[0061] Step S406: construct the training sample based on the edited audio frame sequence and the quantized label sequence.
[0062] Exemplarily, the audio frame interval can be set based on experience, for example, the audio frame interval is set to a. According to the audio frame interval, the audio clipped into a film can be converted into multiple audio frames along the time dimension to obtain an audio frame sequence x. The audio frame sequence x can include multiple audio frames (a total of N). According to the audio frame interval, the transition points of the clipped film can be quantized to obtain a quantized label sequence. For example, the transition points (timestamps) of the clipped film can be expressed as [1.5, 3.5]. For any timestamp , which can be quantified as , get the quantitative label. Note that the calculation needs to be rounded. =1, and the others are 0. Assuming that the audio frame interval a=0.5*audio sampling rate, the quantization step of the timestamp is a / audio sampling rate=0.5*audio sampling rate / audio sampling rate=0.5, which can be quantized to [3, 7]. Because the 3rd and 7th audio frames include transition points (timestamps), the quantized label sequence y can be expressed as [0, 0, 1, 0, 0, 0, 1]. It can be seen that the value of the i-th bit of the quantized label sequence y can indicate whether the i-th audio frame includes a drum beat suitable for editing. Based on the edited audio frame sequence and the quantized label sequence, a training sample can be constructed.
[0063] In this embodiment, by processing the audio of the edited film and quantizing the time of the transition points, training samples can be efficiently constructed so that the point prediction model can accurately identify drum beats suitable for editing.
[0064] In an optional embodiment, if Figure 5 As shown, step S306 may include: Step S500: input the multiple audio frame sequences edited into a piece into a model to obtain the drum beat prediction sequences edited into a piece.
[0065] Step S502: Calculate a loss function according to the quantized label sequences of the plurality of clips and the drum beat prediction sequence.
[0066] Step S504: Based on the calculation result of the loss function, the model is updated by a back propagation algorithm to obtain the point prediction model.
[0067] After constructing multiple training samples, the point prediction model can be trained in a supervised manner. For example, assuming that the neural network is F, and the input of the neural network is the audio frame sequence x, then the output of the neural network is the drum beat prediction sequence . Based on the quantized label sequence y and the drum beat prediction sequence The binary cross entropy loss function can be calculated The weights of the neural network F are iteratively updated through the back propagation algorithm according to the calculation results of the loss function. After the training converges, a point prediction model can be obtained to predict whether each audio frame in the target audio includes a drum beat suitable for editing, and then obtain the timestamp of the drum beat suitable for editing (target drum beat).
[0068] In this embodiment, the point prediction model is trained in a supervised manner, and a loss function (such as binary cross entropy) is used to evaluate the gap between the drum beat prediction sequence and the true quantized label sequence. Then, the neural network weights can be efficiently updated through the back propagation algorithm to obtain a point prediction model with higher performance level, which helps to improve editing accuracy.
[0069] The above embodiments exemplarily introduce the training method and application of the point prediction model. The following will further exemplarily introduce how to implement intelligent card point editing based on the output of the point prediction model.
[0070] Step S104 , determining the target drum beats of the target audio according to the drum beat prediction sequence.
[0071] Predicting Sequences Based on Drum Beats Each drum beat prediction value in can determine whether each audio frame includes a target drum beat, and further determine the target drum beat included in the entire target audio. An exemplary solution is provided below.
[0072] In an optional embodiment, if Figure 6 As shown, step S104 may include: Step S600, obtaining drum beat prediction values from the drum beat prediction sequence in sequence.
[0073] Step S602: Determine whether the corresponding audio frame includes a target drum beat according to the acquired drum beat prediction value.
[0074] Step S604: when the corresponding audio frame includes the target drum beat, obtain a timestamp of the corresponding audio frame, where the timestamp is used to represent the target drum beat.
[0075] For example, the drum prediction sequence It can be [0, 0, 1, 0, 0, 0, 1], where the value of the i-th bit can indicate whether the i-th audio frame includes the target drum beat. If the value is 0, it means that the corresponding audio frame does not include the target drum beat. If the value is 1, the corresponding audio frame includes the target drum beat. The drum beat prediction value is obtained from the drum beat prediction sequence in sequence, and according to the drum beat prediction value, it can be determined whether each audio frame of the audio frame sequence includes the target drum beat. When an audio frame includes the target drum beat, the timestamp of the audio frame is obtained, and the timestamp can be used to characterize the target drum beat.
[0076] In this embodiment, drum beat prediction values are obtained in sequence to determine whether each audio frame includes a target drum beat, and the timestamp of the audio frame is obtained to characterize the target drum beat, so that drum beats suitable for editing can be extracted efficiently and accurately, thereby improving the efficiency of video editing.
[0077] Step S106 , based on the target drum beats and the target audio, the initial video is edited to obtain a target video.
[0078] Exemplarily, for the target audio, the initial video can be edited into multiple video clips according to the interval between two adjacent target drum beats and preset rules (such as content matching, music matching, length limit, etc.), and finally the multiple video clips and the target audio are combined to obtain the target video.
[0079] An exemplary scheme is provided below.
[0080] In an optional embodiment, if Figure 7 As shown, step S106 may include: Step S700: filtering the initial video to obtain multiple video segments.
[0081] Step S702: Determine a plurality of audio segments and a duration of each audio segment based on the target drum beats and the target audio, wherein the audio segment is determined based on two adjacent target drum beats in the target audio.
[0082] Step S704: determining a first video segment corresponding to each audio segment from the multiple video segments according to the duration of each audio segment, wherein the duration of the first video segment is greater than or equal to the duration of the corresponding audio segment.
[0083] Step S706: Process the corresponding first video segment according to the duration of each audio segment to obtain a second video segment corresponding to each audio segment, wherein the duration of the second video segment is equal to the duration of the corresponding audio segment.
[0084] Step S708: splice the second video segments corresponding to the multiple audio segments to obtain the target video.
[0085] For example, the initial video may be roughly screened by a segment filtering algorithm to obtain multiple video segments. The segment filtering algorithm can be any algorithm that can play a filtering role. For example, a person recognition algorithm (such as RetinaFace+ArcFace supported by the open source face algorithm library InsightFace) can be used to filter out video segments containing specific people. Another example is to use an action recognition algorithm (such as a dual-stream network, I3D, CLIP zero-sample classification algorithm, etc.) to filter out video segments containing specific actions. Based on the target drum beats and target audio, multiple audio segments and the duration of each audio segment can be determined. The audio segment can be determined by two adjacent target drum beats in the target audio. The duration of the audio segment It can be expressed as follows: .
[0086] According to each audio clip duration, can be in multiple video clips In the example, a corresponding video segment (the first video segment) is selected for each audio segment. and The correspondence between can be arbitrary, but it is necessary to ensure , that is, the duration of the first video segment must be greater than or equal to the duration of the corresponding audio segment, so that and Alignment.
[0087] The second video segment can be obtained by cutting the corresponding first video segment according to the duration of each audio segment. , , so that , that is, the duration of the second video segment is equal to the duration of the corresponding audio segment.
[0088] It should be noted that the starting point of cropping can be arbitrary or carefully set according to actual editing needs, which will not affect the editing effect of the card point. After concatenation and adding the target audio, the target video can be output (i.e. edited into a film). The duration and corresponding Consistent, when playing the target video, the segments are replaced The timing of a target drum beat Completely consistent, thus achieving the effect of card point.
[0089] In this embodiment, the process of manual editing is simulated by an algorithm, and a large number of editing results can be output by running the algorithm on a computer, thereby saving costs, realizing the automation of editing, and obtaining an edited film with a high audiovisual impact.
[0090] In order to make this application easier to understand, the following Figure 8 and Fig. 9 An exemplary application is provided.
[0091] (1) Model training: S11: Collect a large number of manually edited videos. For any manually edited video, determine the transition video frame by calculating the inter-frame similarity, obtain the timestamp of the transition video frame, and the timestamp can be used to characterize the transition point. The transition point is the drum beat suitable for editing.
[0092] S12: extract the background music of the manually clipped video, and obtain the audio frame sequence after processing. Perform event quantization on the transition points to obtain the quantized label sequence y.
[0093] S13: Use the audio frame sequence as the neural network input and output the drum beat prediction sequence through the neural network .
[0094] S14: Predict sequence based on quantized label sequence y and drum beats Calculate the loss function BCE.
[0095] S15: The weights of the neural network are iteratively updated through the back propagation algorithm, and the point prediction model is obtained after the training converges.
[0096] (2) Model application to video editing: S21: Select background music and initial video.
[0097] S22: Process the background music to obtain an audio frame sequence, where the audio frame sequence includes multiple audio frames.
[0098] S23: Input the audio frame sequence into the pre-trained point prediction model to obtain the drum beat prediction sequence of the target audio .
[0099] The drum beat prediction sequence includes multiple drum beat prediction values. The drum beat prediction sequence and the audio frame sequence have the same length. The multiple drum beat prediction values correspond to the multiple audio frames one by one. The drum beat prediction values are used to indicate whether the corresponding audio frames include a target drum beat, and the editing fitness of the target drum beat is greater than a first preset threshold. It can be seen that the target drum beat is a drum beat suitable for editing.
[0100] S24: Determine the target drum beat of the background music according to the drum beat prediction sequence.
[0101] S25: Determine multiple audio clips and durations of the multiple audio clips based on the target drum beats and the target audio.
[0102] S26: Filter the initial video segments to obtain multiple video segments.
[0103] S27: Based on the duration of each audio segment, a corresponding video segment is selected for each audio segment (the duration of the video segment is required to be greater than or equal to the duration of the corresponding audio segment).
[0104] S28: based on the duration of each audio segment, crop the corresponding video segment, and the duration of the cropped video segment is required to be equal to the duration of the corresponding audio segment.
[0105] S29: Connect multiple cropped video clips in series, add background music, and output a target video.
[0106] In this exemplary application: (1) A large number of manually edited videos are collected. The transition timestamps of these videos can be considered as the time points (target drum beats) that are suitable for editing according to the editors, which include a higher level of understanding of the rhythm of the music. The transition timestamps and audio of the manually edited videos can be obtained to form the "background music-drum beats suitable for editing" data pairs used to train the model. After the training converges, the model can be used to identify drum beats suitable for editing in the music, thereby making the output of the intelligent editing algorithm have rhythmic changes, more audiovisual appeal, and closer to the effect of manual editing. (2) By simulating the process of manual editing through algorithms, and then running the algorithm on a computer to output a large number of editing results, costs can be saved, editing automation can be achieved, and editing with a high audiovisual impact can be obtained.
[0107] Embodiment 2 Fig.10 The block diagram of the video editing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.10 As shown, the device 1000 may include: a first determination module 1100, an acquisition module 1200, a second determination module 1300 and a clipping module 1400, wherein: A first determination module 1100 is configured to obtain a target audio and an initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames; An acquisition module 1200 is used to input the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes a plurality of drum beat prediction values, wherein the plurality of drum beat prediction values correspond to the plurality of audio frames in a one-to-one manner, and the drum beat prediction values are used to indicate whether the corresponding audio frame includes a target drum beat, and the editing fitness of the target drum beat is greater than a first preset threshold; A second determination module 1300 is used to determine a target drum beat of the target audio according to the drum beat prediction sequence; The editing module 1400 is used to edit the initial video based on the target drum beats and the target audio to obtain a target video.
[0108] As an optional embodiment, the target drum beats include one or more; correspondingly, determining the target drum beats in the target audio according to the drum beat prediction sequence includes: Sequentially acquiring drum beat prediction values from the drum beat prediction sequence; Determine whether the corresponding audio frame includes the target drum beat according to the obtained drum beat prediction value; In a case where the corresponding audio frame includes a target drum beat, a timestamp of the corresponding audio frame is acquired, where the timestamp is used to represent the target drum beat.
[0109] As an optional embodiment, based on the target drum beats and the target audio, the initial video is edited to obtain a target video, including: Performing segment filtering on the initial video to obtain multiple video segments; Based on the target drum beats and the target audio, determine a plurality of audio segments and the duration of each audio segment, wherein the audio segment is determined according to two adjacent target drum beats in the target audio; Determine, according to the duration of each audio segment, a first video segment corresponding to each audio segment from the multiple video segments, wherein the duration of the first video segment is greater than or equal to the duration of the corresponding audio segment; Processing the corresponding first video segment according to the duration of each audio segment to obtain a second video segment corresponding to each audio segment, wherein the duration of the second video segment is equal to the duration of the corresponding audio segment; The second video segments corresponding to the multiple audio segments are spliced together to obtain the target video.
[0110] As an optional embodiment, the point prediction model is obtained by the following operations: Get multiple clips into a movie; Extract the audio of each edited video and determine the transition point of each edited video; Based on the multiple audio clips and transition points, multiple training samples are constructed, and model training is performed to obtain the point prediction model.
[0111] As an optional embodiment, the clip includes a plurality of video frames, each video frame having a corresponding time stamp; Correspondingly, determine the transition points of each clip into a film, including: Acquire the multiple video frames edited into a film; Sequentially calculating the similarity between two adjacent video frames in the plurality of video frames; Determine the latter of the two adjacent video frames whose similarity is less than a second preset threshold as a transition video frame; Determining the transition point according to the transition video frame; The timestamp corresponding to the transition video frame is used to represent the transition point.
[0112] As an optional embodiment, the training sample is obtained by the following operations: Get the audio frame interval; Converting the clipped audio into an audio frame sequence according to the audio frame interval; quantizing the transition points of the clip according to the audio frame interval to obtain a quantized label sequence; The training samples are constructed based on the clipped audio frame sequence and the quantized label sequence.
[0113] As an optional embodiment, multiple training samples are constructed based on the multiple audio clips and transition points, and model training is performed to obtain the point prediction model, including: Inputting the multiple audio frame sequences edited into a film into a model to obtain the drum beat prediction sequences edited into the multiple films; Calculating a loss function according to the plurality of quantized label sequences clipped into pieces and the drum beat prediction sequence; Based on the calculation result of the loss function, the model is updated through a back propagation algorithm to obtain the point prediction model.
[0114] Embodiment 3 Fig.11 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing a video editing method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.11 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of a video editing method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.
[0115] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.
[0116] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.
[0117] It should be pointed out that Fig.11 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.
[0118] In this embodiment, the video editing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.
[0119] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video editing method in the embodiment are implemented.
[0120] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video editing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.
[0121] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.
[0122] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0123] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A video editing method, characterized in that: The method comprises: Acquire a target audio and an initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames; Inputting the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes a plurality of drum beat prediction values, wherein the plurality of drum beat prediction values correspond to the plurality of audio frames in a one-to-one manner, and the drum beat prediction values are used to indicate whether the corresponding audio frame includes a target drum beat, and the clipping fitness of the target drum beat is greater than a first preset threshold; Determining a target drum beat of the target audio according to the drum beat prediction sequence; Based on the target drum beats and the target audio, the initial video is edited to obtain a target video.
2. The method according to claim 1, characterized in that: The target drum beats include one or more; correspondingly, determining the target drum beats in the target audio according to the drum beat prediction sequence includes: Sequentially acquiring drum beat prediction values from the drum beat prediction sequence; Determine whether the corresponding audio frame includes the target drum beat according to the obtained drum beat prediction value; In a case where the corresponding audio frame includes a target drum beat, a timestamp of the corresponding audio frame is acquired, where the timestamp is used to represent the target drum beat.
3. The method according to claim 2, characterized in that Based on the target drum beat and the target audio, the initial video is edited to obtain a target video, including: Performing segment filtering on the initial video to obtain multiple video segments; Based on the target drum beats and the target audio, determine a plurality of audio segments and the duration of each audio segment, wherein the audio segment is determined according to two adjacent target drum beats in the target audio; Determine, according to the duration of each audio segment, a first video segment corresponding to each audio segment from the multiple video segments, wherein the duration of the first video segment is greater than or equal to the duration of the corresponding audio segment; Processing the corresponding first video segment according to the duration of each audio segment to obtain a second video segment corresponding to each audio segment, wherein the duration of the second video segment is equal to the duration of the corresponding audio segment; The second video segments corresponding to the multiple audio segments are spliced together to obtain the target video.
4. The method according to claim 1, characterized in that: The point prediction model is obtained by the following operations: Get multiple clips into a movie; Extract the audio of each edited video and determine the transition point of each edited video; Based on the multiple audio clips and transition points, multiple training samples are constructed, and model training is performed to obtain the point prediction model.
5. The method according to claim 4, characterized in that The clip includes a plurality of video frames, each of which has a corresponding time stamp; Correspondingly, determine the transition points of each clip into a film, including: Acquire the multiple video frames edited into a film; Sequentially calculating the similarity between two adjacent video frames in the plurality of video frames; Determine the latter of the two adjacent video frames whose similarity is less than a second preset threshold as a transition video frame; Determining the transition point according to the transition video frame; The timestamp corresponding to the transition video frame is used to represent the transition point.
6. The method according to claim 4, characterized in that The training samples are obtained by the following operations: Get the audio frame interval; Converting the clipped audio into an audio frame sequence according to the audio frame interval; quantizing the transition points of the clip according to the audio frame interval to obtain a quantized label sequence; The training samples are constructed based on the clipped audio frame sequence and the quantized label sequence.
7. The method according to claim 6, characterized in that Constructing a plurality of training samples based on the plurality of audio clips and transition points, performing model training, and obtaining the point prediction model, including: Inputting the multiple audio frame sequences edited into a film into a model to obtain the drum beat prediction sequences edited into the multiple films; Calculating a loss function according to the plurality of quantized label sequences clipped into pieces and the drum beat prediction sequence; Based on the calculation result of the loss function, the model is updated through a back propagation algorithm to obtain the point prediction model.
8. A video editing device, characterized in that: The device comprises: A first determination module is used to obtain a target audio and an initial video, and determine an audio frame sequence according to the target audio, wherein the audio frame sequence includes a plurality of audio frames; An acquisition module, used for inputting the audio frame sequence into a pre-trained point prediction model to obtain a drum beat prediction sequence, wherein the drum beat prediction sequence includes a plurality of drum beat prediction values, wherein the plurality of drum beat prediction values correspond to the plurality of audio frames in a one-to-one manner, and the drum beat prediction values are used to indicate whether the corresponding audio frame includes a target drum beat, and the editing fitness of the target drum beat is greater than a first preset threshold; A second determination module, configured to determine a target drum beat of the target audio according to the drum beat prediction sequence; The editing module is used to edit the initial video based on the target drum beats and the target audio to obtain a target video.
9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.