Audio-guided video generation method and device, and medium

Automatically obtaining audio and video semantic alignment information through neural networks, solving the problem of ignoring audio and video semantic alignment and relying on detailed annotations in the existing technology, realizing automatic video editing and trailer generation without manual annotation, improving the scale and editing of data sets efficiency.

CN119071568AActive Publication Date: 2024-12-03BEIJING INST OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411174397.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-12-03
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Existing music-based movie trailer generation methods ignore semantic alignment between video and audio information and rely on detailed annotations or additional information, resulting in limited data set size and model overfitting.

Method used

The audio-guided video generation method based on neural network is adopted, and the semantic alignment information between audio and video is automatically obtained through audio and video feature encoding, cross attention module and Sinkhorn algorithm to generate high-quality movie trailers.

Benefits of technology

Automatic video editing and trailer generation without manual annotation is realized, the data set size is expanded, the risk of model overfitting is reduced, and the editing work efficiency and creator inspiration is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071568A_ABST
    Figure CN119071568A_ABST
Patent Text Reader

Abstract

The invention provides an audio-guided video generation method, audio-guided video generation equipment and a medium. The audio-guided video generation method comprises the following steps: segmenting an audio and a video into a sequence audio clip and a sequence video clip; inputting the audio clip and the video clip into a neural network to obtain a corresponding relationship between the audio clip and the video clip; according to the time sequence and the duration of the audio clip and the corresponding relationship between the audio clip and the video clip, adjusting the duration of the corresponding video clip, and splicing in sequence to generate a target video; the audio is used as background music of the target video. By adopting the technical scheme, the target video and the training data set can be generated by an automatic technology, challenges caused by manual annotation are avoided, the scale of the training data set is expanded, the working efficiency of an editor is improved, and meanwhile, inspiration of a creator is stimulated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of video generation technology, and in particular to a method, apparatus and medium for generating audio-guided videos based on neural networks. Background Technology

[0002] Movie trailers, as a collection of captivating clips, play a crucial role in film promotion. Trailers need to select engaging scenes while simultaneously rearranging them to subtly conceal the film's plot. The selection and order of these clips are determined by numerous factors, such as the semantics of the background music, the synchronization of video and audio content, and the logical flow of dialogue, reflecting a deep understanding of the film. Therefore, generating high-quality movie trailers involves complex editing work, making it a highly challenging task.

[0003] Early methods primarily relied on supplementary information to select and prioritize video footage, such as attempting to identify compelling audiovisual elements through sentiment analysis to aid trailer generation. Later, with the development of machine learning, several learning-based methods were proposed, gradually automating the trailer generation process. Among these, point-process-based methods learn visual appeal models using viewer visual gaze information collected by specialized eye trackers. Sentiment-relevance-based methods manually label video and audio clips with corresponding sentiment categories, aligning images, text, and audio in a latent space based on these sentiment categories, and selecting and reorganizing video clips by maximizing sentiment scores. Commercial software like Muvee enables music-based video editing and trailer synthesis.

[0004] Existing music-based movie trailer generation methods primarily focus on synchronizing movie clip transitions with the rhythm of the music, such as using accented notes to correspond to scene changes. However, they neglect the semantic alignment between video and audio information. For instance, low-volume, slow-paced music clips often appear alongside video clips depicting natural environments. Furthermore, existing learning-based methods heavily rely on videos with detailed annotations or additional information, such as frame-level gaze scores and manually defined sentiment labels. This data annotation is difficult and time-consuming, limiting dataset size and leading to a high risk of overfitting in the trained models. Summary of the Invention

[0005] To partially or completely solve the above-mentioned technical problems, this application proposes an audio-guided video generation method, device, and medium based on neural networks to automatically obtain semantic alignment information between audio and video, which can automatically edit video to generate trailers for given audio and movies.

[0006] The above-mentioned technical objective is achieved through the following technical solution.

[0007] In a first aspect, this application provides an audio-guided video generation method, comprising: segmenting audio and video into sequential audio segments and video segments; and inputting the audio segments and video segments into a neural network to obtain the correspondence between the audio segments and the video segments; Based on the timing and duration of the audio segments, as well as the correspondence between the audio and video segments, the duration of the corresponding video segments is adjusted and then spliced ​​together in sequence to generate the target video; the audio is used as the background music for the target video.

[0008] According to some embodiments, the neural network includes a video encoder, an audio encoder, a video selector, and an audio-video matcher; the video encoder is used to encode video segments to obtain video segment features; the audio encoder is used to encode audio segments to obtain audio segment features; the video selector is used to obtain video segment fusion features and audio segment fusion features based on the video segment features and audio segment features, and to obtain the selection probability of each video segment based on the video segment fusion features; the audio-video matcher is used to obtain the correspondence between audio segments and video segments by aligning audio and video segments based on the audio segment fusion features, video segment fusion features, and the selection probability of video segments.

[0009] According to some embodiments, the video encoder includes a cross-modal feature extraction model, a multilayer perceptron (MLP), and a self-attention module (SA) connected in sequence; the audio encoder includes a cross-modal feature extraction model, a multilayer perceptron (MLP), and a self-attention module (SA) connected in sequence.

[0010] According to some embodiments, the cross-modal feature extraction model is a pre-trained ImageBind model.

[0011] According to some embodiments, the video selector includes a cross-attention module, a second multilayer perceptron, and an activation function connected in sequence; the cross-attention module is used to capture the interaction between the input video segment features and audio segment features, and output video segment fusion features and audio segment fusion features; the second multilayer perceptron is used to reduce the dimensionality of the video segment fusion features to a vector, and the vector is passed through an activation function to obtain the selection probability of each video segment; the activation function is the Sigmoid function.

[0012] According to some embodiments, the audio / video matcher obtains the optimal transmission matrix by solving the entropy-optimal transmission (EOT) problem. To obtain the correspondence between audio segments and video segments:

[0013] in, The distance matrix representing the fused features of video segments and the fused features of audio segments;T For the transmission scheme, a matrix representing the correspondence between video and audio segments is provided. Representation matrix sum matrix Calculation of the inner product; , indicating all transmission schemes The set of, where Requirements: (The last part is incomplete and likely refers to a specific condition or requirement.) The elements of the column vector obtained by summing each row are equal to The element at the corresponding position in the vector. yes The sum of the elements in the column vector. It is the selection probability vector of video segments; for , Indicates length is The values ​​are all The row vector, The number of audio segments; hyperparameters It is an entropy regularization term. The weight, Representation matrix sum matrix The inner product calculation.

[0014] According to some embodiments, the EOT optimization problem is solved using the Sinkhorn algorithm; the Sinkhorn algorithm process is as follows: Step 1: Set up the matrix Initialize a length and A column vector with the same dimensions and all elements equal to 1. ; Step 2: According to and renew and ; Step 3: Repeat step 2 until convergence. The convergence criterion is... If the sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than a preset threshold, then diag This indicates the generation of a vector. A square matrix whose elements are diagonal elements. This indicates the generation of a vector. A square matrix whose elements are diagonal elements; according to Calculate the optimal transmission matrix .

[0015] According to some embodiments, the loss function during neural network training is:

[0016] in, This indicates the number of data points in the training dataset. This represents the set of parameters of the neural network. It is the optimal transfer matrix predicted by the EOT problem. Alignment matrix with the real Between divergence, Indicates the first in the training dataset The number of video clips in the target video. Indicates that the training dataset is based on the first Alignment matrix constructed by the part video segment and the target video segment; It is the first one predicted by the neural network. Video segment selection probability of the video A column vector consisting of the probability of selecting real video clips. Binary cross-entropy between loss, , Represents a length of A column vector consisting entirely of 1s, hyperparameters yes The weight of the loss.

[0017] According to some embodiments, before obtaining the correspondence between audio segments and video segments using the audio-video matcher, according to Choose the one with the highest probability value. Calculate the video clip corresponding to each element. Video segment fusion features and Distance matrix between fused features of audio segments , The element in the i-th row and j-th column The calculation method is as follows hyperparameters Indicates weight, This represents the fusion feature of the selected i-th video segment. This represents the fusion feature of the j-th audio segment. Let represent the Euclidean distance between the fused features of the i-th video segment and the fused features of the j-th audio segment. This represents the duration of the i-th video segment. This represents the absolute value of the duration difference between the i-th video segment and the j-th audio segment; [Settings are missing from the original text]. For length is All values ​​are Column vectors.

[0018] In a second aspect, this application also provides an electronic device, characterized in that it includes: a processor; and a memory storing a program executable by the processor, wherein when the program is executed, the processor performs the method described in any one of the preceding first aspects.

[0019] Thirdly, this application also provides a storage medium that stores a program executable by a processor, wherein when the program is executed, the processor performs the method described in any of the preceding first aspects.

[0020] As will be described in detail below, the neural network-based audio-guided video generation method, device, and medium according to embodiments of this application automatically select and sort exciting segments from long videos such as movies and TV series for a given audio clip by constructing a neural network model based on partial back-optimal transport, and then organizing them into a promotional video for that long video. Using the above technical solution, not only can trailers be generated based on the semantic information between audio and video, but the target video and training dataset can also be generated entirely by automated technology, eliminating the challenges of manual annotation, expanding the scale of the training dataset, improving the efficiency of editors, and also helping to inspire creators. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 A schematic diagram of the IPOT neural network model architecture according to an example embodiment is shown; Figure 2 This diagram illustrates the computation flow of the Sinkhorn algorithm according to an example embodiment. Figure 3 This illustrates a flowchart of an audio-guided video generation method according to an example embodiment. Detailed Implementation

[0023] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The audio-guided video generation method proposed in this application specifically defines the task of generating the target video as the selection and sorting of video segments based on given audio segments, and uses a learning model with an Inverse Partial Optimal Transport (IPOT) architecture to achieve this task. First, after extracting features from video and audio segments using an encoder, the IPOT framework applies a selection network with an attention mechanism to parameterize the distribution of video segments and selects some video segments for building trailers based on this distribution. Then, it calculates the partial optimal transport distance between the latent features of these video and audio segments to achieve alignment between them. Finally, it arranges the movie segments based on semantic alignment information.

[0025] This application applies to scenarios where video segments are selected from a given audio source and the given audio and selected video segments are combined to create a target video. The most typical application is generating movie trailers from music and original film clips. The technical solution proposed in this application will be explained in detail below using the generation of movie trailers as an example. The generation of short clips from other long videos such as TV series based on given audio is handled similarly and will not be elaborated further.

[0026] The generation of movie trailers Given a movie and a piece of music, the goal of this application is to generate a trailer for the movie based on the music. Furthermore, it transforms the music-guided movie trailer generation task into the selection and ranking of movie segments based on audio clips. To achieve this, this application proposes an IPOT neural network model that learns a multimodal representation, selection, and matching process in a supervised manner. Of course, it is not limited to the IPOT neural network model; any neural network can be used, as long as it can select video segments from the video that match the audio content based on semantic information.

[0027] IPOT Neural Network Model Figure 1 and Figure 3 The network architecture of the IPOT model proposed in this application is shown, which consists of three parts: two encoders for encoding video and audio features, a cross-attention module as a video segment selector, and a Sinkhorn matching network as a matcher between video and audio segments. Two of these parts involve neural network structure: the audio / video feature encoder and the cross-attention module.

[0028] For a given movie, use a video segmentation tool such as BaSSL to divide the movie into movie segments. ,in Include The trailer audio was divided into audio segments using the audio segmentation tool Ruptures. ,in Include The film clips and audio clips are processed together, and then the pre-trained ImageBind model is used to extract the features of the film clips and audio clips. and ,in and These represent the ImageBind models used to extract features from movie clips and audio clips, respectively. The dimension of the extracted features is defined as follows. Considering the temporal correlation between film and audio features—that is, different film clips have a chronological order in the original film—we process the extracted features through two multi-layer perceptrons (MLPs), and then further encode them through two self-attention (SA) modules (extracting temporal features). A pre-trained ImageBind model (for multi-modal feature extraction), a multi-layer perceptron (for adjusting feature dimensions), and a self-attention module (for extracting interactions between different clips) constitute the video encoder and audio encoder. Based on the film clip features and audio clip features, the purpose of this application is to select key film clips and align them with the audio clips of the trailer, thereby generating a trailer composed of key film clips. This part will be implemented through the following two components.

[0029] The second part of the neural network is a movie clip selector based on a cross-attention mechanism, used to predict which movie clips will appear in the movie trailer. The cross-attention (CA) module in IPOT captures the interaction between movie and audio features and generates fused movie clip features. and audio segment features The newly fused movie clip features will be processed through a multilayer perceptron (reduced to a vector) and a sigmoid function (the probability of each movie clip being selected), resulting in a vector of length [length missing]. column vectors Each element in the vector is a decimal between [0,1] because the projection of the Sigmoid function represents the probability that the corresponding movie clip will be selected for the trailer.

[0030] Besides selecting key film clips, it's also necessary to determine the order of the selected film clips and align them with the audio clips. The third part is a film-audio clip matcher based on the Sinkhorn algorithm. Given the fused features of the film and audio clips, a distance matrix between the features can be constructed. The elements in the matrix Indicates the first The characteristics of the first film clip and the first The distance between audio segment features, such as Euclidean distance and Chebyshev distance, is considered. The Sinkhorn matching network achieves alignment of different feature representations by solving the following entropic optimal transport (EOT) problem: , in, Representation matrix sum matrix inner product calculation, , indicating all transmission schemes The set of, where The requirement to be met is: for The elements of the column vector obtained by summing each row are equal to The element at the corresponding position in the vector. yes The sum of the elements in a column vector; The row vector obtained by summing each column is a vector of length . All elements are row vectors The solution to the above EOT problem. The optimal transmission scheme is the expected distribution of movie clip features and audio clip features that minimizes the sum of Euclidean distances between features. The elements in the column vector determine the distribution of movie clips, which is also the result of normalizing the predicted probabilities of the movie clip selector. The closer this result is to the selection of movie clips in the trailer, the better. Since each audio clip in the trailer is used in the trailer's generation, this will... Set an element to all The row vectors. Finally, the entropy regularization term improves the smoothness of the EOT problem. This is the weight that controls the entropy regularization term, and its value is a real number greater than 0. The calculated optimal transmission scheme... Let the elements in this matrix be . You can select relatively aligned movie clips based on audio segments, specifically by calculating a matrix. The row index corresponding to the maximum value in each column of elements. ,Right now ,in The row index corresponds to the first row. Given several movie clips, we can obtain movie clips with corresponding audio clips. The EOT problem described above can be solved using the Sinkhorn algorithm, the algorithm flow of which is as follows: Figure 2As shown, the specific steps are as follows: Step 1: Initialize the matrix ,in It is the Euclidean distance matrix between the features of film clips and audio clips. This is the weight of the entropy regularization term. Initialize a term of length... A column vector whose elements are all 1 .

[0031] Step 2: According to and renew and .

[0032] Step 3: Repeat step 2 until convergence. The convergence criterion is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0033] Model training algorithm The above IPOT model was trained using a two-layer alternating optimization algorithm to obtain better model network parameters.

[0034] Let the network parameter set in the above IPOT model be denoted as... In a given dataset In this case, the IPOT model can be learned in a supervised manner by solving the Inverse Partial Optimal Transport Problem (IPOT). The Inverse Partial Optimal Transport Problem (IPOT) is defined as follows:

[0035]

[0036] The IPOT problem is a two-layer optimization problem. The upper layer problem is the loss function for training the network, and the solution of the lower layer problem is used to calculate the value of the upper layer loss.

[0037] In the higher-level problem, movie clips in the known dataset are... and trailer audio clips Alignment matrix between (in the matrix) OK An element of the column being 1 indicates the movie's... The first clip and the trailer The audio clips match in the official trailer, except for the matching positions where the elements are 1. The remaining elements in the matrix are 0. We normalize the alignment matrix to... This was used as the actual optimal transmission scheme to supervise the learning of the movie-audio clip matcher. The calculated length is column vectors The elements in the expression represent the true probability of each movie clip being selected (1 if it appears in a trailer, 0 otherwise), used to supervise the learning of the movie clip selector. For each movie-trailer pair, the upper-level problem contains... and Of these two items, The term is the optimal transfer matrix predicted by the model. and the true optimal transmission scheme The Kullback-Leibler divergence (KL divergence) between them. Indicates the first The number of audio clips in a movie trailer; The term is a column vector consisting of the probability of each movie segment being selected, predicted by the model. A column vector consisting of the probability of each actual movie clip being selected. The binary cross-entropy loss (BCE loss) between the two factors predicts the results. Compared with the actual results The closer the values ​​are, the better the selector has been trained. Hyperparameters The weight ratio between KL divergence and BCE loss is controlled, preferably set to 1.0 in this application. (The last sentence appears to be incomplete and possibly refers to a higher-level problem.) It is obtained by solving lower-level problems.

[0038] The lower-level model is the entropy-partial optimal transport problem, a variation of the EOT problem, which only... Replaced with , and These are respectively represented as model parameters. The distance matrix and optimal transmission scheme are calculated in real time. Since only a small portion of all movie clips in a film are selected for the trailer, the column vector... It is relatively sparse, consisting mostly of selection probabilities of 0 and a small portion of selection probabilities of 1. This is relevant in solving the lower-level problem of predicting the matching relationship between selected movie clips and trailer audio clips. In this case, only the correspondence between the selected film clips and audio clips needs to be considered, not the correspondence between all film clips and audio clips. Therefore, IPOT is called the Partial Optimal Transmission Problem.

[0039] Known It is by Calculated yes , Indicates length is The values ​​are all The row vector. When the network model parameters are Distance matrix between all movie clip features and audio clip features In the matrix OK The elements of the column represent the first... The characteristics of the first film clip and the first The Euclidean distance between the features of each audio segment. Representation matrix sum matrix The inner product calculation. The goal of the entropy-partial optimal transport problem is to know the distribution of all real movie clips. and audio distribution Predict the optimal transmission scheme under these circumstances. ,matrix This demonstrates the matching relationship between film clips and audio clips (similar to the above). ), Predicted results vs. actual results The closer the matcher is to the original, the better it is trained. It can be calculated using the Sinkhorn algorithm, the algorithm flow is as follows: Step 1: Initialize the matrix ,in It is the Euclidean distance matrix between the features of film clips and audio clips. This is the weight of the entropy regularization term. Initialize a term of length... A column vector whose elements are all 1 .

[0040] Step 2: According to and renew and , For matrix The transpose of .

[0041] Step 3: Repeat Step 2 until convergence. The criterion for convergence is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0042] Model network parameter set The training is solved using stochastic gradient descent (SGD). Specifically, given a dataset... First, the optimal transmission scheme between movie clips and audio clips is obtained by solving the underlying problems. This allows us to calculate the sum of the KL divergence and BCE loss in the upper-level problem as the complete loss of the model, and then use SGD to update the model parameters based on this loss. The model is trained repeatedly using the dataset for 500 epochs (training epoch=500), and the learning rate for updating the model parameters using SGD is 1e-5.

[0043] Model training process Given a set of movies and their corresponding trailers, denoted as . .in, Indicates the first The film contains A movie clip, which is a series of different video segments that occur in a movie; Indicates the first The trailer for the movie contains These trailer clips are selected from movie clips, so an alignment matrix can be constructed. ,when Time indicates the first The first movie The trailer clip corresponds to the first... A movie clip; This indicates that the trailer audio will be based on The audio segments obtained after splitting each trailer clip by its timestamp are all one-to-one correspondences between audio and trailer clips, so each audio clip has the same duration as its corresponding trailer clip. Also includes Audio clips, alignment matrix It can also be used to establish a correspondence between film clips and audio clips. Additionally, from... The vector can be calculated ,in Represents a length of A column vector consisting entirely of 1s. Each element in the text represents the probability that each movie clip will be selected for the trailer.

[0044] Given input , Step 1, based on and Constructing the alignment matrix Through the formula Calculate vectors ,in Represents a length of A column vector consisting entirely of 1s. Each element in the text represents the probability that each movie clip will be selected for the trailer.

[0045] Step 2, place the first A collection of movie clips and audio clip collection Video segment features are obtained by inputting the video encoder and audio encoder respectively. and audio segment features ,in The dimension of the extracted features; and These represent the number of film clips and audio clips, respectively; both the video encoder and the audio encoder are neural networks composed of the ImageBind model, a multilayer perceptron, and a self-attention module.

[0046] Step 3, and The input cross-attention module captures the interaction between film and audio features, generating fused film clip features. and audio segment features ;Will The length is obtained after passing through a multilayer perceptron and a sigmoid function. column vectors The selector is composed of a cross-attention module, a multilayer perceptron, and a sigmoid function.

[0047] Step 4, based on and Calculate the distance matrix between movie clip features and audio clip features. , This represents the set of parameters for the video encoder, audio encoder, and selector neural network, where... OK The elements of the column represent the first... The characteristics of the first film clip and the first Euclidean distance between features of each audio segment. Initialization For length is All of The row vector.

[0048] Step 5: Using the formula Calculate matrix hyperparameters This is the weight of the entropy regularization term, set to 1.0. Initialize a variable of length... A column vector whose elements are all 1 .

[0049] Step 6, according to and renew and , For matrix The transpose of .

[0050] Step 7: Repeat step 6 until convergence. The criterion for convergence is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0051] Step 8: According to the formula Calculate the loss function, where The term is the optimal transfer matrix predicted by the model. and the true optimal transmission scheme The Kullback-Leibler divergence (KL divergence) between them. The term is a column vector consisting of the probability of each movie segment being selected, predicted by the model. A column vector consisting of the probability of each actual movie clip being selected. The binary cross-entropy loss (BCE loss) is calculated between the two parameters. The weight ratio between KL divergence and BCE loss is controlled and set to 1.0 in this invention. Based on this loss, SGD is used to update the model network parameters. The learning rate for updating model parameters using SGD is 1e-5.

[0052] Step 9: Repeat steps 4 to 8 until the training epochs reach 500 (500 training epochs), obtaining the trained IPOT model. The network parameter set of the model at this point is... .

[0053] Movie trailer generation process After training the model based on the above-mentioned reverse partial optimal transmission framework and two-layer alternating optimization algorithm, let the parameters of the model at this time be denoted as . The following describes the process of generating a movie trailer using a trained model, given a new movie and a piece of music.

[0054] The resolution of the given new movie is reduced to 320p, and the movie is split into segments using the BaSSL video splitting tool. For a given movie clip and its accompanying music, the process begins by using the UVR vocal separation tool to remove vocals, followed by using the Ruptures audio segmentation tool to segment the music into... A few audio segments are used to obtain the input. .

[0055] 1. Compile movie clips and audio clip collection Video segment features are obtained by inputting the video encoder and audio encoder respectively. and audio segment features ,in The dimension of the extracted features; and These represent the number of film clips and audio clips, respectively; both the video encoder and the audio encoder are neural networks composed of the same ImageBind model, multilayer perceptron, and self-attention module as in the training process.

[0056] 2. and The input cross-attention module captures the interaction between film and audio features, generating fused film clip features. and audio segment features ;Will The length is obtained by using the same multilayer perceptron and sigmoid function as the trained model. column vectors .

[0057] 3. According to Select the element with the highest value. A movie clip. Calculate this. Features of film clips and Distance matrix between features of each audio segment The elements in the matrix are calculated as follows: hyperparameters Set the parameter controlling the weight ratio of these two items to 1.0. Initialize a string of length... All of column vectors and a length of All of row vectors .

[0058] 4. Through the formula Calculate matrix hyperparameters This is the weight of the entropy regularization term, set to 1.0. Initialize a variable of length... A column vector whose elements are all 1 .

[0059] 5. According to and renew and , For matrix The transpose of .

[0060] 6. Repeat step 6 until convergence. The criterion for convergence is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0061] 7. Finding the matrix The row index corresponding to the maximum value of each column element represents the correspondence between audio clips and movie clips. For each music clip, if the length of the corresponding movie clip exceeds the length of the music clip, the movie clip will be cut to match the music clip length. If the movie clip length is shorter than the music clip length, adjacent movie clips will be merged to match the music clip length. The processed movie clips are arranged and spliced ​​according to the order of the audio clips, with the complete music used as background music, to obtain the final movie trailer.

[0062] In another implementation, the solution can be implemented as a device, which may include corresponding modules that perform one or more steps in the various embodiments described above. Therefore, each or more steps in the various embodiments can be performed by a corresponding module, and the electronic device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0063] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this solution includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of this solution pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this solution can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0064] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0065] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0066] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating a video guided by audio, characterized in that: Segment audio and video into sequential audio segments and video segments; Inputting the audio clip and the video clip into a neural network to obtain a corresponding relationship between the audio clip and the video clip; According to the timing and duration of the audio clips and the corresponding relationship between the audio and video clips, the duration of the corresponding video clips is adjusted and then spliced ​​in sequence to generate a target video; the audio is used as background music for the target video; The neural network includes a video encoder, an audio encoder, a video selector and an audio and video matcher; The video encoder is used to encode the video clip to obtain the video clip features; the audio encoder is used to encode the audio clip to obtain the audio clip features; The video selector is used to obtain video segment fusion features and audio segment fusion features based on the video segment features and the audio segment features, and obtain the selection probability of each video segment based on the video segment fusion features; The audio and video matcher is used to obtain the correspondence between the audio segment and the video segment through audio and video segment alignment based on the audio segment fusion feature, the video segment fusion feature, and the selection probability of the video segment.

2. The video generation method according to claim 1, characterized in that: The video encoder includes a cross-modal feature extraction model, a multi-layer perceptron MLP and a self-attention module SA connected in sequence; the audio encoder includes a cross-modal feature extraction model, a multi-layer perceptron MLP and a self-attention module SA connected in sequence.

3. The video generation method according to claim 2, characterized in that: The cross-modal feature extraction model is a pre-trained ImageBind model.

4. The video generation method according to claim 3, characterized in that: The video selector comprises a cross attention module, a second multi-layer perceptron and an activation function connected in sequence; the cross attention module is used to capture the interaction between the input video segment features and the audio segment features, and output the video segment fusion features and the audio segment fusion features; The second multi-layer perceptron is used to reduce the dimension of the video segment fusion features into a vector, and the vector is activated by an activation function to obtain the selection probability of each video segment; the activation function is a Sigmoid function.

5. The video generation method according to claim 4, characterized in that: The audio and video matcher obtains the optimal transmission matrix by solving the entropy optimal transmission EOT problem , get the correspondence between audio clips and video clips: in, A distance matrix representing the fusion features of the video clips and the fusion features of the audio clips; T is the transmission scheme, representing the correspondence matrix between video clips and audio clips, Representation Matrix and matrix Inner product calculation of ; , indicating all transmission schemes A collection of In the calculation process, the following needs to be satisfied: The elements of the column vector obtained by summing each row of The elements at corresponding positions in the vector, yes The sum of the elements in a column vector, is the selection probability vector of the video segment; for , Indicates the length is , all values ​​are The row vector of is the number of audio clips; hyperparameter is the entropy regularization term The weight of Representation Matrix and matrix Inner product calculation.

6. The video generation method according to claim 5, characterized in that: The EOT optimization problem is solved by the Sinkhorn algorithm; the Sinkhorn algorithm process is as follows: Step 1: Set up the matrix , initialize a length Column vectors of the same dimension, all elements are 1 ; Step 2: Follow and renew and ; Step 3: Repeat step 2 until convergence. The convergence criterion is The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than the preset threshold, diag Represents the generation of a vector A square matrix with diagonal elements. Represents the generation of a vector A square matrix whose elements are diagonal elements; according to Calculate the optimal transmission matrix .

7. The video generation method according to claim 6, characterized in that: The loss function during the neural network training is: in, represents the number of data in the training dataset, represents the set of parameters of the neural network, is the optimal transmission matrix predicted by the EOT problem and the true alignment matrix Between Divergence, Indicates the training data set The number of video clips of the target video, Indicates the training data set based on An alignment matrix constructed by combining the first video clip with the target video clip; is the first The probability of selecting a video clip from a video and a column vector consisting of the probability of the real video segment being selected The binary cross entropy between loss, , Represents a length of A column vector of all 1s, hyperparameters yes The weight of the loss.

8. The video generation method according to claim 5, characterized in that: Before using the audio and video matcher to obtain the corresponding relationship between the audio segment and the video segment, according to Select the one with the highest probability The video clip corresponding to the element is calculated. The video clip fusion features and The distance matrix between the fused features of the audio clips , The element in row i and column j The calculation method is , where the hyperparameters represents the weight, represents the selected fusion feature of the i-th video clip, represents the fusion feature of the jth audio segment, represents the Euclidean distance between the fusion feature of the i-th video clip and the fusion feature of the j-th audio clip, represents the duration of the i-th video segment, Indicates the absolute value of the time difference between the i-th video segment and the j-th audio segment; set The length is The values ​​of Column vector of .

9. An electronic device, characterized in that: include: processor; A memory storing a program executable by the processor, wherein when the program is executed, the processor executes the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A program executable by a processor is stored, and when the program is executed, the processor executes the method described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Method and device for multi-thread video and audio synchronous control

    CN101984672A

  • Video feature extraction method and device and electronic equipment

    CN115311595A

  • Audio-visual event positioning method and device, model training method and device, equipment and medium

    CN116246214A

  • Video generation method and apparatus, electronic device, and computer readable medium

    US20220223183A1