An audio-guided video generation method, device and medium

Abstract

The application provides an audio-guided video generation method and device and a medium, comprising: segmenting audio and video into sequence audio segments and video segments; inputting the audio segments and video segments into a neural network to obtain the correspondence relationship between the audio segments and the video segments; according to the time sequence and time length of the audio segments and the correspondence relationship between the audio segments and the video segments, adjusting the time length of the corresponding video segments and then splicing to generate a target video in sequence; and the audio is used as background music of the target video. By using the above technical scheme, the target video and the training data set can be generated by automatic technology, the challenge caused by manual labeling can be avoided, the scale of the training data set can be expanded, the work efficiency of a clipper can be improved, and inspiration of a creator can be stimulated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video generation, in particular to a method, device and medium for audio-guided video generation based on a neural network. BACKGROUND

[0002] Movie trailers, as a set of exciting movie clips that can attract audiences, play an important role in the promotion of movies. In a trailer, not only do attractive movie clips need to be selected, but the movie plot also needs to be hidden to some extent, and the selected movie clips need to be reordered. The selection and ordering of movie clips can be determined by many factors, such as the semantics of background music, the synchronization of video and audio content, the logical fluency of character dialogue, etc., which reflect a deep understanding of the movie. Therefore, generating a high-quality movie trailer involves complex editing work, making it a highly challenging task.

[0003] Early methods mainly rely on additional information to select and order video shots, such as trying to identify impressive audiovisual components through emotional content analysis to help generate trailers. Later, with the development of machine learning technology, some learning-based methods were proposed, gradually automating the trailer generation process. Among them, point process-based methods learn visual attraction models through visual fixation information collected by professional eye trackers. Emotion-related methods manually annotate corresponding emotion categories for video and audio clips, rely on emotion categories to align images, text, and audio in a latent space, and select and reorganize video clips by maximizing emotion scores. Commercial software Muvee can achieve music-based video editing and trailer synthesis.

[0004] When using background music information, existing music-based movie trailer generation methods mainly focus on synchronizing the switching of movie clips according to the rhythm of the music, such as the accents of the music corresponding to the switching of the pictures, while ignoring the semantic alignment information between video and audio information, such as low-volume and slow-paced music clips usually appearing simultaneously with video clips showing natural environments. In addition, existing learning-based methods heavily rely on videos with detailed annotations or additional information, such as frame-level fixation scores, manually defined emotion labels, etc. These data annotation is difficult and time-consuming, resulting in limited dataset size and high risk of overfitting of the trained model. SUMMARY

[0005] To partially or completely solve the above technical problems, the present application automatically obtains semantic alignment information between audio and video based on a neural network, and proposes a method, device and medium for audio-guided video generation, which can automatically edit video to generate a trailer for a given audio and movie.

[0006] The above technical purpose is achieved by the following technical solutions.

[0007] In a first aspect, this application provides an audio-guided video generation method, comprising: segmenting audio and video into sequential audio segments and video segments; and inputting the audio segments and video segments into a neural network to obtain the correspondence between the audio segments and the video segments;

[0008] Based on the timing and duration of the audio segments, as well as the correspondence between the audio and video segments, the duration of the corresponding video segments is adjusted and then spliced ​​together in sequence to generate the target video; the audio is used as the background music for the target video.

[0009] According to some embodiments, the neural network includes a video encoder, an audio encoder, a video selector, and an audio-video matcher; the video encoder is used to encode video segments to obtain video segment features; the audio encoder is used to encode audio segments to obtain audio segment features; the video selector is used to obtain video segment fusion features and audio segment fusion features based on the video segment features and audio segment features, and to obtain the selection probability of each video segment based on the video segment fusion features; the audio-video matcher is used to obtain the correspondence between audio segments and video segments by aligning audio and video segments based on the audio segment fusion features, video segment fusion features, and the selection probability of video segments.

[0010] According to some embodiments, the video encoder includes a cross-modal feature extraction model, a multilayer perceptron (MLP), and a self-attention module (SA) connected in sequence; the audio encoder includes a cross-modal feature extraction model, a multilayer perceptron (MLP), and a self-attention module (SA) connected in sequence.

[0011] According to some embodiments, the cross-modal feature extraction model is a pre-trained ImageBind model.

[0012] According to some embodiments, the video selector includes a cross-attention module, a second multilayer perceptron, and an activation function connected in sequence; the cross-attention module is used to capture the interaction between the input video segment features and audio segment features, and output video segment fusion features and audio segment fusion features; the second multilayer perceptron is used to reduce the dimensionality of the video segment fusion features to a vector, and the vector is passed through an activation function to obtain the selection probability of each video segment; the activation function is the Sigmoid function.

[0013] According to some embodiments, the audio / video matcher obtains the optimal transmission matrix by solving the entropy-optimal transmission (EOT) problem. To obtain the correspondence between audio segments and video segments:

[0014]

[0015] in, The distance matrix representing the fused features of video segments and the fused features of audio segments; T For the transmission scheme, a matrix representing the correspondence between video and audio segments is provided. Representation matrix sum matrix Calculation of the inner product; , indicating all transmission schemes The set of, where Requirements: (The last part is incomplete and likely refers to a specific condition or requirement.) The elements of the column vector obtained by summing each row are equal to The element at the corresponding position in the vector. yes The sum of the elements in the column vector. It is the selection probability vector of video segments; for , Indicates length is The values ​​are all column vectors, The number of audio segments; hyperparameters It is an entropy regularization term. The weight, Representation matrix sum matrix The inner product calculation.

[0016] According to some embodiments, the EOT problem is solved using the Sinkhorn algorithm; the Sinkhorn algorithm process is as follows:

[0017] Step 1: Set up the matrix Initialize a length and A column vector with the same dimensions and all elements equal to 1. ;

[0018] Step 2: According to and renew and ;

[0019] Step 3: Repeat step 2 until convergence. The convergence criterion is... If the sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than a preset threshold, then diag This indicates the generation of a vector. A square matrix whose elements are diagonal elements. This indicates the generation of a vector. A square matrix whose elements are diagonal elements; according to Calculate the optimal transmission matrix .

[0020] According to some embodiments, the loss function when training the neural network is:

[0021]

[0022] wherein, denotes the number of data in the training data set, denotes the parameter set of the neural network, is the divergence between the optimal transmission matrix predicted by the EOT problem and the real alignment matrix denotes the number of video segments of the i-th target video in the training data set, denotes the alignment matrix constructed based on the i-th video segment and the target video segment in the training data set; is the binary cross-entropy loss between the video segment selection probability predicted by the neural network for the i-th video and the column vector composed of the real video segment selection probability , denotes a column vector of length with all 1s, and the hyperparameter is the weight of the loss.

[0023] According to some embodiments, before obtaining the corresponding relationship between the audio segment and the video segment by using the audio-video matcher, the video segment corresponding to the element with the highest probability value in is selected, the distance matrix between the fusion features of the video segments and the fusion features of the audio segments is calculated , the calculation method of the element in the i-th row and the j-th column in the distance matrix is , wherein the hyperparameter denotes the weight, denotes the i-th selected video segment fusion feature, denotes the j-th audio segment fusion feature, denotes the Euclidean distance between the i-th video segment fusion feature and the j-th audio segment fusion feature, denotes the length of the i-th video segment, denotes the absolute value of the time length difference between the i-th video segment and the j-th audio segment; and is set to length ​​​​​​​​​the values of the columns of the matrix a column vector.

[0024] In a second aspect, the present application also provides an electronic device, comprising: a processor; and a memory storing a program executable by the processor, when the program is executed, the processor executes the method of any one of the preceding first aspect.

[0025] In a third aspect, the present application also provides a storage medium, the storage medium stores a program executable by a processor, when the program is executed, the processor executes the method of any one of the preceding first aspect.

[0026] As will be described in detail below, according to the neural network-based audio-guided video generation method, device and medium of the embodiments of the present application, by constructing a neural network model based on partial reverse optimal transmission, for a given piece of audio, the highlight clips are automatically selected and sorted from a long video such as a movie or a TV series, and then organized into a trailer of the long video. With the above technical solution, not only can the preview be generated based on the semantic information between audio and video, but also the target video and the training data set can be generated by automatic technology, which eliminates the challenge of manual annotation, expands the scale of the training data set, improves the work efficiency of the editor, and also helps to inspire the inspiration of the creators. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0028] Figure 1 An IPOT neural network model architecture diagram according to an example embodiment is shown;

[0029] Figure 2 A Sinkhorn algorithm calculation flowchart according to an example embodiment is shown;

[0030] Figure 3 An audio-guided video generation method flow according to an example embodiment is shown. DETAILED DESCRIPTION

[0031] The technical solutions of the present application will be described clearly and completely below in conjunction with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0032] The audio-guided video generation method proposed in the present application, specifically, defines the target video generation task as video segment selection and sorting according to a given audio segment, and uses a learning model of an inverse partial optimal transport (IPOT) framework to implement this task. First, the features of the video segment and the audio segment are extracted by using an encoder, then the IPOT framework applies a selection network with an attention mechanism to parameterize the distribution of the video segment, and selects some video segments for constructing the trailer according to the distribution, then calculates the partial optimal transport distance between the latent features of the video segments and the audio segment to realize the alignment between the video segments and the audio segment, and finally arranges the movie segments according to the semantic alignment information.

[0033] The present application is applicable to the application scenario of selecting video segments in a video according to a given audio and synthesizing a target video from the given audio and the selected video segments. The most typical application scenario is generating a movie trailer from music and a movie original, and the technical solutions proposed in the present application will be described and explained in detail below in conjunction with the generation of a movie trailer. The same processing is applicable to the generation of a short film from a given audio for a TV series and other long videos, which will not be described again.

[0034] Generation of a movie trailer

[0035] Given a movie and a piece of music, the goal of the present application is to generate a trailer for the movie based on the music, and further, to convert the music-guided movie trailer generation task into the selection and sorting of movie segments based on audio segments. To achieve the above task, the present application proposes an IPOT neural network model to learn a multi-modal representation, selection and matching in a supervised manner. Of course, it is not limited to the IPOT neural network model, and any neural network can be used as long as it can select the video segments matching the audio content from the video based on semantic information.

[0036] IPOT neural network model

[0037] Figure 1 and Figure 3The network architecture of the IPOT model proposed in the application is shown, which is composed of three parts, two encoders for encoding video and audio features, a cross-attention module as a video segment selector, and a Sinkhorn matching network as a matcher between video segments and audio segments. Among them, the neural network structure involves two parts, which are the audio-video feature encoder and the cross-attention module.

[0038] For a given movie, the movie is segmented into movie segments using a video segmentation tool such as BaSSL , wherein contains movie segments, the trailer audio is segmented into audio segments using an audio segmentation tool Ruptures , wherein contains audio segments, and then the movie segments and audio segments are extracted through the pre-trained ImageBind model to extract movie segment features and audio segment features and , wherein and respectively represent the ImageBind model used to extract movie segment and audio segment features, is the dimension of the extracted features. Considering that the movie and audio features have a correlation in time, i.e., different movie segments have a sequence in the original movie, we further encode the extracted features through two multi-layer perceptions (MLP) and two self-attention (SA) modules (extracting temporal features). The pre-trained ImageBind model (multimodal feature extraction), multi-layer perception (adjusting feature dimension), and self-attention module (extracting the interaction between different segments) constitute the video encoder and the audio encoder. Based on the movie segment features and the audio segment features, the purpose of the application is to select key movie segments and align them with the audio segments of the trailer, thereby generating a trailer composed of key movie segments. This part will be realized through the following two parts.

[0039] The second part of the neural network is a movie segment selector based on cross-attention mechanism, which is used to predict which movie segments will appear in the movie trailer. The cross-attention module (Cross-attention, CA) in IPOT captures the interaction between movie and audio features and generates fused movie segment features and audio segment features The newly fused movie clip features will be processed through a multilayer perceptron (reduced to a vector) and a sigmoid function (the probability of each movie clip being selected), resulting in a vector of length [length missing]. column vectors Each element in the vector is a decimal between [0,1] because the projection of the Sigmoid function represents the probability that the corresponding movie clip will be selected for the trailer.

[0040] Besides selecting key film clips, it's also necessary to determine the order of the selected film clips and align them with the audio clips. The third part is a film-audio clip matcher based on the Sinkhorn algorithm. Given the fused features of the film and audio clips, a distance matrix between the features can be constructed. The elements in the matrix Indicates the first The characteristics of the first film clip and the first The distance between audio segment features, such as Euclidean distance and Chebyshev distance, is considered. The Sinkhorn matching network achieves alignment of different feature representations by solving the following entropic optimal transport (EOT) problem:

[0041] ,

[0042] in, Representation matrix sum matrix inner product calculation, , indicating all transmission schemes The set of, where The requirement to be met is: for The elements of the column vector obtained by summing each row are equal to The element at the corresponding position in the vector. yes The sum of the elements in a column vector; The column vector obtained by summing each column is a vector of length . All elements are column vectors The solution to the above EOT problem. The optimal transmission scheme is the expected distribution of movie clip features and audio clip features that minimizes the sum of Euclidean distances between features. The elements in the column vector determine the distribution of movie clips, which is also the result of normalizing the predicted probabilities of the movie clip selector. The closer this result is to the selection of movie clips in the trailer, the better. Since each audio clip in the trailer is used in the trailer's generation, this will... Set an element to all The column vectors. Finally, the entropy regularization term improves the smoothness of the EOT problem. This is the weight that controls the entropy regularization term, and its value is a real number greater than 0. The calculated optimal transmission scheme... Let the elements in this matrix be . You can select relatively aligned movie clips based on audio segments, specifically by calculating a matrix. The row index corresponding to the maximum value in each column of elements. ,Right now ,in The row index corresponds to the first row. Given several movie clips, we can obtain movie clips with corresponding audio clips. The EOT problem described above can be solved using the Sinkhorn algorithm, the algorithm flow of which is as follows: Figure 2 As shown, the specific steps are as follows:

[0043] Step 1: Initialize the matrix ,in It is the Euclidean distance matrix between the features of film clips and audio clips. This is the weight of the entropy regularization term. Initialize a term of length... A column vector whose elements are all 1 .

[0044] Step 2: According to and renew and .

[0045] Step 3: Repeat step 2 until convergence. The convergence criterion is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0046] Model training algorithm

[0047] The above IPOT model was trained using a two-layer alternating optimization algorithm to obtain better model network parameters.

[0048] Let the network parameter set in the above IPOT model be denoted as... In a given dataset In this case, the IPOT model can be learned in a supervised manner by solving the Inverse Partial Optimal Transport Problem (IPOT). The Inverse Partial Optimal Transport Problem (IPOT) is defined as follows:

[0049]

[0050]

[0051] The IPOT problem is a bi-level optimization problem, where the upper level problem is the loss function of training the network, and the solution of the lower level problem will be used to calculate the value of the upper level loss.

[0052] In the upper level problem, the alignment matrix between the movie segments and the trailer audio segments in the dataset is known. The element in the matrix is 1 if the th segment of the movie is matched with the th audio segment in the official trailer, except for the matched position whose element is 1. The element in the rest of the matrix is 0. We normalize the alignment matrix as and use it as the ground truth optimal transport scheme to supervise the learning of the movie-audio segment matcher. The element in the column vector of length calculated by indicates the real probability of each movie segment being selected (if it appears in the trailer, the probability of the movie segment being selected is 1, otherwise it is 0, which is used to supervise the learning of the movie segment selector. For each movie-trailer pair, the upper level problem contains and two terms, where is the Kullback-Leibler divergence (KL divergence for short) between the optimal transport matrix predicted by the model and the real optimal transport scheme , represents the number of trailer audio segments in the th movie trailer; is the Binary Cross Entropy Loss (BCE loss for short) between the column vector composed of the probability of each movie segment being selected predicted by the model and the column vector composed of the real probability of each movie segment being selected, the closer the predicted result is to the real result , the better the selector is trained. The hyperparameter controls the weight ratio between the KL divergence and the BCE loss, which is preferably set to 1.0 in this application. The in the upper level problem is obtained by solving the lower level problem.

[0053] ​​The lower-level model is the entropy-partial optimal transport problem, a variation of the EOT problem, which only... Replaced with , and These are respectively represented as model parameters. The distance matrix and optimal transmission scheme are calculated in real time. Since only a small portion of all movie clips in a film are selected for the trailer, the column vector... It is relatively sparse, consisting mostly of selection probabilities of 0 and a small portion of selection probabilities of 1. This is relevant in solving the lower-level problem of predicting the matching relationship between selected movie clips and trailer audio clips. In this case, only the correspondence between the selected film clips and audio clips needs to be considered, not the correspondence between all film clips and audio clips. Therefore, IPOT is called the Partial Optimal Transmission Problem.

[0054] Known It is by Calculated yes , Indicates length is The values ​​are all Column vectors. When the network model parameters are Distance matrix between all movie clip features and audio clip features In the matrix OK The elements of the column represent the first... The characteristics of the first film clip and the first The Euclidean distance between the features of each audio segment. Representation matrix sum matrix The inner product calculation. The goal of the entropy-partial optimal transport problem is to know the distribution of all real movie clips. and audio distribution Predict the optimal transmission scheme under these circumstances. ,matrix This demonstrates the matching relationship between film clips and audio clips (similar to the above). ), Predicted results vs. actual results The closer the matcher is to the original, the better it is trained. It can be calculated using the Sinkhorn algorithm, the algorithm flow is as follows:

[0055] Step 1: Initialize the matrix ,in It is the Euclidean distance matrix between the features of film clips and audio clips. is the weight of the entropy regularization term. Initialize a column vector with length .

[0056] Step 2: Update and according to and , is the transpose of matrix .

[0057] Step 3: Repeat Step 2 until convergence, the criterion of convergence is that the sum of the absolute value of the change of the element in the matrix is less than 1e-7. The optimal transport matrix is calculated according to diag means generating a square matrix with the elements in the vector or as the diagonal elements.

[0058] The training of the model network parameter set is solved by stochastic gradient descent (SGD). Specifically, given a data set , first obtain the optimal transport scheme between the movie segments and the audio segments by solving the lower problem , and then the sum of the KL divergence and the BCE loss in the upper problem can be calculated as the complete loss of the model, and the model parameters are updated based on the loss using SGD. Repeat the training of the model using the data set, the number of repeated rounds is 500 (epoch of training = 500), and the learning rate of the SGD to update the model parameters is 1e-5.

[0059] Model training process

[0060] Given a set of movies and the respective trailers of the set of movies, denoted as . Among them, represents the th movie contains movie segments, that is, different video segments that occur continuously in the movie;

[0061] represents the trailer of the th movie, and the trailer contains trailer segments. Since these trailer segments are selected from the movie segments, an alignment matrix can be constructed, where indicates that the th trailer segment of the th movie corresponds to the th movie segment.​ This indicates that the trailer audio will be based on The audio segments obtained after splitting each trailer clip by its timestamp are all one-to-one correspondences between audio and trailer clips, so each audio clip has the same duration as its corresponding trailer clip. Also includes audio clips, alignment matrix It can also be used to establish a correspondence between film clips and audio clips. Additionally, from... The vector can be calculated ,in Represents a length of A column vector consisting entirely of 1s. Each element in the text represents the probability that each movie clip will be selected for the trailer.

[0062] Given input ,

[0063] Step 1, based on and Constructing the alignment matrix Through the formula Calculate vectors ,in Represents a length of A column vector consisting entirely of 1s. Each element in the text represents the probability that each movie clip will be selected for the trailer.

[0064] Step 2, place the first A collection of movie clips and audio clip collection Video segment features are obtained by inputting the video encoder and audio encoder respectively. and audio segment features ,in The dimension of the extracted features; and These represent the number of film clips and audio clips, respectively; both the video encoder and the audio encoder are neural networks composed of the ImageBind model, a multilayer perceptron, and a self-attention module.

[0065] Step 3, and The input cross-attention module captures the interaction between film and audio features, generating fused film clip features. and audio segment features ;Will The length is obtained after passing through a multilayer perceptron and a sigmoid function. column vectors The cross-attention module, the multi-layer perceptron and the Sigmoid function constitute the selector.

[0066] Step 4, based on and Calculate the distance matrix between the movie segment features and the audio segment features , represents the parameter set of the video encoder, the audio encoder, and the selector neural network, wherein The elements in the row The element in the column represents the Euclidean distance between the th movie segment feature and the th audio segment feature. Initialize as a column vector with a length of and all elements being .

[0067] Step 5, calculate the matrix by the formula , wherein the hyperparameter is the weight of the entropy regularization term, which is set to 1.0. Initialize a column vector with a length of and all elements being 1.

[0068] Step 6, update and , and , is the transpose of the matrix .

[0069] Step 7, repeat step 6 until convergence, the criterion for convergence is that the sum of the absolute values of the changes in the elements of the matrix is less than 1e-7. Calculate the optimal transport matrix according to diag represents generating a square matrix with the elements in the vector or as the diagonal elements.

[0070] Step 8, calculate the loss function according to the formula , wherein is the Kullback-Leibler divergence (KL divergence) between the optimal transport matrix predicted by the model and the real optimal transport scheme , is the column vector composed of the selection probabilities of each movie segment predicted by the model, and the column vector composed of the selection probabilities of each movie segment.Binary Cross Entropy Loss (BCE loss) between the two. The hyperparameter controls the weight ratio between the KL divergence and the BCE loss, which is set to 1.0 in this work. The model network parameters are updated using SGD based on this loss , with a learning rate of 1e-5 for the SGD to update the model parameters.

[0071] Step 9, repeat steps 4 to 8 until the training epoch reaches 500 (training epoch = 500), and get the trained IPOT model, at this time the model network parameter set is .

[0072] Movie trailer generation process

[0073] After training the model based on the above reverse partial optimal transport framework and double-layer alternating optimization algorithm, the parameters of the model at this time are , and the process of generating a movie trailer using the trained model given a new movie and a piece of music will be described below.

[0074] The resolution of the given new movie is reduced to 320p, and the movie is segmented into movie segments using the BaSSL video segmentation tool. For the given music, first remove the vocals in the music using the UVR vocal separation tool, and segment the music into audio segments using the Ruptures audio segmentation tool, to get the input .

[0075] 1. Input the movie segment set and the audio segment set into the video encoder and the audio encoder respectively to get the video segment feature and the audio segment feature , where is the dimension of the extracted feature; and represent the number of movie segments and audio segments respectively; the video encoder and the audio encoder are both neural networks composed of the same ImageBind model, multi-layer perception and self-attention module as in the training process.

[0076] 2. Input and into the cross-attention module to capture the interaction between the movie and audio features, and generate the fused movie segment feature and the fused audio segment feature ; input The length is obtained by using the same multilayer perceptron and sigmoid function as the trained model. column vectors .

[0077] 3. According to Select the element with the highest value. A movie clip. Calculate this. Features of film clips and Distance matrix between features of each audio segment The elements in the matrix are calculated as follows: hyperparameters Set the parameter controlling the weight ratio of these two items to 1.0. Initialize a string of length... All of column vectors and a length of All of column vectors .

[0078] 4. Through the formula Calculate matrix hyperparameters This is the weight of the entropy regularization term, set to 1.0. Initialize a variable of length... A column vector whose elements are all 1 .

[0079] 5. According to and renew and , For matrix The transpose of .

[0080] 6. Repeat step 6 until convergence. The criterion for convergence is... The sum of the absolute values ​​of the changes in the element values ​​in the matrix is ​​less than 1e-7. According to... Calculate the optimal transmission matrix ,diag means to generate a vector or A square matrix whose elements are diagonal elements.

[0081] 7. Finding the matrix The row index corresponding to the maximum value of each column element, that is, the correspondence between the audio segment and the movie segment. For each music segment, if the movie segment corresponding thereto has a time length exceeding that of the music segment, the movie segment is cut so as to be consistent with the time length of the music segment, and if the movie segment has a time length shorter than that of the music segment, the movie segments adjacent to the movie segment are combined so as to be consistent with the time length of the music segment. The processed movie segments are arranged according to the order of the audio segments and spliced, the complete music is taken as background music, and a final movie trailer is obtained.

[0082] The scheme, in another implementation, can be implemented by means of an apparatus, which can include corresponding modules for performing each or several steps in the above-described various embodiments. Thus, each step or several steps of the above-described various embodiments can be performed by corresponding modules, and the electronic apparatus can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0083] Any process or method descriptions or any other descriptions herein can be understood as representing any manner of execution of instructions to achieve the desired result, and the schemes of preferred embodiments of the present scheme include additional implementations in which the functions described are performed in an order different from that shown or discussed, including functions performed in substantially simultaneous fashion, or in reverse order of discussion or sequence, as will be appreciated by those skilled in the art of the relevant art. Processors perform the various methods and processes described above. For example, the method embodiments in the present scheme can be implemented as a software program tangibly embodied in a machine-readable medium, such as a memory. In some embodiments, portions of the software program or all of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps of the above-described methods can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above-described methods by any other suitable means, such as by means of firmware.

[0084] The logic and / or steps represented in the flowcharts and / or otherwise described herein can be embodied in any readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0085] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0086] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An audio-guided video generation method, characterized in that, audio and video are segmented into sequence audio segments and video segments; the audio segments and video segments are input into a neural network to obtain the correspondence between the audio segments and video segments; the target video is generated by splicing the video segments in sequence after adjusting the length of the corresponding video segments according to the time sequence and length of the audio segments and the correspondence between the audio and video segments; the audio serves as the background music of the target video; the neural network comprises a video encoder, an audio encoder, a video selector and an audio-video matcher; the video encoder is used to encode the video segments to obtain video segment features; the audio encoder is used to encode the audio segments to obtain audio segment features; the video selector is used to obtain video segment fusion features and audio segment fusion features based on the video segment features and audio segment features, and obtain the selection probability of each video segment based on the video segment fusion features; The audio-video matcher is configured to obtain the corresponding relationship between the audio segments and the video segments by audio-video segment alignment based on the audio segment fusion features, the video segment fusion features, and the selection probabilities of the video segments; and the audio-video matcher is configured to obtain the optimal transmission matrix by solving an entropy optimal transmission (EOT) problem , and obtain the corresponding relationship between the audio segments and the video segments. wherein, denotes the distance matrix between the video segment fusion feature and the audio segment fusion feature; T is the correspondence matrix between the video segment and the audio segment for the transmission scheme, denotes the matrix and the matrix is the inner product calculation; denotes the set of all transmission schemes , wherein the following needs to be met in the calculation process: the element in the column vector obtained by summing each row of is equal to the element in the corresponding position of vector, is the sum of the elements in the column vector , is the selection probability vector of the video segment; is the , denotes the column vector with a length of and all values of , is the number of audio segments; the hyperparameter is the weight of the entropy regularization term , denotes the inner product calculation of the matrix and the matrix .

2. The video generation method of claim 1, wherein, the video encoder comprises a cross-modal feature extraction model, a multi-layer perception (MLP) and a self-attention module (SA) connected in sequence; the audio encoder comprises a cross-modal feature extraction model, a multi-layer perception (MLP) and a self-attention module (SA) connected in sequence.

3. The video generation method of claim 2, wherein, The cross-modal feature extraction model is a pre-trained ImageBind model.

4. The video generation method of claim 3, wherein, The video selector comprises a cross-attention module, a second multi-layer perception and an activation function connected in sequence; the cross-attention module is used to capture the interaction between the input video segment features and audio segment features, and output the video segment fusion features and audio segment fusion features; the second multi-layer perception is used to reduce the video segment fusion features to a vector, and the selection probability of each video segment is obtained by the activation function; the activation function is a Sigmoid function.

5. The video generation method of claim 4, wherein, The EOT problem is solved by a Sinkhorn algorithm; the Sinkhorn algorithm process is as follows: Step 1: Set up the matrix , initialize a column vector of length with all elements equal to 1 ; Step 2: According to and update and ; Step 3: Repeat Step 2 until convergence, the criteria of convergence is the sum of the absolute value of the change of the element value in the matrix is less than a preset threshold, diag represents generating a square matrix with the elements in vector as the diagonal elements, represents generating a square matrix with the elements in vector as the diagonal elements; according to calculating the optimal transmission matrix .

6. The video generation method of claim 5, wherein, The loss function during the training of the neural network is: wherein, denotes the number of data in the training dataset, denotes the parameter set of the neural network, is the optimal transmission matrix predicted by the EOT problem and the real alignment matrix between divergence, denotes the number of video segments of the th target video in the training dataset, denotes the alignment matrix constructed based on the th video segment and the target video segment in the training dataset; is the video segment selection probability of the th video predicted by the neural network and the real video segment selection probability form a column vector between binary cross-entropy loss, , denotes a column vector of 1 with length , the hyperparameter is the weight of the loss.

7. The video generation method of claim 6, wherein, Before using the audio-video matcher to obtain the correspondence between audio segments and video segments, according to... Choose the one with the highest probability value. Calculate the video clip corresponding to each element. Video segment fusion features and Distance matrix between fused features of audio segments , The element in the i-th row and j-th column The calculation method is as follows hyperparameters Indicates weight, This represents the fusion feature of the selected i-th video segment. This represents the fusion feature of the j-th audio segment. Let represent the Euclidean distance between the fused features of the i-th video segment and the fused features of the j-th audio segment. This represents the duration of the i-th video segment. This represents the absolute value of the duration difference between the i-th video segment and the j-th audio segment; [Settings are missing from the original text]. For length is All values ​​are Column vectors.

8. An electronic device, comprising: comprises: a processor; a memory storing programs executable by the processor, when the programs are executed, the processor executes the method of any one of claims 1-7.

9. A storage medium, characterized by programs executable by the processor, when the programs are executed, the processor executes the method of any one of claims 1-7.

Images