A method, device and storage medium for generating video background music

Abstract

The present invention relates to a method, device and storage medium for generating video background music, and to the technical field of video processing. The present invention decomposes a video to be processed into a target video image frame set and a target video soundtrack sequence, and extracts a target text description sequence and a target subtitle sequence respectively; extracts temporal target video image frame conditional features, target video soundtrack conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, the target video soundtrack sequence, the target text description sequence and the target subtitle sequence respectively through a conditional feature extraction module; processes each conditional feature into a target conditional feature through a background music conditional configurator; and introduces the target conditional feature into the latent space features of each reverse process by a pre-trained music generation diffusion model to generate background music.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device and storage medium for generating video background music. Background Art

[0002] With the development of video-based self-media platforms, videos have become a common way for self-media to deliver content and record information. For videos, a suitable background music is essential to make them more attractive. However, for people who do not understand music or video editing, it is not easy to choose or create music that is suitable or perfectly matched with videos. Most self-media select existing music that matches the content of the video as background music. In the process of commercialization of videos, the use of existing music may cause copyright issues and affect the commercial value of the video.

[0003] The task of video background music generation was first proposed by CMT (controllable music transformer) and has received increasing attention. Most existing background music generation methods use a Transformer-based framework and establish a relationship between video and music. CMT first encodes chords and notes to give the expression of the music, trains the model to understand the logic of the music, and then establishes three rhythmic relationships between the video and the background music (such as the movement speed of the video corresponds to the density of notes in the audio), narrowing the gap between the two expressions. V-MusProd and Video2Music also focus on the correspondence between the semantic level and the background music. The style of the generated music in the multimodal Transformer model is controlled by extracting the semantic features of the video. However, it is difficult for Transformer-based methods to control the end-to-end generation process, resulting in poor interpretability, and sometimes the Transformer model generates repetitive or overly patterned music, which may make the generated music sound monotonous and boring. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides a method, device and storage medium for generating video background music.

[0005] In a first aspect, the present invention provides a method for generating background music for a video, comprising:

[0006] Decomposing the video to be processed into a target video image frame set and a target video original sound sequence;

[0007] Extract a target text description sequence from a target video image frame set by extracting a text description sequence from a video image frame set; extract a target subtitle sequence from a target video original sound sequence by extracting a subtitle sequence corresponding to speech in a video original sound sequence, align the text description in the target text description sequence with the corresponding video image frame in the target video image frame set, and align the target subtitle sequence with the corresponding speech in the target video original sound sequence;

[0008] Extracting temporal target video image frame conditional features, target video original sound conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, target video original sound sequence, target text description sequence and target subtitle sequence respectively through a pre-trained conditional feature extraction module;

[0009] The user selects the combination of various target condition features at each stage of the video to be processed and configures the weights of various target condition features through the background music condition configurator according to the needs; based on the combination of target condition features and the weights of various target condition features, condition feature fusion is performed to finally obtain the target condition features at each stage of the video that meet the user's needs;

[0010] The pre-trained music generation diffusion model is used to perform forward Gaussian diffusion on the latent space features of the vector representation of the reference music. In the reverse process, the music generation diffusion model introduces the target conditional features into the latent space features of each reverse process through the pre-trained Unet model to obtain the latent space features of the background music, and decodes the latent space features of the background music to obtain the vector representation of the background music; the vector representation of the background music is restored to the background music that conforms to the content of the video to be processed.

[0011] Furthermore, the process of extracting the text description sequence method of the video image frame set includes: splitting the video image frame set into a plurality of continuous and non-overlapping groups with a set number of adjacent image frames as a group; processing the image frames of each group through the Video2Text model or the Picture2Text model to obtain the corresponding text description; the text descriptions of different groups form a text description sequence arranged in time sequence.

[0012] Furthermore, the method for extracting a subtitle sequence corresponding to speech in a video original sound sequence includes: obtaining a speech segment in a target video original sound sequence, converting the speech segment into text through speech-to-text conversion, aligning the text with the speech segment on a time axis, and obtaining a subtitle sequence.

[0013] Furthermore, the conditional feature extraction module includes: a text description conditional encoder and a subtitle conditional encoder based on the BERT model: the text description conditional encoder is used to extract target text description conditional features from the text description sequence of the video to be processed, and the subtitle conditional encoder is used to extract target subtitle conditional features from the subtitle sequence of the video to be processed;

[0014] A video conditional encoder based on a visual Transformer, wherein the video conditional encoder is used to extract target video image frame conditional features from image frames in a target video image frame set;

[0015] An audio conditional encoder based on a wav2vec model is used to extract target video original sound conditional features from a target video original sound sequence.

[0016] Furthermore, the background music condition configurator is expressed as:

[0017] ,

[0018] in, is the target video image frame condition feature, is the original sound condition feature of the target video, Describe conditional features for the target text, is the target subtitle condition feature, N is the number of iterations in the reverse process, and n is the current iteration number; , , and It is a user-configurable weight parameter;

[0019] For the video part with only image frames but no original sound The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature and the target text description condition feature ;

[0020] For video parts that have image frames and original sound, but the original sound does not contain voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature and the target text description condition feature. ;

[0021] For video parts with image frames and original sound, the original sound contains voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature, the target subtitle condition feature and the target text description condition feature. ;

[0022] Then, corresponding to all the iterations of the reverse process, N target condition features will be obtained: in the early stage of the reverse process, The associated conditional features mainly participate in the reverse process. In the later stage of the reverse process, The associated conditional features mainly participate in the reverse process.

[0023] Furthermore, the music generation diffusion model includes: a vector representation encoder for encoding the vector representation of the reference music into latent space features of the vector representation of the reference music;

[0024] Forward diffusion process: vector represents the latent space features of the encoder output After the forward diffusion process is processed, the forward diffusion process is iterated for N steps to transfer the probability distribution according to the Gaussian distribution, so that the latent space feature is diffused step by step, and each iteration step is expressed as:

[0025] ;

[0026] in, In the forward diffusion process The latent space features of the step, In the forward diffusion process The latent space features of the step, In the forward diffusion process, Transfer to The probability distribution of is a Gaussian distribution, In the forward diffusion process, Transfer to The adjustment coefficient of Gaussian distribution in the probability distribution transfer process, I is the covariance matrix; each iteration step obtains the corresponding latent space feature , N latent space features are obtained through N steps of iteration ;

[0027] Reverse process: Each iteration of the reverse process uses the Unet model to model the probability distribution of the reverse transfer process to convert the final latent space features obtained by the forward diffusion process into Restore Or will Convert to the conditional features , the probability distribution transfer of each iteration step of the reverse process is expressed as:

[0028] ;

[0029] in, Indicates the reverse process from Transfer to The probability distribution of Represents the reverse process parameterized by the Unet model The mean of Represents the reverse process parameterized by the Unet model The variance of

[0030] A vector representation decoder is used to decode the final latent space features of the backward process into a vector representation.

[0031] Furthermore, the vector representation encoder and the vector representation decoder are implemented based on a convolutional neural network.

[0032] Furthermore, the Unet model includes: multiple levels of encoders, an intermediate layer connecting the encoder and the decoder and a decoder of the corresponding level, the encoder in the Unet model includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to downsampling, and the decoder in the Unet model also includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to upsampling; in the Unet model, the output of the encoder of the previous level and the output of the decoder of the same level are combined and input to the encoder of the next level; the intermediate layer is a residual block, spatial attention and residual block structure;

[0033] The spatial attention includes the self-attention of the residual structure, the cross-attention of the residual structure and the feedforward fully connected layer of the residual structure; after the latent space feature is self-attentioned, the output of the self-attention is used as the query, and the self-conditional features of the corresponding reverse step in the target conditional features are used as the key and value for cross-attention.

[0034] Furthermore, the loss function of the training process of the music generation diffusion model includes: in the training dataset domain, in the forward diffusion process, each step introduces real noise by probability distribution transfer according to Gaussian distribution, and in the reverse process, each step Unet model predicts the expected L2 distance between the noise based on the given target feature conditions and the given initial latent space features;

[0035] The music generation diffusion model is trained with the goal of minimizing the loss function.

[0036] Furthermore, in the training dataset domain, the similarity of features between the generated background music and the actual background music is used to evaluate the quality of the music generation diffusion model, wherein the features include: structural index, pitch classification histogram entropy and singing pattern similarity.

[0037] In a second aspect, the present invention provides a video background music generating device, comprising: at least one processing unit, the processing unit is connected to a storage unit via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the video background music generating method is implemented.

[0038] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating video background music is implemented.

[0039] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0040] The present application decomposes the video to be processed into a target video image frame set and a target video soundtrack sequence, and extracts a target text description sequence and a target subtitle sequence respectively; extracts the target video image frame conditional features, target video soundtrack conditional features, target text description conditional features and target subtitle conditional features of the time sequence from the target video image frame set, the target video soundtrack sequence, the target text description sequence and the target subtitle sequence respectively through the conditional feature extraction module; processes each conditional feature into a target conditional feature through a background music condition configurator; and the pre-trained music generation diffusion model introduces the conditional features into the latent space features of each reverse process to generate background music. The music generation diffusion model of the present application integrates the target conditional features into the latent space features through the Unet model. During the entire reverse process, the latent space features gradually fuse the various target conditional features to ensure that the latent space features in the reverse process are always guided by the target conditional features of the video, thereby generating latent space features related to the video. Thus, high-quality background music is generated based on the latent space features related to the video. The target conditional features of the present application include target video image frame conditional features, target video soundtrack conditional features, target text description conditional features and target subtitle conditional features; the generation process of background music is constrained by multimodal conditional features; from the perspective of video soundtrack and background music, the rhythm and melody of the background music can be adapted to the changes of the video soundtrack, which is conducive to the harmony of the background music and the video soundtrack, and the emotion of the background music can be consistent with the emotion reflected by the semantics of the video soundtrack; from the perspective of video image and background music, the rhythm of the background music can not only be able to mark the sudden changes of the video scene, but also the melody of the background music can adapt to the evolution of the scene, and the emotion of the background music can adapt to the emotion reflected by the video scene.

[0041] For the video part with only image frames but no original sound The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature and the target text description condition feature ; For video parts that have image frames and original sound, but the original sound does not contain voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature and the target text description condition feature. ; For video parts with image frames and original sound, the original sound contains voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature, the target subtitle condition feature and the target text description condition feature. ; Then corresponding to all the iteration steps of the reverse process, N target condition features will be obtained.

[0042] In the background music condition configurator, at the beginning of the reverse process, The associated conditional features mainly participate in the reverse process. In the later stage of the reverse process, The associated conditional features mainly participate in the reverse process. At different stages of the reverse process, different conditional features are integrated into the latent space features to control the evolution of the latent space features in the reverse denoising process, ensuring that the latent space features obtained in the reverse process are consistent with the image and sound conditions of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of a method for generating video background music provided by an embodiment of the present invention;

[0046] Figure 2 A flowchart of a method for extracting a text description sequence of a video image frame set provided by an embodiment of the present invention;

[0047] Figure 3 An architecture diagram of the music generation diffusion model provided in an embodiment of the present invention for generating background music;

[0048] Figure 4 A schematic diagram of a vector representation of music provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a Unet model provided in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of spatial attention provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of a video background music generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0054] Example 1

[0055] like Figure 1 As shown, the technology of the present invention implements a method for generating video background music, including:

[0056] The video to be processed is decomposed into a target video image frame set and a target video original sound sequence.

[0057] A video contains a series of image frames and video soundtracks. The image frames exist in the time domain of the entire video, while the video soundtrack may or may not exist. In view of this situation, when extracting target condition features later, different methods are used to extract target condition features according to different situations of the video.

[0058] A target text description sequence is extracted from a target video image frame set by extracting a text description sequence of a video image frame set.

[0059] In the specific implementation process, Figure 2As shown, the process of the method for extracting a text description sequence of a video image frame set includes: splitting the video image frame set into a number of continuous and non-overlapping groups with a set number of adjacent image frames as a group; processing the image frames of each group through a Video2Text model or a Picture2Text model to obtain a corresponding text description; the text descriptions of different groups form a text description sequence arranged in time sequence. An exemplary Video2Text model or Picture2Text model adopts a BLIP model. The BLIP (Bootstrapping Language-Image Pre-training) model is a multimodal pre-training model that models the correspondence between images and texts. BLIP adopts an encoder-decoder architecture, which includes three main parts: an image encoder, a text encoder, and a decoder, and is used to generate high-quality image-text pair data. The present application uses a BLIP model to generate corresponding text descriptions for video image frames.

[0060] A target subtitle sequence is extracted from a target video original sound sequence by extracting a subtitle sequence corresponding to speech in a video original sound sequence.

[0061] During the specific implementation process, the method for extracting the subtitle sequence corresponding to the speech in the original sound sequence of the video includes: obtaining the speech segment in the original sound sequence of the target video, converting the speech segment into text through speech-to-text conversion, aligning the time axis of the text and the speech segment, and obtaining the subtitle sequence.

[0062] In the present application, the text description in the target text description sequence is aligned with the corresponding video image frame in the target video image frame set, and the target subtitle sequence is aligned with the corresponding speech in the target video original sound sequence; the alignment ensures that the target text description conditional features and target subtitle conditional features subsequently obtained based on the target text description sequence and the target subtitle sequence can be aligned with the video.

[0063] The pre-trained conditional feature extraction module extracts temporal target video image frame conditional features, target video original sound conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, target video original sound sequence, target text description sequence and target subtitle sequence respectively.

[0064] In the specific implementation process, the conditional feature extraction module includes:

[0065] Text description conditional encoder and subtitle conditional encoder based on BERT model: the text description conditional encoder is used to extract target text description conditional features from the text description sequence of the video to be processed, and the subtitle conditional encoder is used to extract target subtitle conditional features from the subtitle sequence of the video to be processed.

[0066] A video conditional encoder based on visual Transformer is used to extract target video image frame conditional features from image frames in a target video image frame set; the process of extracting target video image frame conditional features by the video conditional encoder includes: first, splitting the video image frame set into a plurality of continuous and non-overlapping groups with a set number of adjacent image frames as a group, uniformly sampling from each group of image frames, and extracting target video image frame conditional features from the sampled image frames through the video conditional encoder.

[0067] An audio conditional encoder based on a wav2vec model is used to extract target video original sound conditional features from a target video original sound sequence.

[0068] According to the needs, users select the combination of various target condition features for each stage of the video to be processed and configure the weights of various target condition features through the background music condition configurator; based on the combination of target condition features and the configuration of the weights of various target condition features, conditional feature fusion is performed to finally obtain the target condition features for each stage of the video that meet the user's needs.

[0069] In the specific implementation process, the background music condition configurator is expressed as:

[0070] ,

[0071] in, is the target video image frame condition feature, is the original sound condition feature of the target video, Describe conditional features for the target text, is the target subtitle condition feature, N is the number of iterations in the reverse process, and n is the current iteration number; , , and User-configurable weight parameters; target condition characteristics Contains different forms of sub-category target condition features:

[0072] For the video part with only image frames but no original sound The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature and the target text description condition feature ;

[0073] For video parts that have image frames and original sound, but the original sound does not contain voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature and the target text description condition feature. ;

[0074] For video parts with image frames and original sound, the original sound contains voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature, the target subtitle condition feature and the target text description condition feature. .

[0075] Then, corresponding to all the iterations of the reverse process, N target condition features will be obtained: in the early stage of the reverse process, The associated conditional features mainly participate in the reverse process. In the later stage of the reverse process, The associated conditional features mainly participate in the reverse process. At different stages of the reverse process, different conditional features are controlled and integrated into the latent space features to control the evolution of the latent space features in the reverse denoising process and ensure that the latent space features obtained in the reverse process are consistent with the image and sound conditions of the video.

[0076] Through the background music condition configurator, conditional features are provided for each iteration step of the reverse process of the music generation diffusion model according to the situation that the video contains image frames and original sound.

[0077] Of course, the above example of background music condition configurator only gives one example. The associated conditional features and Examples of associated conditional features may also include other combinations. For example, in the case where there is an original sound but the original sound does not contain speech, the following association method is applied: . The coefficient used to control the integration of conditional features and It can also be evolved according to needs, such as introducing a delay coefficient when the background sound needs to have a certain delay relative to any conditional characteristics. , , so that the conditional features that originally correspond to the nth step of the reverse process are used in the first step of the reverse process During the step, a certain background music delay effect is produced.

[0078] like Figure 3 As shown, the pre-trained music generation diffusion model is used to perform forward Gaussian diffusion on the latent space features of the vector representation of the reference music. In the reverse process, the music generation diffusion model introduces the target conditional features into the latent space features of each reverse step through the pre-trained Unet model to obtain the latent space features of the background music, and decodes the latent space features of the background music to obtain the vector representation of the background music; the vector representation of the background music is restored to the background music that conforms to the content of the video to be processed.

[0079] In a specific implementation process, the music generation diffusion model includes: a vector representation encoder for encoding the vector representation of the reference music into the latent space features of the vector representation of the reference music; the vector representation encoder is implemented based on a convolutional neural network. The vector representation of music is as follows Figure 4 As shown, the MIDI music file is represented in the form of a Piano roll, where the horizontal axis of the Piano roll represents the time step and the vertical axis represents the pitch; in this application, the horizontal axis length of the Piano roll of the MIDI music file is 32 beats, and the vertical axis height is 128, representing 128 pitches. The Piano roll is one-hot encoded to obtain a vector representation. The corresponding process of restoring the music using the vector representation is the process of restoring the Piano roll to a MIDI music file.

[0080] Forward diffusion process: vector represents the latent space features of the encoder output After the forward diffusion process is processed, the forward diffusion process is iterated for N steps to transfer the probability distribution according to the Gaussian distribution, so that the latent space feature is diffused step by step, and each iteration step is expressed as:

[0081] ;

[0082] in, In the forward diffusion process The latent space features of the step, In the forward diffusion process The latent space features of the step, In the forward diffusion process, Transfer to The probability distribution of is a Gaussian distribution, In the forward diffusion process, Transfer to The adjustment coefficient of Gaussian distribution in the probability distribution transfer process, I is the covariance matrix; each iteration step obtains the corresponding latent space feature , N latent space features are obtained through N steps of iteration ; The forward diffusion process means adding Gaussian noise to the latent space features of the reference music through a Markov chain.

[0083] Reverse process: Each iteration of the reverse process uses the Unet model to model the probability distribution of the reverse transfer process to convert the final latent space features obtained by the forward diffusion process into Restore Or will Convert to the conditional features , the probability distribution transfer of each iteration step of the reverse process is expressed as:

[0084] ;

[0085] in, Indicates the reverse process from Transfer to The probability distribution of Represents the reverse process parameterized by the Unet model The mean of Represents the reverse process parameterized by the Unet model The variance of

[0086] A vector representation decoder is used to decode the final latent space features of the reverse process into a vector representation. The vector representation decoder is implemented based on a convolutional neural network.

[0087] like Figure 5 As shown, the Unet model includes: multiple levels of encoders, an intermediate layer connecting the encoder and the decoder and a decoder of the corresponding level, the encoder in the Unet model includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to downsampling, the decoder in the Unet model also includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to upsampling; in the Unet model, the encoder output of the previous level and the output of the decoder of the same level are combined and input to the encoder of the next level; the intermediate layer is a residual block, spatial attention and residual block structure; the spatial attention uses the latent space feature after self-attention as a query, and uses the target conditional feature as the key and value for cross-attention.

[0088] like Figure 6 As shown, the spatial attention includes the self-attention of the residual structure, the cross-attention of the residual structure and the feedforward fully connected layer of the residual structure; after the latent space feature is self-attentioned, the output of the self-attention is used as the query, and the target condition feature is used as the key and value for cross-attention. The target condition feature is integrated into the latent space feature through cross-attention. In the entire reverse process, the latent space feature gradually integrates each target condition feature to ensure that the latent space feature in the reverse process is always guided by the target condition feature of the video, thereby generating the latent space feature related to the video. Thus, high-quality background music is generated based on the latent space feature related to the video.

[0089] In order to train the music generation diffusion model, the present application constructs a data set including: a video image frame set, a video soundtrack sequence and corresponding background music, wherein the video soundtrack sequence can be empty; for any form of data in the data set, its video image frame set and background music are time-aligned.

[0090] In the data set, the background music data is represented by a MIDI music file, which contains the music information of the three tracks of the main melody, secondary melody and accompaniment instrument of the background music. The music information includes the note start time, duration, pitch, intensity and instrument number. For background music, annotations are made by tonality, beat, chord, style and emotion. Tonality is the basic timbre or tone in music, such as major and minor. Beat is a regular and repetitive description of the length of notes; beat and rhythm information already contained in the MIDI music file determine the speed and rhythm of the music. Chord is a combination of notes composed of multiple notes, which will produce a musical harmony effect when played simultaneously; when annotating chords, the start time and end time of the chord and the chord name are determined. Style is the representative and unique appearance of the background music as a whole. Emotion is the emotion expressed by each part of the background music.

[0091] In the data set, when the video soundtrack sequence contains speech, the corresponding subtitle sequence is obtained through the subtitle tool, and the subtitle sequence is aligned with the corresponding speech part in the video soundtrack sequence. The above process annotates the video soundtrack sequence through the subtitle sequence. In the data set, the video image frame set is split into several non-overlapping groups with a set number of adjacent image frames as a group, and the image frames of each group are processed by the Video2Text model to obtain the corresponding text description; the text descriptions of different groups form a text description sequence arranged in time sequence. The above process annotates the video image frame set through the text description arranged in time sequence.

[0092] The loss function of the training process of the music generation diffusion model includes: in the training dataset domain, in each step of the forward diffusion process, the probability distribution transfer is performed according to the Gaussian distribution to introduce real noise, and in each step of the reverse process, the Unet model predicts the expectation of the L2 distance between the noise based on the given target feature conditions and the given initial latent space features; the music generation diffusion model is trained with the goal of minimizing the loss function. In the training dataset domain, the similarity of the features between the generated background music and the actual background music is used to evaluate the quality of the music generation diffusion model, and the features include: structural index, pitch classification histogram entropy and chanting mode similarity.

[0093] like Figure 3 As shown, given the initial latent space features Based on , the forward diffusion process of the music generation diffusion model is: , real noise is introduced in the process in the manner of Markov chain In the reverse process, under the guidance of the target-free conditional features, the noise is removed in a Markov chain manner. , the prediction noise is gradually removed in the Markov chain manner; in order to correspond the real noise to the prediction noise, the order of the prediction noise in the reverse process is reversed: In the reverse process, under the guidance of the target condition features, the noise is removed in a Markov chain manner. , in the process, the prediction noise is gradually removed in a Markov chain manner; Determined by the Gaussian distribution involved in the forward process and the background music given in the training dataset, is the target condition feature. Due to the randomness of the diffusion process of the music generation diffusion model, the generated background music has diversity.

[0094] Example 2

[0095] See also Figure 7 As shown, an embodiment of the present invention provides a video background music generation device, including: at least one processing unit, the processing unit is connected to a storage unit via a bus unit, the storage unit is a computer-readable storage medium, and can be used to store software programs, computer executable programs and modules, such as the software programs, computer executable programs and modules corresponding to a video background music generation method in an embodiment of the present invention. The processing unit implements the above-mentioned video background music generation method by running the software programs, computer executable programs and modules stored in the storage unit, including:

[0096] Decomposing the video to be processed into a target video image frame set and a target video original sound sequence;

[0097] Extract a target text description sequence from a target video image frame set by extracting a text description sequence from a video image frame set; extract a target subtitle sequence from a target video original sound sequence by extracting a subtitle sequence corresponding to speech in a video original sound sequence, align the text description in the target text description sequence with the corresponding video image frame in the target video image frame set, and align the target subtitle sequence with the corresponding speech in the target video original sound sequence;

[0098] Extracting temporal target video image frame conditional features, target video original sound conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, target video original sound sequence, target text description sequence and target subtitle sequence respectively through a pre-trained conditional feature extraction module;

[0099] The user selects the combination of various target condition features at each stage of the video to be processed and configures the weights of various target condition features through the background music condition configurator according to the needs; based on the combination of target condition features and the weights of various target condition features, condition feature fusion is performed to finally obtain the target condition features at each stage of the video that meet the user's needs;

[0100] The pre-trained music generation diffusion model is used to perform forward Gaussian diffusion on the latent space features of the vector representation of the reference music. In the reverse process, the music generation diffusion model introduces the conditional features into the latent space features of each reverse process through the pre-trained Unet model to obtain the latent space features of the background music, and decodes the latent space features of the background music to obtain the vector representation of the background music; the vector representation of the background music is restored to the background music that conforms to the content of the video to be processed.

[0101] Of course, the computer program stored in the storage unit of the video background music generating device provided by the embodiment of the present invention is not limited to the operation of the method described above, and can also execute the related operations in the video background music generating method provided by any embodiment of the present invention.

[0102] Example 3

[0103] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the method for generating background music for a video is implemented, including:

[0104] Decomposing the video to be processed into a target video image frame set and a target video original sound sequence;

[0105] Extract a target text description sequence from a target video image frame set by extracting a text description sequence from a video image frame set; extract a target subtitle sequence from a target video original sound sequence by extracting a subtitle sequence corresponding to speech in a video original sound sequence, align the text description in the target text description sequence with the corresponding video image frame in the target video image frame set, and align the target subtitle sequence with the corresponding speech in the target video original sound sequence;

[0106] Extracting temporal target video image frame conditional features, target video original sound conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, target video original sound sequence, target text description sequence and target subtitle sequence respectively through a pre-trained conditional feature extraction module;

[0107] The user selects the combination of various target condition features at each stage of the video to be processed and configures the weights of various target condition features through the background music condition configurator according to the needs; based on the combination of target condition features and the weights of various target condition features, condition feature fusion is performed to finally obtain the target condition features at each stage of the video that meet the user's needs;

[0108] The pre-trained music generation diffusion model is used to perform forward Gaussian diffusion on the latent space features of the vector representation of the reference music. In the reverse process, the music generation diffusion model introduces the conditional features into the latent space features of each reverse process through the pre-trained Unet model to obtain the latent space features of the background music, and decodes the latent space features of the background music to obtain the vector representation of the background music; the vector representation of the background music is restored to the background music that conforms to the content of the video to be processed.

[0109] A computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in a method for generating video background music provided by any embodiment of the present invention.

[0110] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for generating background music for a video, characterized in that: include: Decomposing the video to be processed into a target video image frame set and a target video original sound sequence; Extract a target text description sequence from a target video image frame set by extracting a text description sequence from a video image frame set; extract a target subtitle sequence from a target video original sound sequence by extracting a subtitle sequence corresponding to speech in a video original sound sequence, align the text description in the target text description sequence with the corresponding video image frame in the target video image frame set, and align the target subtitle sequence with the corresponding speech in the target video original sound sequence; Extracting temporal target video image frame conditional features, target video original sound conditional features, target text description conditional features and target subtitle conditional features from the target video image frame set, target video original sound sequence, target text description sequence and target subtitle sequence respectively through a pre-trained conditional feature extraction module; The user selects the combination of various target condition features at each stage of the video to be processed and configures the weights of various target condition features through the background music condition configurator according to the needs; Based on the combination of target condition features and the configuration of weights of various target condition features, condition feature fusion is performed to finally obtain target condition features of each stage of the video that meet the user's needs; wherein the background music condition configurator is expressed as: ， in, is the target video image frame condition feature, is the original sound condition feature of the target video, Describe conditional features for the target text, is the target subtitle condition feature, N is the number of iterations in the reverse process, and n is the current iteration number; , , and It is a user-configurable weight parameter; For the video part with only image frames but no original sound The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature and the target text description condition feature ; For video parts that have image frames and original sound, but the original sound does not contain voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature and the target text description condition feature. ; For video parts with image frames and original sound, the original sound contains voice The background music condition configurator obtains the subclass target condition feature corresponding to the current iteration number n by combining the target video image frame condition feature, the target video original sound condition feature, the target subtitle condition feature and the target text description condition feature. ; The pre-trained music generation diffusion model is used to perform forward Gaussian diffusion on the latent space features of the vector representation of the reference music. In the reverse process, the music generation diffusion model introduces the target conditional features into the latent space features of each reverse process through the pre-trained Unet model to obtain the latent space features of the background music, and decodes the latent space features of the background music to obtain the vector representation of the background music; wherein, corresponding to all the iterations of the reverse process, there will be N target conditional features: in the early stage of the reverse process, The associated conditional features mainly participate in the reverse process. In the later stage of the reverse process, The conditional features of the association mainly participate in the reverse process; The vector representation of the background music is restored to the background music that matches the content of the video to be processed.

2. The method for generating video background music according to claim 1, characterized in that: The process of the method for extracting a text description sequence of a video image frame set includes: taking a set number of adjacent image frames as a group, splitting the video image frame set into a plurality of continuous and non-overlapping groups; processing the image frames of each group through a Video2Text model or a Picture2Text model to obtain a corresponding text description; and the text descriptions of different groups form a text description sequence arranged in time sequence.

3. The method for generating video background music according to claim 1, characterized in that: The method for extracting a subtitle sequence corresponding to speech in a video original sound sequence includes: obtaining a speech segment in a target video original sound sequence, converting the speech segment into text through speech-to-text conversion, aligning the text with the speech segment on a time axis, and obtaining a subtitle sequence.

4. The method for generating video background music according to claim 1, characterized in that: The conditional feature extraction module includes: a text description conditional encoder and a subtitle conditional encoder based on the BERT model: the text description conditional encoder is used to extract target text description conditional features from the text description sequence of the video to be processed, and the subtitle conditional encoder is used to extract target subtitle conditional features from the subtitle sequence of the video to be processed; A video conditional encoder based on a visual Transformer, wherein the video conditional encoder is used to extract target video image frame conditional features from image frames in a target video image frame set; An audio conditional encoder based on a wav2vec model is used to extract target video original sound conditional features from a target video original sound sequence.

5. The method for generating video background music according to claim 1, characterized in that: The music generation diffusion model includes: a vector representation encoder for encoding the vector representation of the reference music into latent space features of the vector representation of the reference music; Forward diffusion process: vector represents the latent space features of the encoder output After the forward diffusion process is processed, the forward diffusion process is iterated for N steps to transfer the probability distribution according to the Gaussian distribution, so that the latent space feature is diffused step by step, and each iteration step is expressed as: ； in, In the forward diffusion process The latent space features of the step, In the forward diffusion process The latent space features of the step, In the forward diffusion process, Transfer to The probability distribution of is a Gaussian distribution, In the forward diffusion process, Transfer to The adjustment coefficient of Gaussian distribution in the probability distribution transfer process, I is the covariance matrix; each iteration step obtains the corresponding latent space feature , N latent space features are obtained through N steps of iteration ; Reverse process: Each iteration of the reverse process uses the Unet model to model the probability distribution of the reverse transfer process to convert the final latent space features obtained by the forward diffusion process into Restore Or will Convert to the conditional features , the probability distribution transfer of each iteration step of the reverse process is expressed as: ； in, Indicates the reverse process from Transfer to The probability distribution of Represents the reverse process parameterized by the Unet model The mean of Represents the reverse process parameterized by the Unet model The variance of A vector representation decoder is used to decode the final latent space features of the backward process into a vector representation.

6. The method for generating video background music according to claim 5, characterized in that: The vector representation encoder and the vector representation decoder are implemented based on convolutional neural networks.

7. The method for generating video background music according to claim 1, characterized in that: The Unet model includes: multiple layers of encoders, an intermediate layer connecting the encoder and the decoder and a decoder of the corresponding layer, the encoder in the Unet model includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to downsampling, the decoder in the Unet model also includes stacked residual blocks and spatial attention, the stacked residual blocks and the spatial attention are connected to upsampling; in the Unet model, the output of the encoder of the previous layer and the output of the decoder of the same layer are combined and input to the encoder of the next layer; the intermediate layer is a residual block, spatial attention and residual block structure; The spatial attention includes the self-attention of the residual structure, the cross-attention of the residual structure and the feedforward fully connected layer of the residual structure; after the latent space feature is self-attentioned, the output of the self-attention is used as the query, and the self-conditional features of the corresponding reverse step in the target conditional features are used as the key and value for cross-attention.

8. The method for generating video background music according to claim 1, characterized in that: The loss function of the training process of the music generation diffusion model includes: in the training dataset domain, in the forward diffusion process, each step introduces real noise by probability distribution transfer according to Gaussian distribution, and in the reverse process, each step Unet model predicts the expectation of the L2 distance between the noise based on the given target feature conditions and the given initial latent space features; The music generation diffusion model is trained with the goal of minimizing the loss function.

9. The method for generating video background music according to claim 1, characterized in that: In the training dataset domain, the similarity of features between the generated background music and the actual background music is used to evaluate the quality of the music generation diffusion model, wherein the features include: structural index, pitch classification histogram entropy and singing pattern similarity.

10. A video background music generating device, characterized in that: include: At least one processing unit, the processing unit is connected to a storage unit via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the method for generating video background music as described in any of claims 1-9 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating video background music as described in any one of claims 1 to 9 is implemented.

Images