Music matching method and device for long-term rhythmic video

Abstract

The present invention relates to the field of video processing technology and proposes a method and device for composing music for long-duration rhythmic videos. The method comprises: extracting audio features through an implicit unconditional generative model; extracting video features from the long-duration rhythmic video through a context-aware conditional encoder; and interactively fusing the audio and video features through an implicit conditional diffusion generative model using a cross-modal attention module to generate a soundtrack. The music generated by the present invention has better rhythmic matching and musical quality; the length of the generated music is extended from 2-6 seconds in the prior art to 25-50 seconds; and it can be applied to a wider range of rhythmic video scenarios, expanding the scope from simple dance scenes to sports scenes.

Description

Technical Field

[0001] The present invention generally relates to the field of video processing technology. Specifically, the present invention relates to a method and device for composing music for a long-term rhythmic video. Background Art

[0002] The purpose of the video soundtrack task is to enable the model to generate music that matches the given video, and to make the music as melodious and rhythmic as possible (that is, to make the video match the rhythm of the music).

[0003] For video soundtrack tasks, existing technologies include video soundtrack methods based on music representation and video soundtrack methods based on waveform graphs. The video soundtrack method based on music representation uses a specific music representation form (such as Musical Instrument Digital Interface (MIDI) and Remote Integration Model (REMI)) to convert the music waveform graph into a corresponding music representation, and models the music representation to achieve controllable music generation. The video soundtrack method based on waveform graphs does not rely on specific music representations, but directly uses a generative model to model the waveform graph to achieve a more flexible music generation effect.

[0004] However, in video soundtracking methods based on music representation, the highly stylized nature of the music representation results in limited flexibility and diversity in the generated music. While more flexible, waveform-based methods still suffer from the following issues: the generated audio is short, currently limited to 2 to 6 seconds; the video types are limited to dance; and the quality of the generated music is poor, making it difficult to achieve stable results. Summary of the Invention

[0005] To at least partially solve the above-mentioned problems in the prior art, the present invention provides a method for composing music for a long-term rhythmic video, comprising the following steps:

[0006] Extracting audio features through implicit unconditional generative models;

[0007] Extracting video features from long-form rhythmic videos via a context-aware conditional encoder; and

[0008] The audio features and the video features are interactively fused using a cross-modal attention module through an implicit conditional diffusion generative model to generate a soundtrack.

[0009] In one embodiment of the present invention, extracting audio features through an implicit unconditional generative model includes:

[0010] An unconditional generative model pre-trained on a self-collected audio dataset is used as the backbone network, in which an implicit diffusion model is used to model the audio waveform. The implicit diffusion model learns the probability distribution of the target data through a T-step denoising process.

[0011] In one embodiment of the present invention, modeling the audio waveform using an implicit diffusion model includes:

[0012] An implicit diffusion model is used to encode the input audio waveform into an implicit variable and optimized using a loss function, where the loss function is expressed as follows:

[0013]

[0014] Among them, L LD represents the implicit diffusion loss function, D θ represents the denoising neural network, λ represents an optional weighting function to control the size of the loss function value, z represents the implicit distribution of the audio, P data represents the actual distribution of audio data, t represents the number of denoising steps, and T is the total denoising step length).

[0015] In one embodiment of the present invention, the video features include RGB features, rhythm features of the video, and category features of the video, wherein extracting the video features from the long-term rhythmic video by the context-aware conditional encoder includes:

[0016] The video features are temporally modeled using a context-aware model to capture content changes and rhythm changes in long-duration rhythmic videos.

[0017] In one embodiment of the present invention, extracting RGB features includes:

[0018] Use the pre-trained I3D network for feature extraction and a two-layer LSTM network for modeling, which is expressed as follows:

[0019] c v ,(h,mc)=BiLSTM(Wnc(i1,i2,...,i T ),h0,mc0),

[0020] Among them, Enc represents the visual encoder, i represents the input visual frame, BiLSTM represents the bidirectional long short-term memory network, h represents the hidden vector, mc represents the memory vector, and c v Represents visual condition characteristics;

[0021] Extracting rhythm features of videos includes:

[0022] Extract 2D posture features from videos;

[0023] Extract the video rhythm, expressed as follows:

[0024]

[0025]

[0026]

[0027] Among them, M(t, j) represents the motion feature of the jth joint at the tth moment, ∠M(t, j) represents the angle between the motion feature and the x-axis, 1 θ is an indicator function that is used to classify visual features into different bins according to the angle, K is the number of bins, 0(t) is the visual rhythm at the t-th moment, and η represents the normalization function that normalizes the visual rhythm to the range of 0-1; and

[0028] A rhythm curve is generated according to the video rhythm, and a local maximum of the rhythm curve is selected to generate a binary vector, wherein the binary vector indicates whether each temporal position is a visual rhythm point, wherein the condition for selecting the local maximum of the rhythm curve is expressed as follows:

[0029] c r (t i )=max(O[t i -pre m :t+post m |),

[0030] c r (t i )≥mean(O|t i -pre a :t+post a |+δ),

[0031] t i -t i-1 >ω

[0032] Among them, c r (t i ) represents the rhythm condition feature of the moment where the i-th visual rhythm point is located, pre m , post m , pre a , post a Represent the thresholds before and after taking the local mean and maximum respectively, ω represents the minimum interval between the temporal positions of two visual rhythm points, and δ is a threshold. Only when the local maximum minus the local mean at the current moment is greater than this value can it be considered as a visual rhythm point; and

[0033] The category features of the extracted video include:

[0034] Perform global feature modeling, where the category label of the video is mapped to a high-dimensional global feature through a learnable matrix, which is expressed as follows:

[0035] c g =Embcd(G).

[0036] Among them, c g Represents the category conditional feature. Embed is a learnable dimensionality-raising layer that converts a single category label into a high-dimensional feature vector. G is the category label of the video.

[0037] In one embodiment of the present invention, the method for composing music for a long-term rhythmic video further includes converting the rhythm feature from a binary vector into a high-dimensional rhythm feature tensor, including:

[0038] Using a learnable matrix for feature mapping, which adds additional positional features; and

[0039] The Hawkes process is introduced to model the position of the binary vector while adding a bias based on the proximity information of the rhythm point, which can be expressed as follows:

[0040] Hawkws(t i )=sin(w×i+w×t i ),

[0041]

[0042] Among them, c r is the original rhythmic conditional characteristic, is the rhythm conditional feature after bias, W r is a learnable parameter, Hawkes represents the Hawkes process function, and w is a bias hyperparameter.

[0043] In one embodiment of the present invention, the interactive fusion of the audio features and the video features using a cross-modal attention module through an implicit conditional diffusion generative model includes:

[0044] Conditional generation is performed using a cross-modal attention module based on the rhythm feature tensor, as shown in the following formula:

[0045]

[0046] Among them, Att() represents the attention function, d represents the dimension of the feature, and W Q , W K , W Vis a learnable vector, c represents the conditional feature, T represents the matrix transpose, and i represents the i-th intermediate layer, because the attention module participates in several intermediate layers; and

[0047] Add conditional features to optimize the diffusion model, which can be expressed as follows:

[0048]

[0049] Among them, L CLD represents the conditional implicit diffusion function, and C represents the conditional feature.

[0050] The present invention also provides a music arrangement device for long-term rhythmic video, comprising:

[0051] an audio feature extraction module configured to extract audio features through an implicit unconditional generative model;

[0052] a video feature extraction module configured to extract video features from the long-term rhythmic video through a context-aware conditional encoder; and

[0053] A soundtrack generation module is configured to interactively fuse the audio features and the video features using a cross-modal attention module through an implicit conditional diffusion generation model to generate a soundtrack.

[0054] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method when executed by a processor.

[0055] The present invention further provides a computer system comprising:

[0056] a processor configured to execute machine-executable instructions; and

[0057] A memory having machine-executable instructions stored thereon, which, when executed by a processor, perform the steps according to the method.

[0058] The present invention has at least the following technical effects: the music generated by the present invention has better rhythm matching and music quality; the length of the generated music is expanded from 2-6 seconds in the prior art to 25-50 seconds; it can be applied to more rhythmic video scenes, expanding the scenes from simple dance to sports scenes (such as floor gymnastics and figure skating). Its principle is to perform temporal modeling through context-aware conditional encoder video to capture the context information of each frame of the video in order to generate long sequences of music; and through the implicit conditional diffusion generation model, the cross-modal attention module is used to interactively fuse the audio features and the video features, thereby performing cross-modal conditional generation, which greatly improves the generation quality, even if the generated soundtrack has a better sense of rhythm and is more matched with the background video. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] To further illustrate the advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings depict only typical embodiments of the present invention and are not to be considered as limiting the scope of the present invention. In the drawings, for clarity, identical or corresponding components will be represented by the same or similar reference numerals.

[0060] Figure 1 A computer system is shown that implements the system and / or method according to the present invention.

[0061] Figure 2 The figure shows a flow chart of a method for composing music for a long-term rhythmic video in one embodiment of the present invention.

[0062] Figure 3 A schematic diagram of a framework of a method for composing music for a long-term rhythmic video in one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] It should be noted that the components in the drawings may be shown exaggeratedly for the sake of illustration and are not necessarily correct to scale. In the drawings, identical or functionally identical components are provided with the same reference numerals.

[0064] In the present invention, unless otherwise specified, the phrases "disposed on," "disposed above," and "disposed above" do not exclude the presence of intermediate components. Furthermore, "disposed on or above" merely indicates the relative positional relationship between two components and, in certain circumstances, such as after reversing the product orientation, can be converted to "disposed below or below," and vice versa.

[0065] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0066] In the present invention, unless otherwise specified, the quantifiers "a" and "an" do not exclude the presence of multiple elements.

[0067] It should also be noted that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added according to the needs of the specific scenario. In addition, unless otherwise stated, the features of different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can be used to replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the scope of disclosure or description of this application.

[0068] It should also be noted that, within the scope of the present invention, terms such as "same," "equal," and "equal to" do not imply absolute equality of values, but rather allow for a certain reasonable error. In other words, such terms also encompass "substantially the same," "substantially equal," and "substantially equal." Similarly, in the present invention, terms such as "perpendicular to" and "parallel to" indicating direction also encompass the meaning of "substantially perpendicular to" and "substantially parallel to."

[0069] In addition, the numbering of the steps of the methods of the present invention does not limit the order in which the steps are to be performed. Unless otherwise specified, the steps of the methods may be performed in different orders.

[0070] The present invention will be further described below with reference to the accompanying drawings in conjunction with specific embodiments.

[0071] Figure 1 The computer system 100 for implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention may be Figure 1 The present invention may be implemented in the computer system 100 shown, or the present invention may be implemented in a distributed manner in multiple computer systems 100 according to the present invention via a network, such as a local area network or the Internet. The computer system 100 of the present invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multi-processor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, network servers, tablet computers, etc.

[0072] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard drive interface 109, an optical drive interface 113, a network interface 114, and a universal serial bus (USB) interface 112. The system bus 101 can be any of several types of bus structures, such as a memory bus or memory controller, a peripheral bus, and a local bus using various types of bus architectures. The system bus 101 is used for communication between various bus devices. In addition to Figure 1In addition to the bus devices or interfaces shown in , other bus devices or interfaces are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 can, for example, store basic input / output system (BIOS) data for implementing basic routines for information transmission at startup, and RAM 104 is used to provide the system with a running memory with faster access speed. Computer system 100 also includes a hard disk drive 109 for reading and writing to hard disk 110, an optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 can, for example, store an operating system and application programs. The drive and its associated computer-readable medium provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer system 100. Computer system 100 can also include a video adapter 105 for image processing and / or image output, which is used to connect to output devices such as display 106. The computer system 100 may further include an audio adapter 107 for audio processing and / or audio output, which is used to connect to output devices such as speakers 108. In addition, the computer system 100 may further include a network interface 114 for network connection, wherein the network interface 114 may be connected to the Internet 116 via a network device such as a router 115, wherein the connection may be wired or wireless. In addition, the computer system 100 may further include a universal serial bus (USB) interface 112 for connecting to peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as a microphone, a camera, etc.

[0073] When the present invention Figure 1 When implemented on the computer system 100, the generated music can have better rhythm matching and music quality; the length of the generated music can be expanded from 2-6 seconds in the existing technology to 25-50 seconds; it can be applied to more rhythmic video scenes, expanding the scenes from simple dance to sports scenes (such as floor gymnastics and figure skating).

[0074] Furthermore, the various embodiments may be provided as a computer program product that may include one or more machine-readable media having machine-executable instructions stored thereon, which, when executed by one or more machines, such as computers, computer networks, or other electronic devices, may cause the one or more machines to perform operations according to the various embodiments of the present invention. The machine-readable medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (Compact Disk Read Only Memory) and a magneto-optical disk, a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic or optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0075] Furthermore, various embodiments may be downloaded as a computer program product, wherein the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Thus, a machine-readable medium as used herein may, but is not required to, include such a carrier wave.

[0076] Figure 2 FIG. 1 is a flow chart showing a method for composing music for a long-term rhythmic video according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0077] Step 201: Extract audio features through an implicit unconditional generative model.

[0078] Step 202: Extract video features from the long-term rhythmic video using a context-aware conditional encoder.

[0079] Step 203: Using a cross-modal attention module through an implicit conditional diffusion generative model, the audio features and the video features are interactively fused to generate a soundtrack.

[0080] For long-term rhythmic videos, the present invention uses a method based on an implicit conditional diffusion model to model the input video. The method includes a context-aware conditional encoder and an implicit conditional diffusion generation model.

[0081] In the context-aware conditional encoder, RGB features, video rhythm features, and video category features are used as conditional features, and a series of context-aware models are used for temporal modeling to capture content changes and rhythm changes in long-term rhythmic videos.

[0082] In the implicit conditional diffusion generation model, a cross-modal attention module is used to interact and fuse the features generated by the conditional encoder with the audio features, thereby performing cross-modal conditional generation.

[0083] In the present invention, each module of the system according to the present invention can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented by a computer program flow, for example, the module can be implemented by a code segment (such as a code segment in a language such as C, C++) stored in a storage device (such as a hard disk, memory, etc.), wherein when the code segment is executed by a processor, the corresponding function of the module can be implemented. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, by hardware programming a programmable device such as a field programmable gate array (FPGA) to implement the function of the module, or by designing an application-specific integrated circuit (ASIC) including electronic devices such as a plurality of transistors, resistors and capacitors to implement the function of the module. When the module is implemented using firmware, the function of the module can be written into a read-only memory such as an EPROM or EEPROM of the device in the form of program code, and when the program code is executed by the processor, the corresponding function of the module can be implemented. In addition, certain functions of the module may need to be implemented by separate hardware or through collaboration with the hardware, for example, the detection function is implemented by corresponding sensors (such as proximity sensors, accelerometers, gyroscopes, etc.), the signal transmission function is implemented by corresponding communication devices (such as Bluetooth devices, infrared communication devices, baseband communication devices, Wi-Fi communication devices, etc.), the output function is implemented by corresponding output devices (such as displays, speakers, etc.), and so on.

[0084] Figure 3 A schematic diagram of a framework of a method for composing music for a long-term rhythmic video according to an embodiment of the present invention is shown below. Figure 3 This method is described in detail.

[0085] Part 1: Unconditional implicit diffusion generation.

[0086] The audio waveform is modeled using an implicit diffusion model. This implicit diffusion model learns the probability distribution of the target data through a T-step denoising or noise-adding process, achieving stable and diverse data generation. The denoising process can be viewed as a Markov chain, while the denoising process is optimized using an L2 loss function.

[0087] Since there are many audio sampling points, an implicit diffusion model can be used to encode the input audio waveform into implicit variables and optimized using a loss function. The loss function can be expressed as follows:

[0088]

[0089] L LD represents the implicit diffusion loss function, D θ represents the denoising neural network, λ represents an optional weighting function to control the size of the loss function value, z represents the implicit distribution of the audio, p data represents the actual distribution of audio data, t represents the number of denoising steps, and T is the total denoising step length).

[0090] To improve the quality of data generation, an unconditional generative model pre-trained on a large self-collected audio dataset (Flavio, S., Ryan, ZT, and Eugenio, H.-B. archinetai / audio-diffusion-pytorch, 2022. https: / / github.com / archinetai / audio-diffusion-pytorch) can be used as the backbone network.

[0091] Part II, Conditional Encoder.

[0092] Existing techniques typically compress a given video into images, which reduces computational overhead but loses crucial temporal information, resulting in a very short audio output. To process long videos and generate long sequences of music, this paper employs temporal modeling to capture the contextual information of each frame. This involves the following steps:

[0093] For RGB (red, green, and blue color model) features, a pre-trained I3D (Inflated 3D) network is used for feature extraction, and a two-layer LSTM (Long Short-Term Memory) network is used for modeling, which is expressed as follows:

[0094] cv,(h,mc)=BiLSTM(Enc(i1,i2,...,i T ),h0,mc0),

[0095] Among them, Enc represents the visual encoder, i represents the input visual frame, BiLSTM represents the bidirectional long short-term memory network, h represents the hidden vector, mc represents the memory vector, and c v Represents visual condition characteristics;

[0096] For the rhythm feature of the video, first extract the 2D posture feature of the video and extract the video rhythm, which is expressed as follows:

[0097]

[0098]

[0099]

[0100] Among them, M(t, j) represents the motion feature of the jth joint at the tth moment, ∠M(t, j) represents the angle between the motion feature and the x-axis, 1 θ It is an indicator function that is used to classify visual features into different bins according to the angle. K is the number of bins, 0(t) is the visual rhythm at the t-th moment, and η represents the normalization function that normalizes the visual rhythm to the range of 0-1.

[0101] A rhythm curve is generated according to the extracted video rhythm, and the local maximum of the rhythm curve is selected to generate a binary vector. The binary vector indicates whether each temporal position is a visual rhythm point. The condition for selecting the local maximum of the rhythm curve is expressed as follows:

[0102] c r (t i )=max(Q|t i -pre m :t+post m |),

[0103] c r (t i )>meaan(Q|t i -pre a :t+post a |+δ),

[0104] t i -t i-1 >ω

[0105] Among them, c r (t i ) represents the rhythm condition feature of the moment where the i-th visual rhythm point is located, pre m , post m , pre a , post a Represent the front and back thresholds for taking the local mean and maximum values, ω represents the minimum interval between the temporal positions of two visual rhythm points, and δ is a threshold. Only when the local maximum value minus the local mean value at the current moment is greater than this value can it be considered as a visual rhythm point.

[0106] For the category features of the video, global feature modeling can be performed, where the category label of the video is mapped to a high-dimensional global feature through a learnable matrix, which is expressed as follows:

[0107] c g =Embed(G).

[0108] Among them, c g Represents the category conditional feature. Embed is a learnable dimensionality-raising layer that converts a single category label into a high-dimensional feature vector. G is the category label of the video.

[0109] Part III: Implicit conditional diffusion generation.

[0110] Conditional generation using a cross-modal attention module in implicit conditional diffusion generation.

[0111] Since the rhythm features of a video are binary vectors, they must first be converted into high-dimensional feature tensors. A learnable matrix can be used for feature mapping, which adds additional positional features. The Hawkes Process is introduced to model the position of the binary vector while adding a bias based on the proximity of the rhythm points, as shown in the following formula:

[0112] Hawkes(t i )=sin(w×i+w×t i ),

[0113]

[0114] Among them, c r is the original rhythmic conditional characteristic, is the rhythm conditional feature after bias, W r is a learnable parameter, Hawkes represents the Hawkes process function, and w is a bias hyperparameter.

[0115] Through the above process, the rhythm feature tensor of the video can be obtained. According to the rhythm feature tensor of the video, the cross-modal attention module can be used for conditional generation, which is expressed as follows:

[0116]

[0117] Among them, Att() represents the attention function, d represents the dimension of the feature, and W Q , W K , W V is a learnable vector, c represents the conditional feature, T represents the matrix transpose, and i represents the i-th intermediate layer, because the attention module participates in several intermediate layers;

[0118] Add conditional features to optimize the diffusion model, which can be expressed as follows:

[0119]

[0120] Among them, L CLDrepresents the conditional implicit diffusion function, and C represents the conditional feature.

[0121] In addition, similar generation effects can be obtained by using other generative models, such as a method based on a generative adversarial network, a method based on a variational autoencoder, or other methods based on a diffusion model, and using the conditional encoder and conditional generation method proposed in the present invention.

[0122] In one embodiment of the present invention, a device for composing music for a long-term rhythmic video is also provided, the device comprising:

[0123] an audio feature extraction module configured to extract audio features through an implicit unconditional generative model;

[0124] a video feature extraction module configured to extract video features from the long-term rhythmic video through a context-aware conditional encoder; and

[0125] A soundtrack generation module is configured to interactively fuse the audio features and the video features using a cross-modal attention module through an implicit conditional diffusion generation model to generate a soundtrack.

[0126] The performance of the present invention and the prior art on different video datasets are compared, and the results are shown in Tables 1-3, where Table 1 shows the performance of the present invention and the prior art on a 25s dance video dataset, Table 2 shows the performance of the present invention and the prior art on a 25s and 50s floor exercise video dataset, and Table 3 shows the performance of the present invention and the prior art on a 25s and 50s figure skating video dataset. In Tables 1-3, BCS represents the rhythm coverage, BHS represents the rhythm overlap, and F1 represents the comprehensive performance of BCS and BHS. The larger the values ​​of BCS, BHS, and F1, the better; CSD and HSD represent the standard deviation of coverage and overlap. CSD and HSD represent the stability of the generated results. The smaller the values ​​of CSD and HSD, the better. Through experimental comparison, it can be seen that the present invention has better rhythm matching and stronger stability than the prior art.

[0127] Table 1

[0128]

[0129] Table 2

[0130]

[0131] Table 3

[0132]

[0133] Although various embodiments of the present invention have been described above, it should be understood that they are presented by way of example only and not limitation. It will be apparent to those skilled in the relevant art that various combinations, modifications, and variations may be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely in accordance with the appended claims and their equivalents.

Claims

1. A method for composing music for a long-term rhythmic video, characterized in that: The following steps are involved: Extracting audio features through implicit unconditional generative models; Extracting video features from long-duration rhythmic videos via a context-aware conditional encoder; as well as The audio features and the video features are interactively fused to generate the soundtrack using a cross-modal attention module through an implicit conditional diffusion generative model. The video features include RGB features, video rhythm features and video category features. Extracting RGB features includes: Use the pre-trained I3D network for feature extraction and a two-layer LSTM network for modeling, which is expressed as follows: c v ,(h,mc)=BiLSTM(Enc(i1,i2,...,i T ),h0,mc0), Among them, Enc represents the visual encoder, i represents the input visual frame, BiLSTM represents the bidirectional long short-term memory network, h represents the hidden vector, mc represents the memory vector, and c v Represents visual condition characteristics; Extracting rhythm features of videos includes: Extract 2D posture features from videos; Extract the video rhythm, expressed as follows: Among them, M(t,j) represents the motion characteristics of the j-th joint at the t-th moment, ∠M(t,j) represents the angle between the motion characteristics and the x-axis, 1 θ represents the indicator function, K represents the number of bins, O(t) represents the visual rhythm within the t-th moment, and η represents the normalization function; and A rhythm curve is generated according to the video rhythm, and a local maximum of the rhythm curve is selected to generate a binary vector, wherein the binary vector indicates whether each temporal position is a visual rhythm point, wherein the condition for selecting the local maximum of the rhythm curve is expressed as follows: Among them, c r (t i ) represents the rhythm condition feature of the moment where the i-th visual rhythm point is located, pre m 、pos t m 、pre a 、post a Respectively represent the front and back thresholds for taking the local mean and maximum values, ω represents the minimum interval between the temporal positions of two visual rhythm points, and δ represents the threshold of the visual rhythm point; The music composition method further includes converting the rhythm feature from a binary vector into a high-dimensional rhythm feature tensor, and interactively fusing the audio feature and the video feature using a cross-modal attention module through an implicit conditional diffusion generation model, including: Conditional generation is performed using a cross-modal attention module based on the rhythm feature tensor, as shown in the following formula: Among them, Att() represents the attention function, d represents the dimension of the feature, and W Q 、W K 、W V represents a learnable vector, c represents the conditional feature, T represents the matrix transpose, and i represents the i-th intermediate layer; and Add conditional features to optimize the diffusion model, which can be expressed as follows: Among them, L CLD represents the conditional implicit spread function and C represents the conditional feature.

2. The method for composing music for a long-term rhythmic video according to claim 1, wherein: Audio features extracted through implicit unconditional generative models include: An unconditional generative model pre-trained on a self-collected audio dataset is used as the backbone network, in which an implicit diffusion model is used to model the audio waveform. The implicit diffusion model learns the probability distribution of the target data through a T-step denoising process or a denoising process.

3. The method for composing music for a long-term rhythmic video according to claim 2, wherein: Modeling the audio waveform using an implicit diffusion model involves: An implicit diffusion model is used to encode the input audio waveform into an implicit variable and optimized using a loss function, where the loss function is expressed as follows: Among them, L LD represents the implicit diffusion loss function, D θ represents the denoising neural network, λ represents an optional weighting function, z represents the implicit distribution of the audio, and p data represents the actual distribution of the audio data, t represents the number of denoising steps, and T represents the total denoising step size.

4. The method for composing music for a long-term rhythmic video according to claim 1, wherein: The video features extracted from long-term rhythmic videos by the context-aware conditional encoder include: The video features are temporally modeled using a context-aware model to capture content changes and rhythm changes in long-term rhythmic videos.

5. The method for composing music for a long-term rhythmic video according to claim 4, wherein: The category features of the extracted video include: Perform global feature modeling, where the category label of the video is mapped to a high-dimensional global feature through a learnable matrix, which is expressed as follows: c g =Embed(G). Among them, c g represents the category conditional feature, Embed represents the learnable dimensionality-raising layer, and G represents the category label of the video.

6. The method for composing music for a long-term rhythmic video according to claim 5, characterized in that: Also includes: Use a learnable matrix for feature mapping, where additional positional features are added; as well as The Hawkes process is introduced to model the position of the binary vector while adding a bias based on the proximity information of the rhythm point, which can be expressed as follows: Hawkes(t i )=sin(ω×i+w×t i ), Among them, c r Indicates the original rhythmic condition characteristics, represents the rhythm condition feature after bias, W r represents a learnable parameter, Hawkes represents the Hawkes process function, and w represents the bias hyperparameter.

7. A device for composing music for a long-term rhythmic video, which executes the steps of the method according to any one of claims 1 to 6, characterized in that: include: an audio feature extraction module configured to extract audio features through an implicit unconditional generative model; a video feature extraction module configured to extract video features from a long-duration rhythmic video through a context-aware conditional encoder; as well as A soundtrack generation module is configured to interactively fuse the audio features and the video features using a cross-modal attention module through an implicit conditional diffusion generation model to generate a soundtrack.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 6.

9. A computer system comprising: a processor configured to execute machine-executable instructions; as well as A memory having machine executable instructions stored thereon, wherein the machine executable instructions, when executed by a processor, perform the steps of the method according to any one of claims 1 to 6.

Images