A valence-arousal guided video-to-music generation method and system

Abstract

The present application relates to a kind of video to music generation method and system based on valence-arousal guidance, method includes: obtaining the visual description text of input video segment and corresponding valence and arousal value;Calculate global optical flow amplitude average and average motion speed;Average motion speed is quantitatively mapped, and density feature is obtained;Generate motion rhythm feature sequence bearing video motion dynamics and encode, obtain rhythm condition vector;Valence and arousal value are spliced, and obtain emotional feature vector;Emotional feature vector is mapped to the same multi-dimensional hidden space with text feature vector, and obtain the projected emotional feature;Based on modulation weight, emotional feature vector and text feature vector are gate weighted fusion, and obtain emotional enhancement condition vector;Joint condition music decoding is carried out, and target video is converted into music.Compared with prior art, the present application realizes fine emotional guidance and rhythm joint decoding under continuous emotional space.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and multimedia content generation technology, and in particular to a video-to-music generation method and system based on valence-arousal guidance, which is especially suitable for application scenarios that require video and music to be coordinated and aligned in the emotional and rhythmic dimensions, such as film and television post-production music, short video post-production music, and virtual reality scene music generation. Background Technology

[0002] With the rapid development of short video, film and television production, virtual reality, and digital content creation industries, the scale of multimedia content production continues to expand, and automated video soundtrack technology is receiving increasing attention from academia and industry. Video-to-Music generation aims to automatically generate background music that matches the visual content of an input video in terms of semantics, rhythm, and emotion. It is a key step in the intelligent production of multimedia content and is of great significance for lowering the creative threshold and improving soundtrack efficiency.

[0003] However, existing video-to-music conversion methods mainly suffer from the following technical shortcomings: First, there is insufficient modeling of the emotional dimension. Existing methods mainly rely on the semantic or motion features of videos for music generation, lacking explicit modeling and quantitative representation of the emotional state of the video. This results in poor consistency between the generated music and the video in terms of emotional dimension, making it difficult to present an auditory effect that matches the atmosphere of the visuals.

[0004] Second, the precision of emotion control is insufficient. Although some methods introduce emotion labels (such as Russell's four-quadrant labels) as generation conditions, the granularity of emotion labels is relatively coarse, which can only represent a limited number of emotion categories. This makes it impossible to perform fine-grained control in a continuous emotion space, thus limiting the richness and accuracy of musical emotion expression.

[0005] Third, there is a lack of joint modeling for emotion and rhythm. Existing methods usually treat emotion alignment and rhythm alignment as independent optimization objectives, lacking a unified joint condition mechanism. This makes it difficult for the generated music to simultaneously meet the alignment requirements of the video in both the emotion and rhythm dimensions, easily leading to problems such as "emotional fit but rhythm mismatch" or "rhythm alignment but atmosphere mismatch".

[0006] The Valence-Arousal (VA) model is a widely used continuous emotion space representation model in psychology. Valence reflects the degree of pleasure experienced by an emotion, while arousal reflects the intensity of emotional activation. Compared to discrete emotion category labels, VA values ​​can provide a fine-grained characterization of emotional states in a two-dimensional continuous space, offering a more precise conditional signal for emotion-guided content generation.

[0007] The invention disclosed in CN121659224A presents a multimodal sentiment analysis method based on frame-level valence arousal enhancement, comprising: S1. obtaining token-level text features with a dimension of 1024; S2. obtaining frame-level audio features with a dimension of 64; S3. obtaining frame-level visual features with a dimension of 128; S4. obtaining the Valence tag sequence and Arousal tag sequence for each frame; S5. inputting the audiovisual joint feature sequence into a joint cross-modal attention mechanism for recursive feature modeling to generate preliminary audiovisual fusion features; S6. obtaining VA-enhanced audio features and VA-enhanced visual features; S7. using the token-level text features as a query, performing cross-modal attention operations on the VA-enhanced audio features and VA-enhanced visual features, and integrating them through a self-attention mechanism to obtain the final multimodal sentiment features. This invention effectively overcomes the shortcomings of traditional methods in handling fine-grained emotional fluctuations and long-term temporal variations, significantly improving the sensitivity and temporal robustness of sentiment recognition. However, this scheme does not achieve joint alignment of emotion and rhythm during the generation process, resulting in poor accuracy of the conditional signal.

[0008] Therefore, how to achieve refined emotional guidance in a continuous emotional space, and at the same time achieve joint alignment of emotion and rhythm in the generation process, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a video-to-music generation method and system based on valence-arousal guidance.

[0010] The objective of this invention can be achieved through the following technical solutions: A video-to-music generation method based on valence-arousal guidance includes: Obtain the visual description text of the input video segment and its corresponding effectiveness value and arousal value; calculate the average global optical flow amplitude of each frame in the input video segment and the average motion velocity corresponding to each music measure; quantize and map the average motion velocity to obtain the density features of the music measure; Extract the transient burst response of the one-dimensional motion feature signal in the input video segment; quantize and map the transient burst response into discrete beat intensity features; convert the density features and beat intensity features into actual audio waveforms and generate a motion rhythm feature sequence carrying the dynamics of the video motion; encode the motion rhythm feature sequence to obtain a rhythm condition vector; The visual description text is encoded to obtain a text feature vector; the effect value and arousal value are independently mapped nonlinearly and then concatenated to obtain an emotion feature vector; the emotion feature vector is mapped to the same multidimensional latent space as the text feature vector to obtain the projected emotion feature. Based on the text semantics corresponding to the text feature vector, the modulation weight of the emotion feature is calculated; based on the modulation weight, the emotion feature vector and the text feature vector are gated and weighted to obtain the emotion enhancement condition vector; based on the emotion enhancement condition vector and the rhythm condition vector, joint conditional music decoding is performed to generate the target video-to-music.

[0011] Furthermore, the efficacy value and arousal value are independently mapped nonlinearly and then concatenated to obtain the emotional feature vector. The corresponding concatenation expression is: in, For sentiment feature vectors, This indicates a concatenation operation along the feature dimension. Represents a valence mapping network. This indicates a wakefulness mapping network. Indicates effectiveness value. This represents the wake-up level value.

[0012] Furthermore, based on the text semantics corresponding to the text feature vector, the modulation weight of the sentiment feature is calculated, and the corresponding calculation formula is as follows: Where G is the gating weight, and its value reflects the injection intensity of emotional signals on each feature dimension; This indicates a concatenation operation along the feature dimension. The learnable parameter matrix of the gated linear layer is used to convert the sentiment feature vector. Mapped to the text feature vector With the same feature dimensions, the projected sentiment feature vector is obtained. σ is the Sigmoid activation function. is the learnable parameter matrix of the linear projection layer.

[0013] Furthermore, based on the modulation weights, the sentiment feature vector and the text feature vector are fused using a gated weighted method, and the corresponding fusion expression is: in, Enhance the conditional vector for emotion.

[0014] Furthermore, based on the aforementioned emotion enhancement conditional vector and rhythm conditional vector, joint conditional music decoding is performed to generate the target video-to-music conversion. The corresponding decoding expression is: in, To convert the generated target video into music, This is the rhythm condition vector. Enhance the conditional vector for emotion. For the converter input of the decoder, This indicates a decoding operation.

[0015] Furthermore, the average global optical flow amplitude of each frame in the input video segment and the average motion velocity corresponding to each music measure are calculated using the following formulas: in, Let be the average velocity corresponding to the m-th musical measure. The number of video frames contained in each music measure. This represents the set of video frames corresponding to the m-th section. This indicates the pixel position of frame t. The optical flow amplitude at that location, Let H be the average global optical flow amplitude of frame t, and H and W be the height and width of the video frame, respectively.

[0016] Furthermore, the average motion velocity is quantized and mapped to obtain the density features of the music measure. The corresponding quantization mapping expression is: in, The density characteristics of musical measures, For the preset interval threshold vector, (·) indicates a quantization operation.

[0017] Furthermore, the transient burst response is quantized and mapped into discrete beat intensity features, and the corresponding quantization mapping expression is: in, Represents discrete beat intensity characteristics. Indicates quantization operation. Indicates the time when the beat occurs Transient burst response, It is a one-dimensional motion characteristic signal. This indicates high-pass filter processing. This represents the preset intensity threshold vector.

[0018] Furthermore, the density features and beat intensity features are converted into actual audio waveforms, and a motion rhythm feature sequence carrying the dynamics of video motion is generated. The corresponding conversion expression is: in, Represents a sequence of features related to the rhythm of motion. This function selects the corresponding measure rhythm pattern from a pre-defined rhythm pattern library based on density level. A function to generate the corresponding beat accent intensity based on the intensity level. This represents a timing-based concatenation operation. The density characteristics of musical measures, It represents discrete beat intensity characteristics.

[0019] The present invention also provides a system for a video-to-music generation method based on valence-arousal guidance, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0020] Compared with the prior art, the present invention has the following advantages: (1) This invention obtains an emotion enhancement condition vector by gating and weighting the emotion feature vector and the text feature vector; then, the emotion enhancement condition vector (carrying the fused text semantics and emotion information) and the rhythm condition vector (carrying the video motion dynamics information) are jointly conditionally decoded, so that the decoder can refer to both types of condition information when generating audio waveforms, thereby realizing the synergistic constraint of semantic and rhythm levels; avoiding the problem of melody and rhythm being separated from each other caused by the independent generation of emotion semantics and rhythm in traditional methods, greatly reducing the randomness and inconsistency of generated music, so that the final output audio can present a high degree of unity and coherence in multiple dimensions such as rhythm and emotional tension, effectively solving the problem that the two types of objectives are independent and difficult to balance in existing methods.

[0021] (2) This invention decouples the VA values ​​by using mutually independent valence mapping networks and arousal mapping networks, avoiding confusion between the two types of semantics, pleasure and activation, under shared parameters, and improving the expression accuracy of the emotional feature vector. The designed gating adapter realizes the adaptive fusion of text semantic features and continuous emotional features, and constructs a unified emotional enhancement condition vector, taking into account both semantic fidelity and emotional regulation.

[0022] (3) This invention introduces continuous valence-arousal value as an explicit condition into the video-to-music generation framework for the first time. Compared with the traditional discrete emotion tagging scheme, it can achieve refined emotion control in a continuous two-dimensional emotion space, significantly improving the richness of the emotional expression of the generated music. Finally, it realizes refined emotion guidance and rhythm joint decoding in a continuous emotion space, which can be used for film and television scores, automatic music matching for short videos and virtual reality scene music generation. Attached Figure Description

[0023] Figure 1 This is a flowchart of a video-to-music generation method based on valence-arousal guidance provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall process of a video-to-music generation method based on valence-arousal guidance provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the construction process of an emotion feature vector in a video-to-music generation method based on valence-arousal guidance provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the internal structure of an adapter for a valence-arousal-guided video-to-music generation method provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] Example 1 like Figure 1 As shown, this embodiment provides a video-to-music generation method based on valence-arousal guidance, which includes the following steps: S1: Obtain the visual description text of the input video segment and the corresponding effectiveness value and arousal value; calculate the average global optical flow amplitude of each frame in the input video segment and the average motion velocity corresponding to each music measure; quantize and map the average motion velocity to obtain the density features of the music measure; Specifically, Input the video clip into the multimodal large language model to obtain the visual description text of the video clip and the corresponding valence-arousal value (VA value); where the valence value v and the arousal value a are both integers from 1 to 9. The lower the value, the lower the pleasure and arousal, and the higher the value, the higher the pleasure and arousal. Preferred, Input information retrieval includes the following operations: Given a video clip, it is fed into the multimodal large language model GPT-4V. The model outputs two parts simultaneously: (1) visual description text, which includes natural language descriptions such as visual theme, scene description, and overall tone; (2) efficacy value v and arousal value a, both of which are integers in the range of 1 to 9. The efficacy value v represents the pleasantness of the video scene, and the arousal value a represents the activation intensity of the video scene. The lower the value, the lower the pleasantness and arousal. The multimodal large language model GPT-4V has the ability to perceive video emotions during the pre-training or instruction fine-tuning stage and can directly output VA values ​​based on visual semantics.

[0028] S2: Extract the transient burst response of the one-dimensional motion feature signal in the input video segment; quantize and map the transient burst response into discrete beat intensity features; convert the density features and beat intensity features into actual audio waveforms and generate a motion rhythm feature sequence that carries the dynamics of the video motion; encode the motion rhythm feature sequence to obtain a rhythm condition vector; S201: Dense Optical Flow Calculation. Calculate the dense optical flow field for adjacent frames of the input video segment, and calculate the average global optical flow amplitude in frame t. , Where H and W are the height and width of the video frame, respectively. This indicates the pixel position of frame t. The optical flow amplitude at that location.

[0029] S202: Calculation of motion density at the measure level. Calculate the average velocity within the m-th musical measure. : in, The number of video frames contained in each music measure. This represents the set of video frames corresponding to the m-th segment. A preset interval threshold vector is used. right Perform quantization to map the density level of the section. The calculation formula is as follows: The motion density levels are divided into 16 levels, corresponding to discrete levels from extremely low motion density to extremely high motion density.

[0030] S203: Beat-level intensity calculation. For one-dimensional motion feature signals in video. Apply a high-pass filter (HPF) to filter out low-frequency motion trends, retain transient motion changes, and extract them at the moment of beat occurrence. Transient burst response : Similarly, using the intensity threshold vector Continuous response values Mapped to discrete beat intensity levels : There are 6 levels of beat intensity, corresponding to beat strength levels from very weak to very strong.

[0031] S204: Rhythm Point Audio Synthesis. This involves synthesizing measure-level density features. Intensity characteristics of beat level Converted into actual audio waveforms, this generates rhythmic music that reflects the dynamics of the video motion. in, This function selects the corresponding measure rhythm pattern from a pre-defined rhythm pattern library based on density level. This function generates the corresponding beat accent intensity based on the intensity level, where ⊕ represents a temporal splicing operation.

[0032] S205: Rhythm Conditional Encoding. The above motion rhythm feature sequence is encoded using a rhythm conditional encoder. Positional encoding preserves the temporal position information of the beats, and a linear transformation projects the motion beat features at each time step onto a dimensional space with the same dimensions as the decoder's hidden state, generating a rhythm conditional vector. .

[0033] S3: Encode the visual description text to obtain the text feature vector; perform independent nonlinear mapping on the effect value and arousal value, and then concatenate them to obtain the sentiment feature vector; map the sentiment feature vector to the same multidimensional latent space as the text feature vector to obtain the projected sentiment features; S4: Based on the text semantics corresponding to the text feature vector, calculate the modulation weight of the emotion feature; based on the modulation weight, perform gated weighted fusion of the emotion feature vector and the text feature vector to obtain the emotion enhancement condition vector; based on the emotion enhancement condition vector and the rhythm condition vector, perform joint conditional music decoding to generate the target video-to-music.

[0034] Preferred, The construction of the sentiment enhancement conditional vector includes the following operations: 4-1: Text Feature Encoding. A T5 encoder is used to encode the visual descriptive text, resulting in a 768-dimensional text feature vector. .

[0035] 4-2: Independent Valence Mapping. Two independent multilayer perceptron (MLP) networks are designed to perform independent nonlinear mappings on valence and arousal values, respectively, to avoid semantic confusion between the two values ​​when sharing parameters. Valence Mapping Network and wakefulness mapping network Each vector contains two hidden layers, uses ReLU activation, has an input dimension of 1, and an output dimension of 512. Finally, the valence feature vector MLPv(v) and the arousal feature vector MLPa(a) are concatenated (Concat) to obtain a 1024-dimensional sentiment feature vector. : 4-3: Gated Adapter Fusion. This involves fusing sentiment feature vectors... With text feature vectors Common input adapter. The adapter internally uses a linear projection layer. The 1024-dimensional sentiment feature vector Mapped to The same 768-dimensional latent space yields projected emotional features. : The gating unit then calculates the modulation weight G of the sentiment features based on the semantics of the text: Where σ is the Sigmoid activation function. , All are learnable parameter matrices; the dimension of G and Consistent, its value reflects the injection strength of emotional signals across each feature dimension. Emotional Enhancement Conditional Vector The result obtained through gated weighted fusion is: Here, ⊙ represents element-wise multiplication. The gating mechanism can adaptively adjust the injection intensity of sentiment features based on the semantics of the text, avoiding excessive coverage of the text semantics by the sentiment signal; at the same time, it achieves effective alignment and fusion of text features and sentiment features in heterogeneous dimensions through the projection layer inside the adapter.

[0036] Preferred, The joint conditional music decoding and generation operation includes: Enhance the emotional condition vector With rhythm condition vector The target music is generated by inputting the MusicGen decoder and using the following autoregressive formula: Where Y is the transformer input of the decoder, which includes the projection representation of the audio codebook and positional encoding information; at each time step, the decoder controls the beat structure of the generated music according to the rhythm condition, and controls the mode, harmony and timbre style of the generated music according to the emotion condition, thereby achieving the joint alignment and generation of emotion and rhythm.

[0037] This embodiment also provides a system for a video-to-music generation method based on valence-arousal guidance, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0038] Example 2 like Figure 2 As shown, this embodiment provides a specific application step of a video-to-music generation method based on valence-arousal guidance, including: (1) Video input and multimodal analysis: A 10-second video clip at 30 FPS was selected as input. A multimodal large language model (such as GPT-4V) was used as the visual understanding module. Preset prompts guided the model to synchronously output visual descriptive text and valence-arousal values. For example, for a video of "running on the beach at sunset," the model could output the text "Under the golden sunset, people run along the beach, the scene is warm and bright," and output a valence value v=8 and an arousal value a=7, corresponding to a high pleasure level and high activation intensity emotional state.

[0039] (2) Rhythm feature extraction: See Figure 2 The rhythm branch calculates the dense optical flow field for adjacent frames, setting the number of video frames contained in each music measure. =60 (corresponding to a beat rate of 90 BPM). The interval threshold vector p is determined by the quantiles of the motion velocities in the training set. The intensity threshold vector q is determined by the quantiles of the transient response. After dense optical flow calculation, bar-level motion density quantization, beat-level transient intensity quantization, and rhythm point synthesis, a rhythmic music (Rhythm) is generated. Finally, a rhythm conditioner encodes this rhythmic feature sequence, outputting a rhythm condition vector. .

[0040] (3) Construction of emotional characteristics (see) Figure 3 ): The visual description text is encoded using a T5-base encoder, outputting a 768-dimensional text feature vector. Valence mapping network Arousal mapping network Both contain two hidden layers with dimensions of 256 and 512 respectively, using ReLU activation function. The valence value v=8 and arousal value a=7 are fed into MLPv and MLPa respectively, resulting in a 512-dimensional valence feature vector and a 512-dimensional arousal feature vector. These two vectors are then concatenated to obtain a 1024-dimensional sentiment feature vector. .

[0041] (4) Gating adapter fusion (see Figure 4 ): Text feature vectors (768-dimensional) and sentiment feature vector (1024 dimensions) are jointly sent to the entry control adapter. Inside the adapter, the data first passes through the projection layer. (Dimensions 1024×768) Mapped to The same 768-dimensional latent space yields projected emotional features. ;Will and Common input gated linear layer (Dimension 1536×768), and then passed through the Sigmoid function to obtain a 768-dimensional gated vector G; G and After element-wise multiplication, then with... The residuals are summed to output a 768-dimensional emotion enhancement conditional vector. .

[0042] (5) Joint decoding generation: MusicGen's Decoder is used as the music decoder, with a codebook size of 4 and a sampling rate of 32kHz. The decoder simultaneously receives the rhythm condition vector at each time step. With sentiment enhancement conditional vector As input, an autoregressive algorithm generates a stereo music waveform of the same duration as the input video. The generated music closely matches the original video content in terms of emotional atmosphere and rhythm.

[0043] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A video-to-music generation method based on valence-arousal guidance, characterized in that, include: Obtain the visual description text of the input video clip, along with its corresponding effectiveness value and arousal value; Calculate the average global optical flow amplitude of each frame in the input video segment and the average motion velocity corresponding to each music measure; quantize and map the average motion velocity to obtain the density features of the music measure; Extract the transient burst response of the one-dimensional motion feature signal from the input video segment; quantize and map the transient burst response into discrete beat intensity features; The density features and beat intensity features are converted into actual audio waveforms, and a motion rhythm feature sequence carrying the dynamics of video motion is generated; the motion rhythm feature sequence is encoded to obtain a rhythm condition vector; The visual description text is encoded to obtain a text feature vector; the effect value and arousal value are independently mapped nonlinearly and then concatenated to obtain an emotion feature vector; the emotion feature vector is mapped to the same multidimensional latent space as the text feature vector to obtain the projected emotion feature. Based on the text semantics corresponding to the text feature vector, the modulation weight of the sentiment feature is calculated; Based on the modulation weights, the emotion feature vector and the text feature vector are gated and weighted to obtain the emotion enhancement condition vector; based on the emotion enhancement condition vector and the rhythm condition vector, joint conditional music decoding is performed to generate the target video-to-music.

2. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, The efficacy value and arousal value are independently mapped nonlinearly and then concatenated to obtain the emotional feature vector. The corresponding concatenation expression is as follows: in, For sentiment feature vectors, This indicates a concatenation operation along the feature dimension. Represents a valence mapping network. This indicates a wakefulness mapping network. Indicates effectiveness value. This represents the wake-up level value.

3. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, Based on the text semantics corresponding to the text feature vector, the modulation weight of the sentiment feature is calculated, and the corresponding calculation formula is as follows: Where G is the gating weight, and its value reflects the injection intensity of emotional signals on each feature dimension; This indicates a concatenation operation along the feature dimension. The learnable parameter matrix of the gated linear layer is used to convert the sentiment feature vector. Mapped to the text feature vector With the same feature dimensions, the projected sentiment feature vector is obtained. σ is the Sigmoid activation function. is the learnable parameter matrix of the linear projection layer.

4. The video-to-music generation method based on valence-arousal guidance according to claim 3, characterized in that, Based on the modulation weights, the sentiment feature vector and the text feature vector are fused using a gated weighted method, and the corresponding fusion expression is: in, Enhance the conditional vector for emotion.

5. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, Based on the aforementioned emotion enhancement conditional vector and rhythm conditional vector, joint conditional music decoding is performed to generate the target video-to-music conversion. The corresponding decoding expression is: in, To convert the generated target video into music, This is the rhythm condition vector. Enhance the conditional vector for emotion. For the converter input of the decoder, This indicates a decoding operation.

6. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, The average global optical flow amplitude of each frame in the input video segment and the average motion velocity corresponding to each music bar are calculated using the following formulas: in, Let be the average velocity corresponding to the m-th musical measure. The number of video frames contained in each music measure. This represents the set of video frames corresponding to the m-th section. This indicates the pixel position of frame t. The optical flow amplitude at that location, Let H be the average global optical flow amplitude of frame t, and H and W be the height and width of the video frame, respectively.

7. The video-to-music generation method based on valence-arousal guidance according to claim 6, characterized in that, The average motion velocity is quantized and mapped to obtain the density features of the music measure. The corresponding quantization mapping expression is as follows: in, The density characteristics of musical measures, For the preset interval threshold vector, (·) indicates a quantization operation.

8. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, The transient burst response is quantized and mapped to discrete beat intensity features, and the corresponding quantization mapping expression is: in, Represents discrete beat intensity characteristics. Indicates quantization operation. Indicates the time when the beat occurs Transient burst response, It is a one-dimensional motion characteristic signal. This indicates high-pass filter processing. This represents the preset intensity threshold vector.

9. The video-to-music generation method based on valence-arousal guidance according to claim 1, characterized in that, The density and beat intensity features are converted into actual audio waveforms, and a sequence of motion rhythm features carrying the dynamics of video motion is generated. The corresponding conversion expression is: in, Represents a sequence of features related to the rhythm of motion. This function selects the corresponding measure rhythm pattern from a pre-defined rhythm pattern library based on density level. A function to generate the corresponding beat accent intensity based on the intensity level. This represents a timing-based concatenation operation. The density characteristics of musical measures, It represents discrete beat intensity characteristics.

10. A system for video-to-music generation based on valence-arousal guidance, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor invoking the computer program to perform the steps of the method as described in any one of claims 1 to 9.

Images