A video music aesthetic evaluation method based on cross-attention mechanism and weakly supervised learning

Abstract

The present application relates to the technical field of multi-modal machine learning and music information retrieval, and particularly relates to a video music aesthetics evaluation method based on cross-attention mechanism and weakly supervised learning.A video music aesthetics evaluation method based on cross-attention mechanism and weakly supervised learning is disclosed, which realizes the automatic evaluation of the aesthetic quality of video background music by constructing a multi-dimensional music aesthetics evaluation network MEMA (Multi-dimensional Evaluation of Music Aesthetics).The present application adopts a two-stage training strategy: in the pre-training stage, the model learns the deep correspondence between music and scenes through the cross-attention mechanism without manual annotation; in the fine-tuning stage, weakly supervised learning is used, combined with a large language model, to generate weak labels based on user comments to train the feature projector and scoring ability.The evaluation system comprehensively evaluates the music quality from three dimensions: narrative emotional consistency, technical integration, and theme recognition and originality.The present application solves the problems of strong subjectivity, high labeling cost, and insufficient cross-modal understanding of traditional music evaluation, and provides technical support for the automatic evaluation of video music, music creation assistance, and other applications.

Description

Technical Field

[0001] This invention relates to the fields of multimodal machine learning and music information retrieval technology, and in particular to a method for evaluating the aesthetics of video soundtracks based on cross-attention mechanism and weakly supervised learning. Background Technology

[0002] As a comprehensive art form, background music plays a crucial role in shaping emotions and advancing rhythm in video. Excellent video scores can deepen the narrative and enhance the audience's emotional resonance, while inappropriate scores can ruin the overall viewing experience. However, evaluating the aesthetic quality of video scores has always been a challenging issue.

[0003] Existing music evaluation methods have the following limitations:

[0004] First, it is highly subjective and lacks objective standards. Musical aesthetic evaluation is highly dependent on personal taste and cultural background, and traditional manual evaluation methods are costly and difficult to scale. Although there are some automatic evaluation methods based on audio features, these methods usually only focus on the technical characteristics of the music itself, ignoring the core value of video soundtracks, which lies in their fit with the visual narrative.

[0005] Second, the limitations of unimodal evaluation. Most existing music evaluation systems are unimodal, analyzing only the audio signal itself. However, the aesthetic value of video soundtracks largely depends on their coordination with visual elements such as visuals, plot, and characters. Evaluating video soundtracks without considering their visual context is as one-sided as evaluating a song without considering its lyrics.

[0006] Third, labeled data is scarce. High-quality evaluation of musical aesthetics requires specialized knowledge, and obtaining large-scale expert-labeled data is extremely costly. This severely limits the application of supervised learning methods in this field. Although some studies have attempted to use user ratings as labels, these ratings are often too coarse to reflect the multidimensional characteristics of musical aesthetics.

[0007] Fourth, the evaluation dimensions are too narrow. Existing methods typically only output an overall score, failing to provide granular feedback. For video creators, understanding the specific aspects in which the music performs well or poorly is far more valuable. Furthermore, different types of videos have vastly different requirements for music, and a single scoring system is ill-suited to this diversity.

[0008] Fifth, insufficient cross-modal understanding. Semantic alignment between music and video is crucial for evaluating video soundtracks, but most existing multimodal fusion methods remain at the level of feature splicing or simple attention mechanisms, lacking an understanding of the deep correspondence between music and scene. This results in models only capturing superficial statistical correlations, rather than truly understanding the narrative function of the soundtrack.

[0009] In recent years, the development of deep learning and multimodal pre-training techniques has offered new possibilities for solving the aforementioned problems. In particular, self-supervised learning and weakly supervised learning methods can learn effective representations even with limited labeled data. However, designing suitable pre-training objectives for video soundtrack evaluation tasks, extracting reliable training labels from weakly supervised signals such as user comments, and constructing model architectures capable of understanding the deep visual relationship between music and video remain pressing technical challenges.

[0010] In summary, existing technologies have many shortcomings in evaluating the aesthetic quality of video soundtracks, and there is an urgent need for an automated evaluation method that can comprehensively consider the relationship between music and visuals, support multi-dimensional evaluation, and utilize weakly supervised signals.

[0011] In this invention, "cross-modal imagination" refers to the ability of a model to predict or reconstruct the semantic features of another modality through an "imagination" mechanism, without explicit labeling or direct pairing of samples. In other words, it enables the model to "imagine" the possible corresponding musical semantics when seeing a video scene, or to "imagine" the corresponding visual narrative when hearing music. This capability breaks through the shallow associations of traditional feature splicing or attention alignment, giving the model cross-modal semantic reasoning and generation capabilities, thereby improving the understanding of the deep "music-visual" relationship in the evaluation of the aesthetic quality of video soundtracks. Summary of the Invention

[0012] This invention addresses the shortcomings of existing technologies by providing a video soundtrack aesthetic evaluation method based on cross-attention mechanism and weakly supervised learning.

[0013] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0014] A method for evaluating the aesthetics of video soundtracks based on cross-attention mechanism and weakly supervised learning, the method comprising:

[0015] S1: Obtain pre-collected original video audio, corresponding video clip sets, and a weakly labeled dataset generated based on user comments;

[0016] S2: Perform multi-level feature extraction on audio and video data to construct a multimodal representation of music scenes;

[0017] S3: Construct a multi-dimensional music aesthetics evaluation network that includes a segment-level audio-video fusion module, a dual-tower encoder, a cross-modal imagination module, a guided cross-attention alignment module, and an evaluation task module;

[0018] S4: A two-stage training strategy is adopted. First, the model learns the inherent correspondence of music scenes through self-supervised pre-training, and then the scoring criteria are learned through weak supervision fine-tuning.

[0019] S5: Use the trained model to evaluate the aesthetic quality of new video soundtracks in three dimensions, and output the narrative emotional consistency score, technical integration score, theme recognition and originality score, and evaluation text.

[0020] Further, step S1 includes:

[0021] S101: Obtain a list of videos containing original audio and their corresponding video clips from a pre-prepared local video dataset;

[0022] S102: Obtain a pre-collected dataset of user comments on video soundtracks;

[0023] S103: Use a large language model to perform structured analysis on comments and generate weak labels in three dimensions:

[0024] Narrative-emotional consistency: Evaluating the degree of match between the emotional atmosphere of the background music and the narrative of the visuals;

[0025] Technical integration: This evaluates the degree to which the background music is integrated with sound elements such as dialogue and sound effects;

[0026] Theme identification and originality: Evaluating the distinctiveness, memorability, and artistic innovation of the soundtrack;

[0027] S104: For each dimension's weak labels, the final score and confidence level are calculated through like weighting, self-consistent sampling, and multi-model fusion.

[0028] S105: Normalize the score range of the original labeled data from 1 to 10 to the range of 0 to 1. The normalization formula is: Normalized score = (Original score - 1.0) / 9.0.

[0029] S106: Dataset Construction. Store the collected background music, videos, and evaluation texts in a single folder.

[0030] Furthermore, the feature extraction in step S2 includes:

[0031] S201: Audio feature extraction, extracting features from the original audio and audio clip tracks respectively:

[0032] Macro-semantic sequence: 512-dimensional semantic embeddings were extracted using a contrastive language audio pre-trained model, and segmented using a 10-second time window and a 2-second step size;

[0033] Bottom and middle layer features: 37-dimensional traditional audio features are extracted using an audio processing library, including 13-dimensional Mel frequency cepstral coefficients, 12-dimensional chromaticity features, 2-dimensional root mean square energy features, 1-dimensional zero-crossing rate features, and 9-dimensional timbre features. The timbre features include spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, and spectral contrast.

[0034] S202: Video feature extraction, extracting features for each video segment:

[0035] Global semantic features: 768-dimensional video semantic features are extracted using a video understanding pre-trained model, with 8 frames uniformly sampled from each video segment;

[0036] Low-level visual features: The video is divided into 5 time windows, and 40-dimensional features are extracted from each window, including 8-dimensional basic color statistical features, 8-dimensional texture features, 6-dimensional edge features, 6-dimensional frequency domain features, 8-dimensional high-level color features, and 4-dimensional spatial complexity features, resulting in 200 low-level features in total, which are obtained by multiplying the 5 time windows by 40 dimensions.

[0037] S203: Feature standardization and concatenation, concatenating the 768-dimensional global semantic features with the flattened 200-dimensional underlying features to obtain a 968-dimensional standardized multimodal feature representation for each video segment.

[0038] S204: Use the feature extractor of VideoLLaMA2 to extract features for use in subsequent text generation.

[0039] Furthermore, the multi-dimensional music aesthetics evaluation network includes:

[0040] S301: Segment-level audio-visual fusion module, which uses a cross-attention mechanism to fuse the 968-dimensional features of each video segment with the 549-dimensional features of the corresponding audio track into a unified 768-dimensional scene token;

[0041] S302: Dual tower encoder, including:

[0042] Music Coding Tower: Uses a 6-layer Transformer encoder to process the temporal features of the original soundtrack audio. Each layer contains 12 attention heads and a 3072-dimensional feedforward neural network hidden layer.

[0043] Scene Encoding Tower: Uses a 6-layer Transformer encoder to process the fused set of scene tokens, with each layer containing 12 attention heads and a 3072-dimensional feedforward neural network hidden layer;

[0044] S303: Cross-modal imagination module, realizing bidirectional cross-modal reconstruction based on conditional variational autoencoder:

[0045] Sound-to-view conditional variational autoencoder: Reconstructs scene features based on musical features;

[0046] Visual-to-audio conditional variational autoencoder: reconstructs music features based on scene features;

[0047] S304: Guided cross-attention alignment module, which uses a multi-head cross-attention mechanism to align musical narrative sequences with scene representations;

[0048] S305: Evaluation task module, which sets up independent prediction branches for the three evaluation dimensions. Each branch contains a score prediction head and a text generation head.

[0049] Furthermore, the implementation of the segment-level audio-visual fusion module includes:

[0050] S401: Perform layer normalization on the input video features and audio features respectively;

[0051] S402: A linear projection layer is used to project video features into a query vector and audio features into key and value vectors. The weights of the projection layer are initialized using a truncated normal distribution with a standard deviation of 0.02.

[0052] S403: Calculate the dot product of the query vector and the key vector, divide by the square root of 768 as the scaling factor, and obtain the attention score;

[0053] S404: Apply the Softmax function to the attention score to obtain the attention weights, and multiply the attention weights by the value vector to obtain the fused features;

[0054] S405: After performing residual connection between the fused features and the query vector, layer normalization is then applied;

[0055] S406: Further transformation is performed through a feedforward neural network, which contains two linear layers with the GELU activation function in between, and the hidden layer dimension is 4 times the model dimension;

[0056] S407: After performing residual connections between the output and input of the feedforward neural network, layer normalization is performed again to obtain the final scene token.

[0057] Furthermore, the implementation of the music encoding tower includes:

[0058] S501: Perform layer normalization on the input 512-dimensional contrastive language audio pre-trained features and 37-dimensional traditional audio features respectively;

[0059] S502: Uses a linear projection layer to project 512-dimensional contrastive language audio pre-trained features onto a 768-dimensional model space;

[0060] S503: Uses a one-dimensional convolutional neural network to process 37-dimensional traditional audio features. The convolutional neural network contains three convolutional layers with hidden dimensions of 256, 512, and 768, and outputs a fixed number of topic feature vectors.

[0061] S504: Concatenate the projected macroscopic semantic features with the topic features output by the convolutional neural network in the time dimension;

[0062] S505: Add sine and cosine position codes to the spliced ​​feature sequence;

[0063] S506: Encoded using a 6-layer Transformer encoder, each layer containing 12 attention heads, a 3072-dimensional feedforward neural network, a GELU activation function, layer normalization, and residual connections.

[0064] Furthermore, the implementation of the cross-modal imagination module includes:

[0065] S601: The encoder maps the input features to Gaussian distribution parameters mean μ and log-variance logσ² after average pooling, with a latent variable dimension of 128.

[0066] S602: Sample the latent variable z using the reparameterization technique. The sampling formula is:

[0067]

[0068] Where ε is random noise sampled from a standard normal distribution;

[0069] S603: The conditional decoder uses a 4-layer Transformer decoder, with latent variable z as input query and another modality feature as conditional memory, and generates imagined features of the target modality through a cross-attention mechanism;

[0070] S604: Calculate the reconstruction loss and KL divergence loss. The reconstruction loss uses mean squared error to measure the difference between imagined features and real features. The KL divergence loss regularizes the latent variable distribution to approximate the standard normal distribution.

[0071] Furthermore, the implementation of the guided cross-attention alignment module includes:

[0072] S701: Extract the first M tokens from the feature sequence output by the music coding tower as the macro-semantic part, where M is the sequence length of the contrastive language audio pre-trained features;

[0073] S702: Use a linear projection layer to project macroscopic semantic features into query vectors and project the output of the scene coding tower into key vectors and value vectors;

[0074] S703: Through a multi-head cross-attention mechanism with 12 attention heads, cross-modal attention is calculated using macro-level semantics of music as the query and scene features as the key and value.

[0075] S704: Perform residual connections and layer normalization on the attention output;

[0076] S705: Further transformation is performed through a feedforward neural network, which contains a 3072-dimensional hidden layer and the GELU activation function;

[0077] S706: Perform residual connection and layer normalization again to output the aligned music narrative sequence.

[0078] Furthermore, the two-stage training strategy includes:

[0079] S801: Self-supervised pre-training phase:

[0080] Data from all music and video pairs is used, eliminating the need for manual annotation;

[0081] Learn the intrinsic correspondence between music and scene through cross-modal reconstruction tasks;

[0082] The weighted combination of reconstruction loss, KL divergence loss, and alignment loss was optimized, with weight coefficients of 1.0, 0.01, and 0.5, respectively.

[0083] S802: Weak Supervision Fine-tuning Phase

[0084] Use a subset of data with weak labels;

[0085] Freeze the cross-modal imagination module, retain or fine-tune the pre-trained encoder weights;

[0086] Training the score prediction head and text generation head;

[0087] Use confidence-weighted mean squared error loss and cross-entropy loss.

[0088] Furthermore, the pre-training loss function is:

[0089]

[0090] in, To reconstruct the loss, mean squared error is used to measure the difference between imagined features and real features; KL divergence loss, regularized latent variable distribution; To align the loss, guide the distribution of attention. , , These are the weighting coefficients.

[0091] Furthermore, the evaluation task module design for the fine-tuning phase includes:

[0092] S901: Information preparation, preparing different feature representations for the three tasks respectively:

[0093] The narrative emotional consistency task uses the average value of the aligned musical narrative sequence output by the guided cross-attention alignment module;

[0094] The topic identification and originality tasks use the average value of the overall musical context output by the music coding tower;

[0095] The technology integration task uses the global features of the original audio track, the global features of the audio clip track, and their differences to splice together. The spliced ​​dimension is 768 multiplied by 3, which equals 2304 dimensions.

[0096] S902: Score prediction. Each task uses an independent two-layer multilayer perceptron. The first layer maps to a 512-dimensional hidden layer, and the second layer outputs a 1-dimensional score. The Sigmoid activation function is used to limit the output to the range of 0 to 1.

[0097] S903: Text generation, using a multilayer perceptron plus query transformer (simplified Q-Former) architecture to extract 16 query tokens, input into the language model to generate evaluation text;

[0098] S904: Loss Function Calculation and Optimization Strategy

[0099] The model employs a multi-task joint loss design during the weakly supervised fine-tuning stage to simultaneously optimize score prediction and text generation performance. The overall loss is defined as follows:

[0100]

[0101] NEC stands for Narrative Emotional Consistency;

[0102] TI stands for Technology Integration;

[0103] TIO stands for Topic Identification and Originality;

[0104] and The weighting coefficient is adjustable (default is 1.0 and 0.5).

[0105] 1) Score prediction loss (Loss_score)

[0106] For each task, a confidence-weighted mean squared error loss is used to calculate the difference between the predicted score and the weak label:

[0107]

[0108] in, A mask indicating whether a label exists (0 for missing labels). This indicates the confidence weight of the weak label. and These are the predicted and actual scores, respectively.

[0109] (2) Text generation loss

[0110] The text generation part employs a teacher-forced learning strategy, calculating cross-entropy loss only on the target evaluation text segment:

[0111]

[0112] in, For the target text token, This represents the multimodal feature embedding after mapping by the feature projector. For the frozen Qwen2-7B parameters, only the Feature Projector (MLP Projector) participates in gradient updates. Labels in non-target text regions are padded with −100 and do not participate in gradient calculation.

[0113] (3) Overall optimization strategy

[0114] The model only updates the feature projector parameters; the remaining large language model parameters remain frozen. The optimizer used is AdamW, with a learning rate of 5×10⁻⁻. 5 β1=0.9, β2=0.999, weight decay 0.01.

[0115] Supports mixed-precision training (bfloat16 / float16) and gradient accumulation, enables gradient checkpointing and memory optimization.

[0116] Furthermore, the method also includes a numerical stability guarantee mechanism:

[0117] Add a normalization layer to all input layers to standardize the features to a mean of 0 and a variance of 1;

[0118] Initialize with small variance weights and a truncated normal distribution with a standard deviation of 0.02;

[0119] The logarithmic variance of the conditional variational autoencoder is truncated and limited to the range of -10 to 10 to prevent exponential overflow.

[0120] The present invention also discloses a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0121] The present invention also discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0122] Compared with the prior art, the advantages of the present invention are as follows:

[0123] 1. For the first time, a multi-dimensional quantitative evaluation of video soundtracks has been achieved: The subjective aesthetic quality of video soundtracks is decomposed into three quantifiable theoretical dimensions: narrative emotional consistency, technical integration, and theme recognition and originality. This transforms the vague problem of art evaluation into a clear machine learning task, filling a gap in the field.

[0124] 2. Deep cross-modal understanding: Through the design of cross-modal imagination tasks, the model can truly understand the deep semantic relationship between music and visuals, rather than just learning the surface statistical correlation, which significantly improves the accuracy and interpretability of the evaluation.

[0125] 3. Efficient data utilization strategy: A two-stage training strategy is adopted to make full use of a large amount of unlabeled data for pre-training. Only a small number of weak labels are needed to complete task adaptation, which solves the problem of scarce professional labeled data.

[0126] 4. Scalable weakly supervised learning framework: Supports automatic generation of training labels from user comments, and handles label noise through confidence weighting, giving the system good scalability.

[0127] 5. Numerical stability guarantee: A numerical stability guarantee mechanism is introduced, which solves the numerical problems in deep model training through layer normalization and small variance initialization, thereby improving the robustness of the system.

[0128] 6. Broad application prospects: This invention can be used not only for video soundtrack evaluation, but also extended to multiple application areas such as video soundtrack recommendation and soundtrack creation assistance. Attached Figure Description

[0129] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0130] Figure 1 This is an overall flowchart of the video soundtrack aesthetic quality evaluation method based on cross-modal imagination pre-training and weakly supervised learning, according to an embodiment of the present invention.

[0131] Figure 2 This is a schematic diagram of the overall architecture of the multi-dimensional music aesthetics evaluation network according to an embodiment of the present invention;

[0132] Figure 3 This is a schematic diagram illustrating the effect of an embodiment of the present invention. Detailed Implementation

[0133] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0134] Reference Figure 1This embodiment provides a method for evaluating the aesthetics of video soundtracks based on cross-attention mechanism and weakly supervised learning, including the following steps:

[0135] Step S1: Data Acquisition and Weak Tag Generation

[0136] S101: Data Source Acquisition

[0137] Obtain a list of videos containing original soundtrack audio information from a pre-prepared local video dataset. The dataset contains original soundtrack audio files and corresponding video clips, ensuring that each soundtrack track has at least one and at most twenty video clips for use in various scenarios.

[0138] S102: User Comments Collection

[0139] Obtain a pre-defined dataset of user comments on video soundtracks. The dataset contains user comments sorted by the number of likes, prioritizing detailed comments. Each track should contain at least 100 comments to ensure representativeness of the evaluations.

[0140] S103: Weak Tag Generation

[0141] The comments are structured and analyzed using a large language model. The specific implementation is as follows:

[0142] 1) Comment preprocessing and segmentation

[0143] Each comment is divided into blocks by adding the prefix "[likes=N]" to indicate the number of likes. The weight of each block is calculated as the sum of the number of likes of all comments within that block.

[0144] 2) Three-dimensional scoring and text extraction

[0145] For each comment block, the large language model is prompted to score and analyze the text from the following three dimensions:

[0146] Narrative-emotional consistency: Evaluate whether the emotional atmosphere of the background music matches the narrative content and emotional tone of the visuals, with a score of 1 to 10;

[0147] Technical integration: Evaluates the degree and layering of the blending of the background music with the dialogue, ambient sounds, and sound effects, scored from 1 to 10;

[0148] Theme Identification and Originality: Evaluate whether the melody theme of the soundtrack is recognizable, memorable, and artistically innovative, with a score of 1 to 10.

[0149] 3) Self-consistent sampling and aggregation

[0150] For each block, perform three independent samplings using the same pattern to obtain three sets of scores. Calculate the block-level score:

[0151]

[0152] Where score_j and conf_j are the score and confidence level of the j-th sample, respectively.

[0153] Block-level confidence is calculated through consistency calibration:

[0154]

[0155]

[0156] 4) Cross-model fusion

[0157] The above processing was performed using two different large language models, and the final scores were fused using a confidence-weighted average.

[0158]

[0159]

[0160] S104: Weak Label Score Calculation and Aggregation

[0161] For each dimension of weak labels, the final score and confidence level are calculated through like weighting, self-consistent sampling, and multi-model fusion. The specific steps are as follows:

[0162] S104a: Calculation of comment block weight (like weight)

[0163] For each comment block analyzed by the large language model, its weight The sum of the likes for all comments within that block is used, and logarithmic scaling is applied for normalization. The calculation formula is as follows:

[0164]

[0165] Using logarithmic scaling can prevent a few comments with extremely high likes from having too much impact on the results, while preserving the positive incentive effect of likes; adding 1 is to handle the case where the number of likes is 0, ensuring the validity of the calculation.

[0166] S104b: Self-consistent Sampling and Block-Level Fractional Aggregation

[0167] For each comment block, the same large language model was used to perform 3 independent samplings to obtain 3 sets of scores. ) and confidence level ( ). Calculate the initial score for this comment block. and confidence level :

[0168]

[0169] S104c: Cross-model fusion and final score calculation

[0170] The above S104a-S104b processing was performed using two different large language models (e.g., GPT-4 and Claude-3) to obtain two sets of track dimension scores. , and confidence level The final score is obtained by weighting the scores based on confidence levels.

[0171]

[0172] S105: Data Normalization

[0173] The original fractions, ranging from 1 to 10, are normalized to the range of 0 to 1 using the following formula:

[0174] Normalized score = (raw score - 1.0) / 9.0

[0175] This normalization step is crucial; failure to perform normalization will result in excessive training loss and severely affect model convergence.

[0176] S106: Dataset Construction

[0177] The processed data is organized into a uniform format, with each sample containing:

[0178] Original audio file (mp3 format);

[0179] The corresponding collection of video clips (mp4 format);

[0180] The annotation file (JSON format) contains normalized scores and average confidence levels for the three dimensions, as well as the analysis text for each dimension.

[0181] Step S2: Multi-level feature extraction

[0182] S201: Audio Feature Extraction

[0183] 1) Regarding the original audio:

[0184] Macro-level semantic sequence: The sequence is divided into 10-second windows with 2-second steps. For each window, a CLAP model is used to extract 512-dimensional contrastive language audio pre-trained embeddings to form a temporal semantic sequence. Its dimension is M×512, where M is the number of windows after segmentation.

[0185] Low-level and mid-level features: Extracting 37-dimensional traditional features for each window using the torchaudio library:

[0186] 13-Vimel frequency cepstral coefficients;

[0187] 12-dimensional chromaticity features (pitch category distribution);

[0188] 2D root mean square energy characteristics (mean and variance);

[0189] 1-dimensional zero-crossing rate feature;

[0190] Nine-dimensional timbre characteristics, including spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, spectral contrast, and Mel spectral statistics;

[0191] Forming the characteristic matrix Its dimensions are M×37.

[0192] 2) For audio clips:

[0193] For the audio track in each video segment, extract the global contrastive language audio pre-trained embedding (512 dimensions) and the conventional features (37 dimensions) using average pooling to obtain the audio track features for each segment. Its dimensions are 549.

[0194] S202: Video Feature Extraction

[0195] For each video segment j:

[0196] 1) Global semantic features:

[0197] Using the InternVideo2 model, 8 frames are uniformly sampled from each video segment to extract... The layer's output serves as global semantic features. Its dimensions are 768.

[0198] 2) Low-level visual features:

[0199] The video is divided into 5 time windows, and 40-dimensional features are extracted from each window:

[0200] 8-dimensional basic color statistical characteristics: mean brightness, standard deviation of brightness, mean saturation, standard deviation of saturation, mean hue, standard deviation of hue, mean motion intensity, standard deviation of motion intensity;

[0201] 8-dimensional texture features: mean and standard deviation of Gabor filter response, local binary mode approximation, contrast, energy, homogeneity, correlation, and entropy;

[0202] 6-dimensional edge features: edge density, principal direction angle, direction standard deviation, average intensity, intensity standard deviation, and edge connectivity;

[0203] Six-dimensional frequency domain characteristics: mean and standard deviation of discrete cosine transform coefficients, low-frequency energy ratio, mid-frequency energy ratio, high-frequency energy ratio, and spectral complexity;

[0204] 8-dimensional advanced color features: mean and standard deviation of hue histogram, mean and standard deviation of saturation histogram, color complexity, color contrast, dominant hue percentage, and color dispersion.

[0205] 4-dimensional spatial complexity features: image entropy, spatial complexity, fractal dimension approximation, and Laplacian variance;

[0206] get Its dimensions are 5×40.

[0207] 3) Feature splicing:

[0208] The 768-dimensional global semantic features are concatenated with the flattened 200-dimensional low-level features:

[0209] =[ flatten( )]

[0210] in The dimension is 968, and flatten is to flatten the dimension of the object. Expanded into one dimension.

[0211] S203: Feature Standardization and Assembly

[0212] All extracted features are standardized using layer normalization to ensure a mean of 0 and a variance of 1, preventing numerical overflow in subsequent calculations, and finally concatenated.

[0213] Step S3: Construction of a Multi-Dimensional Music Aesthetics Evaluation Network

[0214] Reference Figure 2 A multi-dimensional music aesthetics evaluation network is constructed, comprising the following modules:

[0215] S301: Segment-level audio-visual fusion module

[0216] For each video segment j and its corresponding audio track:

[0217] 1) Input normalization:

[0218] For the input 968-dimensional video features and 549-dimensional audio features Perform layer normalization separately:

[0219]

[0220]

[0221] 2) Linear projection (initialized using a truncated normal distribution with a standard deviation of 0.02):

[0222] The output dimension is 768.

[0223] The output dimension is 768.

[0224] The output dimension is 768.

[0225] 3) Cross-attention fusion:

[0226] Calculate query vector With key vector The dot product, divided by the square root of 768, is used as a scaling factor:

[0227]

[0228] Multiply the attention weights by the value vector:

[0229]

[0230] 4) Feedforward network and residual connection:

[0231] After residual concatenation of the fused features and the query vector, layer normalization is performed.

[0232]

[0233] The transformation is performed using a feedforward neural network, which contains two linear layers and has a hidden layer dimension of 768 × 4 = 3072.

[0234]

[0235] Perform residual connections and layer normalization again:

[0236]

[0237] Output scene token set Its dimension is P×768, where P is the number of video segments.

[0238] S302: Dual-Tower Encoder

[0239] The dual-tower encoder includes:

[0240] 1) Music Encoding Tower:

[0241] Layer normalization is performed on the input 512-dimensional contrastive language audio pre-trained features and 37-dimensional traditional audio features respectively;

[0242] A linear projection layer is used to project the 512-dimensional contrastive language audio pre-trained features onto the 768-dimensional model space;

[0243] A one-dimensional convolutional neural network is used to process 37-dimensional traditional audio features. The convolutional neural network contains three convolutional layers with hidden dimensions of 256, 512, and 768, and outputs a fixed number of topic feature vectors K.

[0244] The M macro-semantic features and K thematic features after projection are concatenated in the time dimension to obtain a feature sequence of (M+K)×768.

[0245] Add sine and cosine position codes to the concatenated feature sequences;

[0246] Encoding is performed using a 6-layer Transformer encoder, each layer containing 12 attention heads, a 3072-dimensional feedforward neural network, a GELU activation function, layer normalization, and residual connections;

[0247] Output Its dimensions are (M+K)×768.

[0248] 2) Scene Coding Tower:

[0249] Input normalized scene token Its dimension is P×768;

[0250] The input is subjected to layer normalization to prevent numerical problems from the segment-level audio-visual fusion module from continuing to propagate;

[0251] Position encoding is not used because the set of fragments is unordered;

[0252] Processed through a 6-layer Transformer encoder, each layer contains 12 attention heads, a 3072-dimensional feedforward neural network, a GELU activation function, layer normalization, and residual connections;

[0253] Output Its dimensions are P×768.

[0254] S303: Cross-modal imagination module

[0255] The core idea of ​​this module is to simulate the human ability to associate auditory and visual information, i.e., "imagination." By introducing a Conditional Variational Autoencoder (VAE), the model can learn latent semantic mapping relationships between different modalities. When the input is music features, the model "imagines" the scene features that match it; conversely, when the input is scene features, the model "imagines" the corresponding music semantics.

[0256] This imaginative mechanism is not a simple reconstruction, but rather, through the constraints of the latent variable space, it enables the model to form an implicit cognitive structure of cross-modal semantic alignment during the learning process. Compared to direct multimodal alignment, the cross-modal imaginative mechanism can capture correspondences at the abstract level, such as emotional transfer and rhythmic resonance, thereby improving the model's understanding accuracy in the dimension of narrative emotional consistency.

[0257] This module implements bidirectional cross-modal reconstruction based on a conditional variational autoencoder.

[0258] 1) Audio-to-visual conditional variational autoencoder:

[0259] Encoder: for Average pooling is performed to obtain a 768-dimensional global scene representation, which is then mapped to Gaussian distribution parameters μ_scene and logσ²_scene through a multilayer perceptron with a dimension of 128.

[0260] Reparameterization: Sampling latent variables ;

[0261] Where ε is random noise sampled from a standard normal distribution;

[0262] Conditional decoder: converting 128-dimensional latent variables Projecting to 768 dimensions expands the query sequence to a length P. (Dimensions of (M+K)×768) are used for conditional memory, generated through a 4-layer Transformer decoder. The dimension is P×768.

[0263] 2) Visual-to-audio conditional variational autoencoder:

[0264] Encoder: for Average pooling is performed to obtain a 768-dimensional global music representation, which is then mapped to Gaussian distribution parameters using a multilayer perceptron. and The dimension is 128;

[0265] Reparameterization: Sampling latent variables ;

[0266] Conditional decoder: converting 128-dimensional latent variables Projecting to 768 dimensions expands the query sequence to a length of (M+K), so that... (Dimension P×768) is a conditional memory, generated through a 4-layer Transformer decoder. The dimension is (M+K)×768;

[0267] S304: Guided Cross-Attention Alignment Module

[0268] This module aligns musical narrative sequences with scene representations:

[0269] 1) From Extracting the macro-semantic part from the first M tokens The dimension is M×768;

[0270] 2) Use a linear projection layer to Projection as query vector The dimension is M×768;

[0271] 3) Use a linear projection layer to Projection as key vector Sum value vector All dimensions are P×768;

[0272] 4) Through a multi-head cross-attention mechanism with 12 attention heads, in order to For query, For key, For the value, calculate cross-modal attention;

[0273] 5) Perform residual connections and layer normalization on the attention output;

[0274] 6) Transformation is performed using a feedforward neural network, which contains a 3072-dimensional hidden layer and the GELU activation function;

[0275] 7) Perform residual connections and layer normalization again to output the aligned musical narrative sequence. The dimension is M×768.

[0276] S305: Evaluation Task Module

[0277] Design independent prediction branches for the three evaluation dimensions:

[0278] 1) Feature preparation:

[0279] Narrative-emotional consistency task: using The average value is calculated by weighting the features over M time steps (using a mask). The dimension is 768;

[0280] Topic Recognition and Originality Tasks: Using The average value is calculated by weighting the features of (M+K) time steps (using a mask). The dimension is 768;

[0281] Technology integration task: Concatenate three global features:

[0282] :equal The dimension is 768;

[0283] : Characteristics of audio tracks A weighted average is calculated with 768 dimensions.

[0284] diff: and The difference has a dimension of 768;

[0285] =[ ; [;diff], with dimensions of 768×3=2304.

[0286] 2) Score prediction head:

[0287] Each task uses an independent two-layer multilayer perceptron:

[0288] First layer: Linear layer, input dimension is task-specific dimension (narrative sentiment consistency and theme recognition and originality are 768, technology integration degree is 2304), output is 512-dimensional hidden layer;

[0289] Activation function: ReLU;

[0290] Dropout layer: probability is 0.1;

[0291] Second layer: Linear layer, outputting 1-dimensional fractions;

[0292] Activation function: Sigmoid, which restricts the output to the range of 0 to 1.

[0293] 3) Text generation module based on a large language model:

[0294] This module is used to generate structured video soundtrack aesthetic evaluation text based on multimodal features. Its implementation process is as follows:

[0295] (1) Input Construction

[0296] Construct the input sequence, including task-specific cue words and multimodal feature placeholders. <mm>The input sequence contains semantic guidance information corresponding to task features such as narrative emotional consistency, technological integration, theme recognition, and originality.

[0297] (2) Feature projection and fusion

[0298] Task-specific features are mapped to the embedding space of a large language model using a multilayer perceptron (MLP) projector.

[0299] Narrative Emotional Consistency Projector: 768→1536(GELU)→3584

[0300] Subject recognition and originality projector: 768→1536(GELU)→3584

[0301] Technology integration projector: 2304→3072(GELU)→3584

[0302] The projected features are concatenated with the video and audio features extracted by VideoLLaMA2 in the temporal dimension to form a fused embedding sequence, which is used as input for the text embedding of the language model.

[0303] (3) Teacher-mandated learning mechanism

[0304] The left-side text embeddings (system prompts and task prompts), the fused multimodal feature embeddings, and the target evaluation text embeddings are concatenated and fed into a large language model with frozen parameters. Only the parameters of the MLP projector are trained, while all parameters of the language model are frozen.

[0305] The gradient is calculated in the target text region using cross-entropy loss, and the labels are filled with −100 in the non-target text region.

[0306] (4) Training and optimization

[0307] The model is trained using a mixed-precision (bfloat16 / float16) and 4-bit quantization inference strategy, supporting gradient checkpointing and gradient accumulation to save GPU memory. The optimizer is AdamW with a learning rate of 5e-5.

[0308] (5) Generation stage

[0309] During inference, the model takes task features and multimodal features as input, and outputs natural language evaluation text through an autoregressive generation mechanism. The content covers a comprehensive analysis of the emotional fit of the background music, the degree of technical integration, and the originality of the theme.

[0310] Step S4: Two-stage training

[0311] Employing a pre-training fine-tuning strategy:

[0312] S4-01: Self-supervised pre-training

[0313] 1) Training data: All music video pairs were used, no annotations required;

[0314] 2) Loss function:

[0315]

[0316] in:

[0317] Mean squared error loss, a measure and Differences;

[0318] Mean squared error loss, a measure and Differences;

[0319] KL divergence loss: In regularized scenarios, the distribution of latent variables approximates a standard normal distribution.

[0320] KL divergence loss, regularized music latent variable distribution approximates standard normal distribution;

[0321] Attention alignment loss, guiding the distribution of attention across attention alignment modules;

[0322] Weight settings: =1.0, =0.01, =0.5;

[0323] 3) Optimizer: AdamW, learning rate 2e-4, warmup in the first 1000 steps;

[0324] 4) Train for 10 epochs until the pre-training loss converges.

[0325] S4-02: Fine-tuning under weak supervision

[0326] 1) Training data: A subset of data with weak labels;

[0327] 2) Model settings:

[0328] Load pre-trained encoder weights (segment-level audio-video fusion module, music coding tower, scene coding tower, guided cross-attention alignment module);

[0329] Randomly initialize the task headers (score prediction header and text generation header);

[0330] During the fine-tuning phase, freeze the parameters of the cross-modal imagination module and disable its loss term;

[0331] 3) Loss function:

[0332] For each task ∈ {narrative emotional consistency, technological integration, theme identification and originality}:

[0333]

[0334]

[0335] in Indicates whether a label is missing (handles the is_null flag). These are the normalized confidence weights;

[0336] The total loss is the weighted sum of the losses from each task.

[0337] The total loss function is the weighted sum of the losses from each task: .

[0338] Weighting coefficient settings: where , .

[0339] 4) Training settings:

[0340] Optimizer: AdamW, learning rate 5e-5;

[0341] Batch size: 16;

[0342] Train for 5 epochs until convergence.

[0343] Step S5: Reasoning and Application

[0344] For the new video soundtrack:

[0345] 1) Extract audio and video features according to step S2;

[0346] 2) Forward propagation through a well-trained multi-dimensional music aesthetics evaluation network;

[0347] 3) Output the normalized scores (range 0 to 1) for the three dimensions;

[0348] 4) Generate the corresponding evaluation text;

[0349] 5) Visualize and display the evaluation results ( Figure 3 This provides creators with suggestions for improvement.

[0350] Technical details

[0351] 1. Numerical stability guarantee mechanism

[0352] This invention employs several numerical stability assurance measures at key locations:

[0353] Layer normalization: Add layer normalization to all input layers (input to the segment-level audio-video fusion module, input to the music coding tower, and input to the scene coding tower) to normalize the features to a mean of 0 and a variance of 1.

[0354] Weight initialization: The weights of the linear projection layer are initialized using a truncated normal distribution with a standard deviation of 0.02, and the bias is initialized to 0;

[0355] Log-variance truncation: The log-variance of the conditional variational autoencoder is truncated and limited to the range of -10 to 10 to prevent overflow of the exp operation;

[0356] Masking: A small constant 1e-8 is added when calculating average pooling to prevent division by zero errors.

[0357] 2. Masking mechanism

[0358] This invention implements a complete masking mechanism and supports variable-length sequences:

[0359] Audio mask: Marks valid time steps of the OST (ost_clap_mask); Video mask: Marks valid video clips (clip_video_mask); Composite mask: The composite mask output by the music encoding tower contains the CLAP mask and thematic feature mask; Transformer mask conversion: Converts the boolean mask (True indicates valid) to Transformer format (True indicates that it should be ignored).

[0360] 3. Evaluation Task Module Design

[0361] The three evaluation task modules use different feature representations:

[0362] Narrative emotional consistency: Only scene-aligned musical narrative sequences are used, emphasizing the match between music and visuals; Theme identification and originality: Only the overall musical context is used to independently evaluate the artistic value of the music itself; Technical integration: The global features and differences between the soundtrack and the excerpt tracks are explicitly compared to evaluate the mixing quality.

[0363] This design avoids information leakage between tasks, allowing each task to focus on its evaluation objectives.

[0364] Through the above technical solution, this invention achieves automated, multi-dimensional, and interpretable evaluation of the aesthetic quality of video background music, with good accuracy and robustness.

[0365] It should be noted that the specific values ​​of various weight parameters (such as like weight, loss function weight coefficient, etc.) involved in the embodiments of this invention are determined by those skilled in the art based on conventional knowledge in the relevant technical field (such as the KL divergence weight of variational autoencoders often being set to a small value) and combined with conventional hyperparameter tuning (such as grid search or random search) performed on specific datasets. These values ​​aim to balance the influence between different tasks or signals to achieve optimal overall performance. Those skilled in the art can make appropriate adjustments to the above parameters according to their specific application scenarios and data distribution without creative effort.

[0366] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used in the operation of the aforementioned video soundtrack aesthetic quality evaluation method.

[0367] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0368] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the video soundtrack aesthetic quality evaluation method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by a processor.

[0369] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.< / mm>

Claims

1. A method for evaluating the aesthetics of video background music based on cross-attention mechanism and weakly supervised learning, characterized in that, The method includes the following steps: S1: Obtain the pre-collected original video audio, the corresponding set of video clips, and the weakly labeled dataset generated based on user comments; Step S1 includes: S101: Obtain a list of videos containing original audio and their corresponding video clips from a pre-prepared local video dataset; S102: Obtain a pre-collected dataset of user comments on video soundtracks; S103: Use a large language model to perform structured analysis on the comments and generate weak labels in three dimensions, namely narrative sentiment consistency, technological integration, and theme identification and originality; Narrative emotional consistency is used to evaluate the degree of matching between the emotional atmosphere of the background music and the narrative of the visuals; Technological integration is used to evaluate the degree of integration of sound elements; Theme identification and originality are used to evaluate the distinctiveness, memorability, and artistic innovation of a soundtrack; S104: For each dimension's weak labels, the final score and confidence level are calculated through like weighting, self-consistent sampling, and multi-model fusion. S105: Normalize the score range of the original labeled data from 1 to 10 to the range of 0 to 1; S106: Dataset construction, storing the collected background music, videos, and evaluation texts in a single folder; S2: Perform multi-level feature extraction on the audio and video data to construct a multimodal representation of the music scene; S3: Construct a multi-dimensional music aesthetics evaluation network, which includes a segment-level audio-video fusion module, a dual-tower encoder, a cross-modal imagination module, a guided cross-attention alignment module, a feature projector, and a multi-dimensional task head; The implementation of the cross-modal imagination module includes: S601: The encoder maps the input features to Gaussian distribution parameters; S602: Sampling latent variables through reparameterization techniques; S603: The conditional decoder uses the Transformer decoder, with latent variables as input queries and another modality feature as conditional memory, to generate imagined features of the target modality; S604: Calculate reconstruction loss and KL divergence loss; S4: A two-stage training strategy is adopted. First, the model learns the intrinsic correspondence between music and scene through self-supervised pre-training, and then the scoring criteria are learned through weak supervision fine-tuning. S5: Use the trained model to evaluate the aesthetic quality of new video soundtracks in three dimensions, and output the narrative emotional consistency score, technical integration score, theme recognition and originality score, and corresponding evaluation text.

2. The method according to claim 1, characterized in that, The feature extraction in step S2 includes: S201: Audio feature extraction, extracting macroscopic semantic sequences and low-level and mid-level features from original audio and audio clip tracks respectively; S202: Video feature extraction, extracting global semantic features and low-level visual features for each video segment; S203: Feature standardization and concatenation, concatenating global semantic features with flattened low-level features to obtain a standardized multimodal feature representation for each video segment; S204: Use the feature extractor of VideoLLaMA2 to extract features for use in subsequent text generation.

3. The method according to claim 1, characterized in that, The multi-dimensional music aesthetics evaluation network in step S3 includes: S301: Segment-level audio-visual fusion module, which uses a cross-attention mechanism to fuse the features of each video segment with the features of the corresponding audio track into a unified scene token; S302: Dual tower encoder, including music encoding tower and scene encoding tower, which respectively use multi-layer Transformer encoder to process audio temporal features and fused scene token set; S303: Cross-modal imagination module, which realizes bidirectional cross-modal reconstruction based on conditional variational autoencoder; S304: Guided cross-attention alignment module, which uses a multi-head cross-attention mechanism to align musical narrative sequences with scene representations; S305: Evaluation task module, which sets up independent prediction branches for the three evaluation dimensions. Each branch contains a score prediction head and a text generation head.

4. The method according to claim 3, characterized in that, The implementation of the segment-level audio-video fusion module includes: S401: Perform layer normalization on the input video features and audio features respectively; S402: Use a linear projection layer to project video features into a query vector and audio features into key and value vectors; S403: Calculate the dot product of the query vector and the key vector, and apply a scaling factor to obtain the attention score; S404: Apply the Softmax function to the attention score to obtain the attention weights, and multiply the attention weights by the value vector to obtain the fused features; S405: Perform residual connection and layer normalization on the fused features and query vector; S406: Further transformation is performed through a feedforward neural network; S407: After performing residual connections between the output and input of the feedforward neural network, layer normalization is performed again to obtain the final scene token.

5. The method according to claim 3, characterized in that, The implementation of the music encoding tower includes: S501: Perform layer normalization on the input macroscopic semantic features and traditional audio features respectively; S502: Uses a linear projection layer to project macroscopic semantic features into the model space; S503: Uses a one-dimensional convolutional neural network to process traditional audio features and outputs a fixed number of topic feature vectors; S504: Concatenate the projected macroscopic semantic features with the topic features output by the convolutional neural network in the time dimension; S505: Add positional encoding to the concatenated feature sequence; S506: Encoding is performed using a multi-layer Transformer encoder.

6. The method according to claim 3, characterized in that, The implementation of the guided cross-attention alignment module includes: S701: Extract the macroscopic semantic component from the feature sequence output by the music coding tower; S702: Use a linear projection layer to project macroscopic semantic features into query vectors and project the output of the scene coding tower into key vectors and value vectors; S703: Calculates cross-modal attention through a multi-head cross-attention mechanism; S704: Perform residual connections and layer normalization on the attention output; S705: Further transformation is performed through a feedforward neural network; S706: Perform residual connection and layer normalization again to output the aligned music narrative sequence.

7. The method according to claim 1, characterized in that, The two-stage training strategy in step S4 includes: S801: In the self-supervised pre-training phase, all music video pairs are used to learn the intrinsic correspondence between music and scenes through a cross-modal reconstruction task; S802: Weakly supervised fine-tuning phase, using a weakly labeled subset of data, freezing the parameters of the cross-modal imagination module and disabling its loss term during the fine-tuning phase, training the score prediction head and text generation head.

8. The method according to claim 7, characterized in that, The evaluation task module design for the fine-tuning phase includes: S901: Information preparation, preparing different feature representations for the three tasks respectively; S902: Score prediction, each task uses an independent multilayer perceptron to output a score; S903: A text generation module based on a large language model. It constructs an input sequence, including system prompts, task-specific prompts, and multimodal feature placeholders. It uses the embedding layer of the large language model to convert the text into embedding vectors. It embeds the task-specific prompts, merges the multimodal feature embeddings, and concatenates them with the target evaluation text embeddings. It trains the module through teacher-forced learning, calculating cross-entropy loss only on the target text segment. It freezes the language model parameters and optimizes only the feature projector parameters. S904: Loss calculation, using confidence-weighted task loss.

Images