AI-based digital collection background music recommendation method

Abstract

The invention discloses an AI-based digital collection background music recommendation method, and relates to the technical field of AI. The method comprises the following steps: acquiring a key frame average image, a dominant tone vector and a style label of a digital collection, and candidate music samples and audio signals of a music library; extracting dynamic features of the digital collections to generate rhythm representation vectors; processing the music audio to generate a rhythm structure vector, and calculating the rhythm similarity between the rhythm structure vector and the rhythm structure vector through a dynamic time warping algorithm; generating a digital collection visual style vector and a music style vector, and calculating a style consistency score by using a cosine similarity algorithm; an auxiliary music feature vector is generated through the digital collection visual style vector, and a guide score is calculated in combination with the bottom layer audio features; and calculating a comprehensive score index according to the rhythm similarity, the style consistency score and the guide score. According to the invention, background music recommendation is carried out on the digital collections through the comprehensive scoring index.

Description

Technical Field

[0001] This invention relates to the field of AI technology, specifically to an AI-based method for recommending background music for digital collectibles. Background Technology

[0002] In the field of digital collectibles, with the increasing number and variety of digital collectibles, how to match suitable background music for digital collectibles has become an important research direction.

[0003] However, traditional methods for matching background music for digital collectibles are often quite simple, relying only on some superficial attributes or simple manual settings. They fail to fully consider the rhythm and style of the digital collectibles and the rhythm and style of the music itself, resulting in low matching accuracy and an inability to provide a good user experience. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an AI-based method for recommending background music for digital collectibles, which solves the problem that traditional methods for matching background music for digital collectibles cannot fully consider factors such as the rhythm and style of the digital collectibles and the rhythm and style of the music itself.

[0005] To achieve the above objectives, the present invention provides a method for recommending background music for digital collections based on AI, comprising the following steps: Step S1: Collect digital collection content data, which includes keyframe average image, dominant color vector, and comprehensive style tag; collect music library attribute data, which includes candidate music samples and music audio signals. Step S2: Extract features from the digital collection content data to obtain the dynamic features of the digital collection. Generate a time-varying signal sequence based on the dynamic features of the digital collection. Process the time-varying signal sequence using a sliding time window method to obtain the rhythm distribution spectrum. Input the rhythm distribution spectrum into the spatiotemporal convolutional residual network TCResNet model to obtain the rhythm representation vector of the digital collection. Step S3: Perform Fourier transform on the music audio signal to obtain the music frequency domain. Based on the music frequency domain, generate the music Mel spectrogram through the Mel filter bank. Input the music Mel spectrogram into the BeatTrackNetAI model to output the music beat time series. Generate the music rhythm structure vector according to the music beat time series. Step S4: Apply the Dynamic Time Warping (DWT) algorithm to the rhythm representation vector of the digital collection and the rhythm structure vector of the music to calculate the rhythm similarity. Step S5: Concatenate the keyframe average image, dominant color vector, and style tag of the digital collection according to the channel dimension to form a tensor. Input the tensor into the StyleAttNet network model to output the visual style vector of the digital collection. Step S6: Input the music Mel spectrogram into the audio feature extraction VGGish model and output the music style vector. Apply the cosine similarity algorithm to the visual style vector and music style vector of the digital collection to obtain the style consistency score. Step S7: Input the visual style vector of the digital collection into the music style generation model to generate an auxiliary music feature vector, extract the underlying audio feature vector from the candidate music samples, and calculate the guidance score based on the auxiliary music feature vector and the underlying audio feature vector. Step S8: Calculate the comprehensive score index based on the rhythm similarity, style consistency score, and guidance score, and recommend background music for the digital collection based on the comprehensive score index.

[0006] Preferably, the digital collection content data includes keyframe average images, dominant color vectors, and comprehensive style tags, including: Keyframe Averaging Image Generation: Time-Series Frames of Digital Collections Calculate keyframes The average image of keyframes is obtained by taking the pixel mean of 0. ,formula:

[0007] in Number of keyframes For pixel coordinates, For color channels, It is the first Frame image in Coordinates, color channels The pixel value below; Keyframe annotation: Keyframe locations are marked through a combination of manual annotation and algorithmic detection. Each element in the set The attribute representing a frame, when When, the frame is a mutation frame; when At this time, it is a gradient frame; in this way, the set K can reflect the rhythmic abrupt change point of the content and clearly distinguish the type of image change; Art style tags: A collection of style tags generated by the creators of the collections or by AI classification models. Examples such as “vaporwave”, “minimalism”, and “cyberpunk” are converted into initial tag embeddings using Word2Vec; The process of embedding art style tags involves collecting art style tags. Input the pre-trained Word2Vec model to generate the embedding vector for each label. The overall style tag is obtained through average pooling:

[0008] in, It is the embedding vector of each label, and M is the number of labels; Dominant color extraction: Calculate the RGB mean of pixels for each frame of the image, and take the principal components of all frames as the dominant color vector. .

[0009] Preferably, the step of extracting features from the digital collection content data to obtain dynamic features of the digital collection includes: Inter-frame variation frequency: Obtain the inter-frame variation frequency The formula is as follows:

[0010] in, Denotes the Euclidean norm. It is width multiplied. It's about height. It is each frame of the image. It is each frame of the image Previous frame; Keyframe information: Annotate keyframes As a feature, the mutation frame Gradient frames ; The above two features are combined to form the dynamic features of digital collections. And combine all the time vectors into a time-varying signal sequence. N is the number of sequences.

[0011] Preferably, the step of processing the time-varying signal sequence using a sliding time window method to obtain the rhythm distribution spectrum includes: A rhythm distribution spectrum is constructed by processing time-varying signals using a sliding time window method. Select a fixed-length time window and step size, and apply it to a time-varying signal sequence. Slide on top; For each time window Calculate the following two features: Beat density: The proportion of keyframes within a time window, i.e., beat density. The formula is as follows:

[0012] in, It's a time window. It is a keyframe annotation; Periodicity of change: the frequency of inter-frame changes within a time window Perform a Fast Fourier Transform to find the principal components in the spectrum, i.e., the periodicity of change. The formula is as follows:

[0013] in, Indicates time window The inter-frame variation frequency sequence within the frame. Its function is to find the frequency value corresponding to the maximum value in the result of the Fast Fourier Transform; Combining the beat density and periodicity of change in each time window into a vector yields the rhythm distribution spectrum:

[0014] Where T represents the number of time windows, it is appropriate to use parentheses to combine the beat density and periodicity of change within the same time window. This clearly indicates that the two parameters corresponding to each time window are a set of data, avoiding confusion.

[0015] Preferably, generating a music rhythm structure vector based on the music beat time sequence includes: Assuming each candidate music fragment needs to contain A series of consecutive shooting points, shooting point time series For the first Each segment, whose start and end times can be determined by the snapshot sequence index:

[0016] in, Each candidate music clip must contain consecutive beats. This formula shows that, and Directly from the time stamp sequence of the shooting point The corresponding elements are determined by controlling the number of beats contained in each segment. Implement beat-based time segmentation to ensure that the start and end times of the segment are aligned with the actual beat positions of the music, thus providing a basis for subsequent energy averaging. The calculations provide an accurate time range; Rhythm structure vector generate: To align with the rhythm vector of digital collections Alignment, dividing the music into equal-length segments. For each segment, the average energy value of the segment is calculated as a musical rhythmic feature:

[0017] in, and For the first The start and end times of each segment; Let dt be the duration of the segment, and dt be the differential symbol in the integral, representing a small increment on the time axis. It is the audio signal of the candidate music sample; Combine the average energy of all segments into a musical rhythmic structure vector: .

[0018] Preferably, the rhythmic similarity is calculated by applying the Dynamic Time Warping (DWT) algorithm to the rhythmic representation vector of the digital collection and the musical rhythmic structure vector, including: Dynamic Time Warping (DTW) rhythm matching: The core logic of the DTW algorithm is an algorithm used to align two time series, allowing for local time warping to resolve differences in tempo between digital artifacts and music, such as matching fast-paced segments of the artifact with accelerated sections of music. Specific steps include: Distance matrix construction: calculation and The Euclidean distance between each element pair in the equation forms Distance matrix ,in ; Path planning: finding the path from arrive The optimal path that satisfies local constraints and minimizes the total path distance; Similarity calculation with offset tolerance factor Introducing offset tolerance factor The allowed time offset range in path planning is expanded to allow ±30% beat offset, enhancing robustness to micro-rhythmic misalignment. The final rhythmic similarity is defined as:

[0019] Among them, DTW ( , , () represents the DTW distance with offset constraints; the smaller the value, the higher the rhythm matching degree. The distance is converted into a similarity score through normalization.

[0020] Preferably, the step of applying a cosine similarity algorithm to the visual style vector and musical style vector of the digital collection to obtain a style consistency score includes: Cosine similarity calculation: Calculating the visual style vector of digital collections With musical style vector The cosine similarity and style consistency scoring formula is as follows:

[0021] in, It is a style consistency score. and These are the visual style vector and the musical style vector of the digital collection. Physical meaning: They measure the semantic matching degree between the artistic style of the collection and the inherent style of the music. The higher the value, the more consistent the style labels are.

[0022] Preferably, the step of calculating the guided score based on the auxiliary music feature vector and the underlying audio feature vector includes: Generate guided score calculation: calculate and Cosine similarity:

[0023] in, It is a guide to rating. It is an auxiliary music feature vector. It is the underlying audio feature vector. The guidance score measures the degree of matching between the target audio features "guided" by the generative model according to the visual style and the actual features of the candidate music. The higher the value, the more consistent the granular style features.

[0024] Preferably, the step of obtaining a comprehensive scoring index based on the rhythm similarity, style consistency score, and guiding score includes: The weights are determined based on factors such as the specific type and characteristics of the digital collectibles and potential user preferences. , , Meanwhile, determine the nonlinear parameters. , , ; If a digital artifact is a still image in the style of a classical oil painting, the focus is on matching the artistic style, then it can be set... , (Emphasizing stylistic consistency), Regarding nonlinear parameters, , This makes the style consistency score contribute more to the overall score. ; For each candidate music Based on the set weights and nonlinear parameters, the comprehensive score is calculated according to the following nonlinear formula:

[0025] in, It is a comprehensive score index. , , These correspond to the weights of rhythm similarity, style consistency score, and guiding score, respectively. , , This is a nonlinear parameter.

[0026] Preferably, the step of recommending background music for digital collectibles based on the comprehensive scoring index includes: All candidate music were ranked according to their overall score. Sort the music from highest to lowest quality, and select the top-ranked tracks based on actual needs, as shown above. head, The value can be set to 5, 10, etc., depending on the specific situation, and provided to the user as a recommendation result; If there are 20 candidate songs, calculate their overall scores and then rank them. ,set up Then it will be recommended first. , , , , These five pieces of music serve as background music to match the digital collection.

[0027] Beneficial effects This invention provides an AI-based method for recommending background music for digital collections, which involves reinforcement learning technology and has the following beneficial effects: (1) The AI-based digital collection background music recommendation method extracts dynamic features from the digital collection content data and further generates time-varying signals and rhythm distribution spectra. At the same time, it performs Fourier transform and Mel spectrogram generation on the music, comprehensively considering the multi-dimensional features of digital collections and music in rhythm and frequency domain, making the analysis of both more in-depth and accurate.

[0028] (2) The AI-based digital collection background music recommendation method uses the spatiotemporal convolutional residual network TCResNet model to process the rhythm-related data of digital collections, the beat tracking network BeatTrackNetAI model to process the music rhythm structure, and the StyleAttNet style attention network model and VGGish audio feature extraction model to process the visual style and music style of digital collections respectively. The use of these advanced models improves the efficiency and accuracy of feature extraction and analysis, and provides a reliable foundation for subsequent matching calculations.

[0029] (3) By inputting the visual style vector of digital collections into the music style generation model to generate auxiliary music feature vectors, and combining them with the underlying audio feature vectors to calculate the guiding score, this method can guide the music features according to the visual style of digital collections, further enhancing the correlation and matching degree between digital collections and background music, providing users with background music recommendations that are more in line with the characteristics of digital collections, and improving the user experience. Attached Figure Description

[0030] Figure 1 This is a flowchart of an AI-based background music recommendation method for digital collectibles proposed in this invention.

[0031] Figure 2 This is a flowchart illustrating the process of obtaining a guided score for an AI-based digital collection background music recommendation method proposed in this invention.

[0032] Figure 3 This invention provides a hierarchical graph of the comprehensive scoring index for an AI-based digital collection background music recommendation method. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Please see Figure 1 This invention provides a technical solution: an AI-based method for recommending background music for digital collectibles. Specifically, the following AI-based method for recommending background music for digital collectibles is provided; please refer to [link / reference]. Figure 1 The method includes the following steps: Step S1: Collect digital collection content data, which includes keyframe average image, dominant color vector, and style tag; collect music library attribute data, which includes candidate music samples and music audio signals.

[0035] Digital collection content collection: Dynamic time-series data acquisition: For collections of animation and sequence frame images, decompose them into time-series frame sequences according to time order. Each frame For RGB images with uniform resolution (e.g., 1024×1024 pixels).

[0036] Keyframe Averaging Image Generation: Time-Series Frames of Digital Collections Calculate keyframes The average image of keyframes is obtained by taking the pixel mean of 0. ,formula:

[0037] in Number of keyframes For pixel coordinates, For color channels, It is the first Frame image in Coordinates, color channels The pixel value below; Keyframe annotation: Keyframe locations are marked through a combination of manual annotation and algorithmic detection. Each element in the set The attribute representing a frame, when When, the frame is a mutation frame; when At this time, it is a gradient frame; in this way, the set K can reflect the rhythmic abrupt change point of the content and clearly distinguish the type of image change; Art style tags: A collection of style tags generated by the creator of the collection or an AI classification model (such as CLIP). (e.g., "vaporwave", "minimalism", "cyberpunk"), and converted into initial tag embeddings via Word2Vec.

[0038] The process of embedding art style tags involves collecting art style tags. Input the pre-trained Word2Vec model to generate the embedding vector for each label. The overall style tag is obtained through average pooling:

[0039] in, It is the embedding vector of each label, and M is the number of labels; Dominant color extraction: Calculate the RGB mean of pixels for each frame of the image, and take the principal components of all frames as the dominant color vector. (For example, cool colors have lower red values, while warm colors have higher red values).

[0040] Multi-dimensional attribute collection of music library: Basic audio information: Collect candidate music samples (WAV / MP3 format, 44.1kHz sampling rate) and extract them using an audio processing library (such as Librosa). audio signal ; Beat labeling: Generating beat time series using the YIN beat detection algorithm. (Unit: seconds), marking the starting position of each beat.

[0041] Segmented Energy: The audio is sliced ​​into 0.5-second segments, and the average temporal energy of each segment is calculated for subsequent rhythmic structure modeling.

[0042] Style and copyright information: Style tags: Manually labeled or automatically classified by a pre-trained model (such as MusicCNN) to generate style tags (such as “electronic”, “classical”, and “rock”) and emotion tags (such as “exhilarating”, “soothing”, and “mysterious”), and then converted into one-hot encoded vectors.

[0043] Data standardization: Unified time scale: Time alignment of digital collection frame sequences with music beat sequences, establishing a timeline in milliseconds to ensure temporal consistency when matching rhythms in the future.

[0044] Outlier handling: Filter image frames with abnormal resolution, music clips that are too short (<10 seconds), and "ghost user" data with no interaction records to ensure dataset quality.

[0045] Two standardized feature libraries were formed: Digital collection feature library: includes time-series frame data, keyframe annotations, style and tone features; Music feature library: includes audio features, beat sequences, and style tags; Step S2: Extract features from the digital collection content dataset to obtain dynamic features of the digital collection. Generate a time-varying signal based on the dynamic features of the digital collection. Process the time-varying signal using a sliding time window method to obtain a rhythm distribution spectrum. Input the rhythm distribution spectrum into the spatiotemporal convolutional residual network TCResNet model to obtain the rhythm representation vector of the digital collection.

[0046] To capture the dynamic changes in digital exhibits over time, it is necessary to calculate the relevant features of each frame of the image and generate a time-varying signal.

[0047] Inter-frame variation frequency: Obtain the inter-frame variation frequency The formula is as follows:

[0048] in, Denotes the Euclidean norm. It is width multiplied. It's about height. It is each frame of the image. It is each frame of the image Previous frame.

[0049] Keyframe information: Annotate keyframes As a feature, the mutation frame Gradient frames .

[0050] The above two features are combined to form the dynamic features of digital collections. And combine all the time vectors into a time-varying signal sequence. N is the number of sequences.

[0051] A rhythm distribution spectrum is constructed by processing time-varying signals using a sliding time window method. Select a time window of fixed length (window length) ) and step size (step size) ), in time-varying signal sequences Slide it up.

[0052] For each time window Calculate the following two features: Beat density: The proportion of keyframes within a time window, i.e., beat density. The formula is as follows:

[0053] in, It is the window length of the time window. It is a keyframe annotation; Periodicity of change: the frequency of inter-frame changes within a time window Perform a Fast Fourier Transform (FFT) to find the principal components in the spectrum, i.e., the periodicity of change. The formula is as follows:

[0054] in, Indicates time window The inter-frame variation frequency sequence within the frame. Its function is to find the frequency value corresponding to the maximum value in the result of the Fast Fourier Transform.

[0055] Combining the beat density and periodicity of change in each time window into a vector yields the rhythm distribution spectrum:

[0056] Where T represents the number of time windows, it is appropriate to use parentheses to combine the beat density and periodicity of change within the same time window. This clearly indicates that the two parameters corresponding to each time window are a set of data, avoiding confusion.

[0057] rhythm distribution The input is fed into the TCResNet model, where features are extracted and encoded through the model's convolutional residual blocks to obtain the rhythm representation vector of the digital collection. The formula is as follows:

[0058] Among them, TCResNet (Temporal Convolutional Residual Network) is a deep learning model for processing time series data and is used to extract dynamic rhythm features of digital collections.

[0059] The following is information about the Temporal Convolutional Residual Network (TCResNet) model: Input: Input dimension is ,in 2 represents the length of the time series, and 2 represents the feature dimension. In this step, the input is the rhythm distribution spectrum obtained through time series analysis of digital collections.

[0060] Output: The output dimension is , It is the rhythm feature dimension, and the output is a rhythm representation vector of the digital collection, which contains the rhythm feature information of the digital collection learned by the model.

[0061] Model structure: It consists of multiple layers of convolutions and residual blocks. The specific components of each convolutional layer and residual block are as follows: Two-dimensional convolutional layers: employ dilated convolution (dilation rate of 2) and padding (padding value of 1). Dilated convolution expands the receptive field, enabling the model to capture a wider range of temporal dependencies; padding ensures that the length of the data remains unchanged before and after convolution.

[0062] Batch normalization layer: Normalizes the output of the convolutional layer, which can accelerate the convergence speed of the model and improve the stability of the model.

[0063] ReLU activation function: Introduces non-linearity to enhance the model's expressive power, enabling the model to learn more complex features.

[0064] Residual connection (SkipConnection): The input is directly added to the output of the convolutional layer, allowing the model to learn the residual information between the input and output. This helps to preserve important temporal features and alleviate the gradient vanishing problem in deep networks.

[0065] Step S3: Perform a Fourier transform on the music audio signal to obtain the music frequency domain, then generate a music Mel spectrogram through a Mel filter bank, input the music Mel spectrogram into the BeatTrackNetAI model, and output the music beat time series. Based on the music beat time series, generate a music rhythm structure vector.

[0066] Audio beat sequence extraction: Candidate music audio signal Please follow these steps: Mel spectrogram generation: The music audio signal is converted into the music frequency domain using a short-time Fourier transform (STFT, window length 1024, step size 512), and then the Mel spectrogram is generated using a Mel filter bank (40 filters). .

[0067] Mel filter bank processing: Filter Design: Design A triangular Mel filter, evenly distributed on the Mel scale, is then used to transform the filter boundaries back to linear frequencies. Each filter... It has a triangular shape in the frequency domain, with the center frequency corresponding to a Mel scale point.

[0068] With time frames as the horizontal axis and Mel frequency channels as the vertical axis, a two-dimensional image is formed, namely the Mel spectrogram of music.

[0069] Point detection: Will Input the BeatTrackNet model as input and output the time series of music beats. ,For example , indicating the start time of each beat.

[0070] Among them, the BeatTrackNet beat tracking network model is a beat detection model based on the CRNN (Convolutional Recurrent Neural Network) architecture, used to accurately extract the time series of beats from audio signals. It combines the local feature extraction capability of the Convolutional Neural Network (CNN) with the temporal dependency modeling capability of the Recurrent Neural Network (RNN), making it suitable for the analysis of complex rhythm patterns.

[0071] Input: Mel spectrogram of the converted audio signal, with dimensions of M×N (40 Mel bands, 96 time frames), generated by the Librosa library, which can effectively preserve the rhythm and timbre information of the audio.

[0072] Output: Time sequence of shooting points B={b1, b2, ..., bQ} (unit: seconds), positioning accuracy error ±50ms, meeting the timing requirements of rhythm alignment.

[0073] Model structure: CNN layer: 2-dimensional convolutional layers (kernel size 3×3, stride 2) to extract local rhythmic features from the spectrogram; LSTM layer: 1-layer bidirectional LSTM (128 hidden units) to capture the rhythmic dependencies of time series; Fully connected layer: Outputs the probability distribution of the beat points, and generates the final beat point sequence by filtering through a threshold.

[0074] Assuming each candidate music fragment needs to contain A series of consecutive shooting points, shooting point time series For the first A segment ( , , (This indicates rounding down), and its start and end times can be determined by the beat sequence index:

[0075] in, Each candidate music clip must contain consecutive beats. This formula shows that, and Directly from the time stamp sequence of the shooting point The corresponding element in is determined by control. (The number of beats contained in each segment) enables beat-based time segmentation, ensuring that the start and end times of the segment are aligned with the actual beat positions of the music, thus providing an average energy value for subsequent segments. The calculations provide an accurate time range.

[0076] Rhythm structure vector generate: To align with the rhythm vector of digital collections Alignment, dividing the music into equal-length segments. For each segment, the average energy value of the segment is calculated as a musical rhythmic feature:

[0077] in, and For the first The start and end times of each segment; Let dt be the duration of the segment (consistent with the time window length of digital collections), where dt is the differential symbol in integrals, representing a small increment on the time axis. It is the audio signal of the candidate music sample.

[0078] Combine the average energy of all segments into a musical rhythmic structure vector: .

[0079] Step S4: Apply the Dynamic Time Warping (DWT) algorithm to the rhythm representation vector of the digital collection and the rhythm structure vector of the music to calculate the rhythm similarity.

[0080] Dynamic Time Warping (DTW) rhythm matching: The core logic of the DTW algorithm is an algorithm used to align two time series, allowing for local time warping to resolve differences in tempo between digital artifacts and music (e.g., matching a fast-paced section of an artifact with an accelerated section of music). Specific steps include: Distance matrix construction: calculation and The Euclidean distance between each element pair in the equation forms Distance matrix ,in .

[0081] Path planning: finding the path from arrive Find the optimal path that satisfies local constraints (allowing the current point to move only from the left, top, and upper left directions) and minimizes the total path distance.

[0082] Similarity calculation with offset tolerance factor Introducing offset tolerance factor (Default value 0.3) This expands the allowed time offset range in path planning (e.g., allowing ±30% beat offset) to enhance robustness against micro-rhythmic misalignments. The final rhythmic similarity is defined as:

[0083] Among them, DTW ( , , () represents the DTW distance with offset constraints; the smaller the value, the higher the rhythm matching degree. The distance is converted into a similarity score through normalization.

[0084] Step S5: Concatenate the keyframe average image, dominant color vector, and style tag of the digital artifact along the channel dimension to form a tensor. Input the tensor into the StyleAttNet model to output the visual style vector of the digital artifact.

[0085] Input tensor construction: averaging keyframe images Dominant color vector Style tags Concatenate the data along the channel dimension to form the model input tensor. This ensures the integration of visual and semantic information.

[0086] StyleAttNet Model Instructions: StyleAttNet is a neural network designed specifically for cross-modal style extraction. It generates vectors representing the artistic style of digital collections by fusing visual image features with semantic labels. Its core architecture combines the visual feature extraction capabilities of convolutional neural networks (CNNs) with the semantic weighting capabilities of attention mechanisms, ensuring that style vectors simultaneously contain visual details and abstract stylistic semantics.

[0087] Input: tensor , dimension .

[0088] Output: Visual style vector The dimensions and art style tags are embedded in a consistent manner, representing the comprehensive art style characteristics of the collection (such as the mechanical feel and high contrast of "cyberpunk").

[0089] Model structure: Convolutional backbone network: The first 10 layers of ResNet50 are used to extract visual features from the average keyframe image. It captures underlying visual information such as color, texture, and composition.

[0090] Multimodal fusion layer: integrates visual features Flattened , with the main color vector Style tag embedding splicing together to form a fusion feature .

[0091] Attention mechanism module: Calculates the association weights between style tags and visual features. The formula is:

[0092] in, For the first The embedding vectors of each style tag are weighted and summed to generate the final style vector. .

[0093] Obtain the visual style vector of digital collections This vector integrates the visual details of the collection (such as primary color and texture) with abstract stylistic semantics (such as art genre tags), providing core input for subsequent style dimension scoring and ensuring that the recommended music is highly consistent with the digital collection in terms of art style.

[0094] Step S6: Input the music Mel spectrogram into the audio feature extraction VGGish model to obtain the music style vector. Apply the cosine similarity algorithm to the visual style vector and music style vector of the digital collection to obtain the style consistency score.

[0095] Short-Time Fourier Transform (STFT): for audio signals A short-time Fourier transform is performed with parameters set to a window length of 1024 and a step size of 512. The time-domain audio signal is then converted to a frequency-domain representation using an STFT to obtain the power spectrum.

[0096] Mel filter bank processing: The power spectrum is processed using 40 Mel filter banks, converting it from a linear frequency scale to a Mel frequency scale to generate a Mel spectrum. Mel frequencies better match the characteristics of human auditory perception and can better preserve the perceptual features of audio.

[0097] Logarithmic compression: Taking the logarithm of the amplitude values ​​of the Mel spectrum compresses the dynamic range. The formula is as follows: ( (This is a very small constant, to avoid the problem of zero values ​​in logarithmic operations).

[0098] Through the above steps, the final dimension is obtained as follows: Mel spectrum It can be input into the VGGish model for further processing.

[0099] VGhish Model User Guide: VGhish is a pre-trained neural network designed specifically for audio feature extraction. Based on a 16-layer convolutional architecture, it can extract high-level semantic features from audio signals, effectively representing the inherent style of music (such as classical, electronic, jazz, etc.).

[0100] Input: Mel spectrogram of the converted audio signal, with dimensions of [dimension number missing]. (40 Mel bands, 96 time frames).

[0101] The generation steps are as follows: For audio signals Perform a short-time Fourier transform (STFT) with a window length of 1024 and a step size of 512 to obtain the power spectrum; The power spectrum is converted into a Mel spectrum using a 40-mel filter bank, and then the dynamic range is compressed by taking the logarithm.

[0102] Output: 128-dimensional music style embedding vector This vector contains semantic information about the musical style (such as genre, rhythm, and mood). Model structure: 16 convolutional layers (kernel size 3×3, stride 1 or 2) extract abstract features from low frequency to high frequency layer by layer; Global average pooling layer: compresses the features of the last convolutional layer into a vector of fixed dimensions; Output layer: No activation function, directly outputs a 128-dimensional feature vector.

[0103] Candidate music The audio signal is used to generate a Mel spectrogram using the method described above. Inputting the VGGish model yields the inherent style embeddings: .

[0104] Cosine similarity calculation: Calculating the visual style vector of digital collections With musical style vector The cosine similarity and style consistency scoring formula is as follows:

[0105] in, It is a style consistency score. and These are the visual style vector and the musical style vector of the digital collection. Physically, they measure the semantic matching degree between the artistic style of the collection (such as the simple composition of "minimalism") and the inherent musical style (such as the soothing rhythm of "ambient music"). The higher the value, the more consistent the style labels are.

[0106] Step S7: Input the visual style vector of the digital collection into the music style generation model to generate an auxiliary music feature vector, extract the underlying audio feature vector from the music sample, and calculate the guidance score based on the auxiliary music feature vector and the underlying audio feature vector.

[0107] Auxiliary music feature generation: Will Input the music style generation model and generate auxiliary features that match the visual style:

[0108] For example, if Characterized by the "cyberpunk" style, It may contain features such as electronic synthesizer timbre, high-frequency noise, and fast syncopation.

[0109] Style2AudioGAN is a cross-modal style transfer model based on generative adversarial networks (GANs) that can convert the visual style of digital collections into corresponding audio style features.

[0110] Input: Visual style vector of digital collections (Step S4 output,) ).

[0111] Output: Auxiliary music feature vector ( This includes fine-grained audio features (such as timbre, melody direction, and rhythmic patterns) that match the visual style.

[0112] Music style generation model structure: Generator G: Input layer: Mapped to a hidden layer of dimension 512; Transposed convolutional layers: 3 layers of transposed convolutions (kernel size 4×4, stride 2), progressively upsampling to generate 256-dimensional audio style features. ; Activation functions: ReLU (first two layers), Tanh (output layer, normalizes features to [-1, 1]).

[0113] Discriminator D: Convolutional layers: 3 convolutional layers (kernel size 4×4, stride 2), used to determine if the input features are generated. Or are they real audio characteristics? ; Output layer: sigmoid activation function, outputs true / false probabilities (used for model training, not used in the inference stage).

[0114] Candidate music audio feature extraction: Extracting underlying audio features from candidate music : MFCC Feature Extraction: Thirteenth-order Mel-frequency cepstral coefficients (MFCCs) are selected. Their first-order difference reflects the rate of change of audio features over time, and their second-order difference reflects the trend of this rate of change. The three are combined (total...). (Dimension) can more comprehensively characterize the time and frequency domain characteristics of audio.

[0115] PCA dimensionality reduction: Principal component analysis (PCA) reduces the dimensionality of 39 features to 256 dimensions by linear transformation while preserving the main features of the data.

[0116] The final result It is a compact feature representation that integrates underlying information such as audio spectrum envelope and rhythm changes, providing a basis for measuring the matching degree between candidate music and the visual guidance features of digital collections.

[0117] Generate guided score calculation: calculate and Cosine similarity:

[0118] in, It is a guide to rating. It is an auxiliary music feature vector. It is the underlying audio feature vector. The guidance score measures the matching degree between the target audio features "guided" by the generative model according to the visual style and the actual features of the candidate music. The higher the value, the more consistent the granular style features (such as the high contrast visual of the collection corresponding to the frequent switching of the intensity of the music).

[0119] Step S8: Based on the rhythm similarity, style consistency score, and guidance score, a comprehensive score index is obtained, and background music is recommended for digital collections based on the comprehensive score index.

[0120] The weights are determined based on factors such as the specific type and characteristics of the digital collectibles and potential user preferences. , , (satisfy ), and simultaneously determine the nonlinear parameters , , .

[0121] If a digital collection is a still image in the style of a classical oil painting, the focus is on matching the artistic style. Then you can set... , (Emphasizing stylistic consistency), Regarding nonlinear parameters, , This makes the style consistency score contribute more to the overall score. .

[0122] For each candidate music Based on the set weights and nonlinear parameters, the comprehensive score is calculated according to the following nonlinear formula:

[0123] in, It is a comprehensive score index. , , These correspond to the weights of rhythm similarity, style consistency score, and guiding score, respectively. , , This is a nonlinear parameter.

[0124] All candidate music were ranked according to their overall score. Sort the music from highest to lowest. Based on actual needs, select several of the top-ranked tracks (as shown above). head, The value can be set to 5, 10, etc., depending on the specific situation, and provided to the user as a recommendation result.

[0125] For example, if there are 20 candidate songs, their overall scores are calculated and then ranked. ,set up Then it will be recommended first. , , , , These five pieces of music serve as background music to match the digital collection.

[0126] This invention collects digital artifact content data, including keyframe average images, dominant color vectors, and comprehensive style tags; it also collects music library attribute data, including candidate music samples and music audio signals; it extracts features from the digital artifact content data to obtain dynamic features of the digital artifacts, generates time-varying signal sequences based on these dynamic features, processes these time-varying signal sequences using a sliding time window method to obtain a rhythm distribution spectrum, inputs this rhythm distribution spectrum into a spatiotemporal convolutional residual network (TCResNet) model to obtain a rhythm representation vector for the digital artifacts; it performs a Fourier transform on the music audio signals to obtain the music frequency domain, generates a music Mel spectrogram based on this music frequency domain using a Mel filter bank, inputs the music Mel spectrogram into a beat tracking network (BeatTrackNetAI) model to output a music beat time sequence, and generates a music rhythm structure vector based on this music beat time sequence; and it then processes the rhythm of the digital artifacts... The rhythm similarity is calculated using the Dynamic Time Warping (DWT) algorithm for both the visual style vector and the music rhythm structure vector. The keyframe average image, dominant color vector, and style tag of the digital artifact are concatenated along the channel dimension to form a tensor. This tensor is then input into the StyleAttNet network model to output the visual style vector of the digital artifact. The Mel spectrogram of the music is input into the VGGish audio feature extraction model to output the music style vector. A cosine similarity algorithm is applied to the visual style vector and music style vector of the digital artifact to obtain a style consistency score. The visual style vector of the digital artifact is then input into a music style generation model to generate an auxiliary music feature vector. A low-level audio feature vector is extracted from candidate music samples. Based on the auxiliary music feature vector and the low-level audio feature vector, a guidance score is calculated. Finally, a comprehensive score index is calculated based on the rhythm similarity, style consistency score, and guidance score. This comprehensive score index is used to recommend background music for the digital artifact.

[0127] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0128] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their likenesses.

Claims

1. A method for recommending background music for digital collectibles based on AI, characterized in that, The process includes the following steps: Step S1: Collect digital collection content data, which includes keyframe average image, dominant color vector, and comprehensive style tag; collect music library attribute data, which includes candidate music samples and music audio signals. Step S2: Extract features from the digital collection content data to obtain the dynamic features of the digital collection. Generate a time-varying signal sequence based on the dynamic features of the digital collection. Process the time-varying signal sequence using a sliding time window method to obtain the rhythm distribution spectrum. Input the rhythm distribution spectrum into the spatiotemporal convolutional residual network TCResNet model to obtain the rhythm representation vector of the digital collection. Step S3: Perform Fourier transform on the music audio signal to obtain the music frequency domain. Based on the music frequency domain, generate the music Mel spectrogram through the Mel filter bank. Input the music Mel spectrogram into the BeatTrackNetAI model to output the music beat time series. Generate the music rhythm structure vector according to the music beat time series. Step S4: Apply the Dynamic Time Warping (DWT) algorithm to the rhythm representation vector of the digital collection and the rhythm structure vector of the music to calculate the rhythm similarity. Step S5: Concatenate the keyframe average image, dominant color vector, and style tag of the digital collection according to the channel dimension to form a tensor. Input the tensor into the StyleAttNet network model to output the visual style vector of the digital collection. Step S6: Input the music Mel spectrogram into the audio feature extraction VGGish model and output the music style vector. Apply the cosine similarity algorithm to the visual style vector and music style vector of the digital collection to obtain the style consistency score. Step S7: Input the visual style vector of the digital collection into the music style generation model to generate an auxiliary music feature vector, extract the underlying audio feature vector from the candidate music samples, and calculate the guidance score based on the auxiliary music feature vector and the underlying audio feature vector. Step S8: Calculate the comprehensive score index based on the rhythm similarity, style consistency score, and guidance score, and recommend background music for the digital collection based on the comprehensive score index.

2. The AI-based digital collection background music recommendation method according to claim 1, characterized in that, The digital collection content data includes keyframe average images, dominant color vectors, and comprehensive style tags, including: Keyframe Averaging Image Generation: Time-Series Frames of Digital Collections Calculate keyframes The average image of keyframes is obtained by taking the pixel mean of 0. ,formula: ； in This refers to the number of keyframes. For pixel coordinates, For color channels, It is the first Frame image in Coordinates, color channels The pixel value below; Keyframe annotation: Keyframe locations are marked through a combination of manual annotation and algorithmic detection. Each element in the set The attribute representing a frame, when When, the frame is a mutation frame; when At this time, it is a gradient frame; in this way, the set K can reflect the rhythmic abrupt change point of the content and clearly distinguish the type of image change; Art style tags: A collection of style tags generated by the creators of the collections or by AI classification models. Examples such as "vaporwave," "minimalism," and "cyberpunk" are converted into initial tag embeddings using Word2Vec; The process of embedding art style tags involves collecting art style tags. Input the pre-trained Word2Vec model to generate the embedding vector for each label. The overall style tag is obtained through average pooling: ； in, It is the embedding vector of each label, and M is the number of labels; Dominant color extraction: Calculate the RGB mean of pixels for each frame of the image, and take the principal components of all frames as the dominant color vector. .

3. The AI-based method for recommending background music for digital collectibles according to claim 2, characterized in that, The process of extracting features from the content data of digital collectibles to obtain dynamic features of the digital collectibles includes: Inter-frame variation frequency: Obtain the inter-frame variation frequency The formula is as follows: ； in, Denotes the Euclidean norm. It is width multiplied. It's about height. It is each frame of the image. It is each frame of the image Previous frame; Keyframe information: Annotate keyframes As a feature, the mutation frame Gradient frames ; The above two features are combined to form the dynamic features of digital collections. And combine all the time vectors into a time-varying signal sequence. N is the number of sequences.

4. The AI-based digital collection background music recommendation method according to claim 3, characterized in that, The method of processing the time-varying signal sequence using a sliding time window to obtain the rhythm distribution spectrum includes: A rhythm distribution spectrum is constructed by processing time-varying signals using a sliding time window method. Select a fixed-length time window and step size, and apply it to a time-varying signal sequence. Slide on top; For each time window Calculate the following two features: Beat density: The proportion of keyframes within a time window, i.e., beat density. The formula is as follows: ； in, It's a time window. It is a keyframe annotation; Periodicity of change: the frequency of inter-frame changes within a time window Perform a Fast Fourier Transform to find the principal components in the spectrum, i.e., the periodicity of change. The formula is as follows: ； in, Indicates time window The inter-frame variation frequency sequence within the frame. Its function is to find the frequency value corresponding to the maximum value in the result of the Fast Fourier Transform; Combining the beat density and periodicity of change in each time window into a vector yields the rhythm distribution spectrum: ； Where T represents the number of time windows, it is appropriate to use parentheses to combine the beat density and periodicity of change within the same time window. This clearly indicates that the two parameters corresponding to each time window are a set of data, avoiding confusion.

5. The AI-based method for recommending background music for digital collectibles according to claim 4, characterized in that, The step of generating a music rhythm structure vector based on the music beat time sequence includes: Assuming each candidate music fragment needs to contain A series of consecutive shooting points, shooting point time series For the first Each segment, whose start and end times can be determined by the snapshot sequence index: ； in, Each candidate music clip must contain consecutive beats. This formula shows that, and Directly from the time stamp sequence of the shooting point The corresponding elements are determined by controlling the number of beats contained in each segment. Implement beat-based time segmentation to ensure that the start and end times of the segment are aligned with the actual beat positions of the music, thus providing a basis for subsequent energy averaging. The calculations provide an accurate time range; Rhythm structure vector generate: To align with the rhythm vector of digital collections Alignment, dividing the music into equal-length segments. For each segment, the average energy value of the segment is calculated as a musical rhythmic feature: ； in, and For the first The start and end times of each segment; Let dt be the duration of the segment, and dt be the differential symbol in the integral, representing a small increment on the time axis. It is the audio signal of the candidate music sample; Combine the average energy of all segments into a musical rhythmic structure vector: .

6. The AI-based method for recommending background music for digital collectibles according to claim 5, characterized in that, The rhythmic similarity is calculated by applying the Dynamic Time Warping (DWT) algorithm to the rhythmic representation vector of digital collectibles and the rhythmic structure vector of music, including: Dynamic Time Warping (DTW) rhythm matching: The core logic of the DTW algorithm is an algorithm used to align two time series, allowing for local time warping to resolve differences in tempo between digital artifacts and music, such as matching fast-paced segments of the artifact with accelerated sections of music. Specific steps include: Distance matrix construction: calculation and The Euclidean distance between each element pair in the equation forms Distance matrix ,in ; Path planning: finding the path from arrive The optimal path that satisfies local constraints and minimizes the total path distance; Similarity calculation with offset tolerance factor Introducing offset tolerance factor The allowed time offset range in path planning is expanded to allow ±30% beat offset, enhancing robustness to micro-rhythmic misalignment. The final rhythmic similarity is defined as: ； Among them, DTW ( , , () represents the DTW distance with offset constraints; the smaller the value, the higher the rhythm matching degree. The distance is converted into a similarity score through normalization.

7. The AI-based method for recommending background music for digital collectibles according to claim 6, characterized in that, The visual style vector and musical style vector of the digital collection are subjected to a cosine similarity algorithm to obtain a style consistency score, including: Cosine similarity calculation: Calculating the visual style vector of digital collections With musical style vector The cosine similarity and style consistency scoring formula is as follows: ； in, It is a style consistency score. and These are the visual style vector and the musical style vector of the digital collection. Physical meaning: They measure the semantic matching degree between the artistic style of the collection and the inherent style of the music. The higher the value, the more consistent the style labels are.

8. The AI-based method for recommending background music for digital collectibles according to claim 7, characterized in that, The process of calculating the guided score based on the auxiliary music feature vector and the underlying audio feature vector includes: Generate guided score calculation: calculate and Cosine similarity: ； in, It is a guide to rating. It is an auxiliary music feature vector. It is the underlying audio feature vector. The guidance score measures the degree of matching between the target audio features "guided" by the generative model according to the visual style and the actual features of the candidate music. The higher the value, the more consistent the granular style features.

9. The AI-based method for recommending background music for digital collectibles according to claim 8, characterized in that, The comprehensive score index is obtained based on the rhythm similarity, style consistency score, and guided score, including: The weights are determined based on factors such as the specific type and characteristics of the digital collectibles and potential user preferences. , , Meanwhile, determine the nonlinear parameters. , , ; If a digital artifact is a still image in the style of a classical oil painting, the focus is on matching the artistic style, then it can be set... , (Emphasizing stylistic consistency), Regarding nonlinear parameters, , This makes the style consistency score contribute more to the overall score. ; For each candidate music Based on the set weights and nonlinear parameters, the comprehensive score is calculated according to the following nonlinear formula: ； in, It is a comprehensive score index. , , These correspond to the weights of rhythm similarity, style consistency score, and guiding score, respectively. , , This is a nonlinear parameter.

10. The AI-based method for recommending background music for digital collectibles according to claim 9, characterized in that, The recommendation of background music for digital collectibles based on the comprehensive scoring index includes: All candidate music were ranked according to their overall score. Sort the music from highest to lowest quality, and select the top-ranked tracks based on actual needs, as shown above. head, The value can be set to 5, 10, etc., depending on the specific situation, and provided to the user as a recommendation result; If there are 20 candidate songs, calculate their overall scores and then rank them. ,set up Then it will be recommended first. , , , , These five pieces of music serve as background music to match the digital collection.

Images