A music recommendation method and system fusing deep audio semantics and collaborative filtering

Abstract

The application relates to a music recommendation method and system fusing deep audio semantics and collaborative filtering, which comprises the following steps: receiving an original audio signal for pretreatment; inputting a preset deep neural network model to obtain a deep audio semantic feature vector; collecting music interaction behavior data of all users and constructing a user-song interaction matrix; adopting a preset matrix decomposition technology to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix; when in a music recommendation scene, calculating a content-based music recommendation score according to the deep audio semantic feature vector, and calculating a collaborative filtering prediction score according to the user latent factor matrix and the song latent factor matrix; fusing the music recommendation score and the collaborative filtering prediction score to obtain a fusion recommendation score, and performing music recommendation according to the fusion recommendation score. Through the fusion of deep audio semantic analysis and collaborative filtering, the accuracy and novelty of music recommendation are improved.

Description

A music recommendation method and system integrating deep audio semantics and collaborative filtering Technical Field

[0001] This invention belongs to the field of music recommendation technology, and in particular relates to a music recommendation method and system that integrates deep audio semantics and collaborative filtering. Background Technology

[0002] In the era of digital music streaming, recommendation systems have become the core bridge connecting users with massive music libraries, and their service quality directly determines user experience and platform stickiness.

[0003] Traditional content-based music recommendation methods heavily rely on manually labeled metadata (e.g., genre, artist, era) or manually designed acoustic features (e.g., MFCC, beat, spectral features). Content-based music recommendation methods suffer from the following problems: manually labeled tags are highly subjective, with different labelers potentially assigning different tags to the same song; manual labeling is also costly and difficult to cover massive music libraries; manually designed acoustic features are often low-level, physical features that cannot fully express a song's melodic beauty, harmonic structure, emotional atmosphere, or complex style; and the understanding of music content is relatively superficial, making it difficult to capture the inherent, complex artistic and emotional attributes of music, resulting in recommendation results that are often limited to superficial similarities and fail to discover deeper content connections.

[0004] Collaborative filtering music recommendation methods can discover potential interests based on user group behavior, but they also have the following problems: (1) Cold start problem: new song cold start, that is, when a new song is released, there is no user behavior data, and traditional collaborative filtering algorithms cannot recommend it, making it difficult for new songs to gain exposure; new user cold start, that is, when a new user registers, the system lacks its historical behavior data, and cannot use collaborative filtering algorithms to make personalized recommendations, and can only provide general recommendations or random recommendations; (2) Data sparsity problem: in the user-song interaction matrix, users only have interaction behavior with a very small number of songs, resulting in an extremely sparse matrix, which affects the recommendation effect of collaborative filtering algorithms; (3) Information cocoon effect: pure collaborative filtering recommendation may cause users to only see songs that are highly similar to their known preferences, lacking novelty and exploration.

[0005] While existing technologies include hybrid recommendation methods that combine content-based music recommendation with collaborative filtering, these methods often simply perform linear weighting or parallel processing of the two recommendation results. They fail to achieve deep integration and synergistic effects between the two technologies, cannot fully leverage their respective advantages, and cannot effectively solve the cold start problem.

[0006] Therefore, there is an urgent need for a technology that can deeply integrate deep audio semantic analysis and collaborative filtering to effectively solve the cold start problem and improve the accuracy and novelty of music recommendations. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a music recommendation method and system that integrates deep audio semantics and collaborative filtering.

[0008] The technical solution adopted in this invention is:

[0009] Firstly, a music recommendation method that integrates deep audio semantics and collaborative filtering is provided, including:

[0010] Receive the raw audio signal and preprocess it.

[0011] The preprocessed raw audio signal is input into a preset deep neural network model to obtain a deep audio semantic feature vector;

[0012] Collect music interaction data from all users and construct a user-song interaction matrix;

[0013] The user-song interaction matrix is ​​decomposed into a user latent factor matrix and a song latent factor matrix using a pre-defined matrix decomposition technique.

[0014] In the context of music recommendation, a content-based music recommendation score is calculated based on the deep audio semantic feature vector, and a collaborative filtering prediction score is calculated based on the user latent factor matrix and the song latent factor matrix.

[0015] A fused recommendation score is obtained by combining the music recommendation score and the collaborative filtering prediction score, and music recommendations are made based on the fused recommendation score.

[0016] Furthermore, music interaction data from all users is collected, and a user-song interaction matrix is ​​constructed, including:

[0017] Collect music interaction data from all users, including playback behavior data, favorite behavior data, skip behavior data, and rating behavior data;

[0018] Based on the data size of playback behavior data, collection behavior data, skip behavior data and rating behavior data, the interaction intensity value R[i][j] between different users i and different songs j is obtained, i≤m, j≤n, where m represents the total number of users and n represents the total number of songs;

[0019] The user-song interaction matrix R is constructed based on the interaction intensity values ​​of all users and all songs.

[0020] Furthermore, a pre-defined matrix factorization technique is used to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix, including:

[0021] The objective function L is optimized using a pre-defined matrix factorization technique, and its expression is:

[0022] ;

[0023] in, This represents a set of user-song pairs representing interactive behaviors. Here, F represents the predefined regularization coefficient; F represents the regularization norm; and U represents the user latent factor matrix with dimension 1. k represents the dimension of the latent factors; Q represents the song's latent factor matrix, with dimension k. ;

[0024] The objective function L is solved using stochastic gradient descent or alternating least squares algorithms to obtain the user latent factor matrix U and the song latent factor matrix Q.

[0025] Furthermore, music recommendation scenarios include regular recommendation scenarios, new song cold start scenarios, and new user cold start scenarios;

[0026] In a typical recommendation scenario, all users are returning users, and all songs are existing songs.

[0027] In the scenario of a new song's cold start, the target song is a new song, and all users are existing users;

[0028] In the new user cold start scenario, the target users are new users, and all songs are old songs.

[0029] Furthermore, when the music recommendation scenario is a regular recommendation scenario;

[0030] Content-based music recommendation scores are calculated based on deep audio semantic feature vectors, and collaborative filtering prediction scores are calculated based on user latent factor matrices and song latent factor matrices, including:

[0031] Determine the deep audio semantic feature vector V_song[s] for each song s;

[0032] Obtain the historical playback records of each user u, extract multiple historical songs from the historical playback records, obtain the deep audio semantic feature vector of each historical song through a pre-set deep neural network model, calculate the average value of the deep audio semantic feature vectors of all historical songs, and obtain the preference profile V_user[u] of each user u.

[0033] Based on the deep audio semantic feature vector V_song[s] and the preference profile V_user[u], the content-based music recommendation score Score_content(s, u) for each user u for each song s is calculated. The expression for the music recommendation score Score_content(s, u) is as follows:

[0034] Score_content(s,u)=cosine(V_song[s],V_user[u]);

[0035] Based on the latent factor matrix U[u] of each user u and the latent factor matrix Q[s] of each song s, the collaborative filtering prediction score Score_cf(s,u) for each user u on each song s is calculated. The expression for the collaborative filtering prediction score Score_cf(s,u) is as follows:

[0036] Score_cf(s, u) = U[u]·Q[s] T +bias_u+bias_s;

[0037] Where bias_u is the preset user bias term, and bias_s is the preset song bias term.

[0038] Furthermore, a fused recommendation score is obtained by fusing the music recommendation score and the collaborative filtering prediction score. Music recommendations are then made based on this fused recommendation score, including:

[0039] The music recommendation score Score_content(s, u) and the collaborative filtering prediction score Score_cf(s, u) are fused to obtain the fused recommendation score Score_final(s, u) for each user u for each song s. The expression for the fused recommendation score Score_final(s, u) is as follows:

[0040] Score_final(s,u) =α×Score_content(s,u) +β×Score_cf(s,u);

[0041] Where α and β are preset weight coefficients, and α+β=1;

[0042] For each user u, the top E corresponding songs are selected as the candidate song set according to the fusion recommendation score Score_final(s, u) sorted from largest to smallest.

[0043] The candidate song collection is used as the music recommendation result for music recommendation.

[0044] Furthermore, when the music recommendation scenario is a cold start scenario for new songs;

[0045] Content-based music recommendation scores are calculated based on deep audio semantic feature vectors, and collaborative filtering prediction scores are calculated based on user latent factor matrices and song latent factor matrices, including:

[0046] Determine the deep audio semantic feature vector V_song[new] of the new song new;

[0047] Obtain the historical playback records of each user u, extract multiple historical songs from the historical playback records, obtain the deep audio semantic feature vector of each historical song through a pre-set deep neural network model, calculate the average value of the deep audio semantic feature vectors of all historical songs, and obtain the preference profile V_user[u] of each user u.

[0048] Based on the deep audio semantic feature vector V_song[new] and the preference profile V_user[u], the content-based music recommendation score Score_content(new, u) for each user u on the new song new is calculated. The expression for the music recommendation score Score_content(new, u) is as follows:

[0049] Score_content(new,u)=cosine(V_song[new],V_user[u]);

[0050] If it is determined that there is no corresponding song latent factor matrix for the new song "new", the song latent factor matrix Q[new] of the new song "new" is set to 0. The collaborative filtering prediction score is calculated to be 0 based on the user latent factor matrix U[u] and the song latent factor matrix Q[new] of each user u.

[0051] A fused recommendation score is obtained by fusing the music recommendation score and the collaborative filtering prediction score. Music recommendations are then made based on this fused recommendation score, including:

[0052] Use the music recommendation score Score_content(new, u) as the fusion recommendation score Score_final(new, u);

[0053] Based on the fusion recommendation score Score_final(new, u) sorted from largest to smallest, the top F corresponding users are selected as recommended users;

[0054] New songs will be pushed to recommended users.

[0055] Furthermore, the methods also include:

[0056] Based on the user latent factor matrix U, determine the user latent factor U[u1] of the first user u1 and the user latent factor U[u2] of the second user u2;

[0057] Calculate the similarity value sim_user(u1, u2) between the first user u1 and the second user u2. The expression for the similarity value sim_user(u1, u2) is:

[0058] sim_user(u1, u2)=cosine(U[u1],U[u2]) =(U[u1]·U[u2]) / (||U[u1]||×||U[u2]||);

[0059] Obtain the similarity value between a single user u and all users. Based on the principle of selecting K neighbor users from largest to smallest similarity value, construct the neighbor user network N(u).

[0060] Furthermore, when the music recommendation scenario is a cold start scenario for new users;

[0061] Content-based music recommendation scores are calculated based on deep audio semantic feature vectors, and collaborative filtering prediction scores are calculated based on user latent factor matrices and song latent factor matrices, including:

[0062] Determine the deep audio semantic feature vector V_song[s] for each song s;

[0063] Get the initial song selected by the new user user_new, obtain the deep audio semantic feature vector of the initial song selected by the pre-set deep neural network model, calculate the average value of the deep audio semantic feature vector of the initial song selected by the new user user_new, and obtain the initial preference profile V_user_initial[new].

[0064] Based on the deep audio semantic feature vector V_song[s] and the initial preference profile V_user_initial[new], the content-based initial music recommendation score Score_content(s,user_new_initial) for each song s by the new user user_new is calculated. The expression for the initial music recommendation score Score_content(s, user_new) is as follows:

[0065] Score_content(s, user_new_initial)=cosine(V_song[s], V_user_initial[new]);

[0066] Based on the initial music recommendation scores Score_content(s, user_new_initial) sorted from largest to smallest, the top H songs are selected as the initial recommended songs;

[0067] Obtain the initial music interaction behavior data of the new user user_new for the initial recommended song, and update the initial preference profile V_user_initial[new] based on the initial music interaction behavior data to obtain the iterative preference profile V_user_updated[new].

[0068] Based on the iterative preference profile V_user_updated[new] and the deep audio semantic feature vector V_song[s] of each song s, the content-based iterative music recommendation score Score_content(s, user_new_updated) for the new user user_new and each song s is calculated. The expression for the iterative music recommendation score Score_content(s, user_new_updated) is as follows:

[0069] Score_content(s, user_new_updated)=cosine(V_song[s], V_user_updated[new]);

[0070] Based on the initial music interaction behavior data, select G neighboring users user_g that are similar to the new user user_new from the neighboring user network N(u);

[0071] Based on the latent factor matrix U[user_g] of the neighboring user user_g and the latent factor matrix Q[s] of each song s, the collaborative filtering prediction score Score_cf(s, user_g) for each song s is calculated. The expression for the collaborative filtering prediction score Score_cf(s, user_g) is as follows:

[0072] Score_cf(s,user_g)=U[user_g]·Q[s] T +bias_u+bias_s;

[0073] Where bias_u is the preset user bias term, and bias_s is the preset song bias term;

[0074] A fused recommendation score is obtained by fusing the music recommendation score and the collaborative filtering prediction score. Music recommendations are then made based on this fused recommendation score, including:

[0075] The iterative music recommendation score Score_content(s, user_new_updated) and the collaborative filtering prediction score Score_cf(s, user_g) are fused to obtain the fused recommendation score Score_final(s, user_new). The expression for the fused recommendation score Score_final(s, user_new) is as follows:

[0076] Score_final(s, user_new) =γ×Score_content(s, user_new_updated) +δ×Score_cf(s, user_g);

[0077] Wherein, γ and δ are dynamic weighting coefficients, γ+δ=1, and the value of δ increases as the music interaction data of new user user_new increases.

[0078] Secondly, a music recommendation system that integrates deep audio semantics and collaborative filtering is provided, including:

[0079] The audio processing module is used to receive the raw audio signal and preprocess the raw audio signal;

[0080] The audio feature extraction module is used to input the preprocessed raw audio signal into a preset deep neural network model to obtain a deep audio semantic feature vector;

[0081] The user-song interaction matrix construction module is used to collect music interaction behavior data of all users and construct the user-song interaction matrix.

[0082] The matrix factorization module is used to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix using a preset matrix factorization technique.

[0083] The content recommendation and coordinated filtering module is used to calculate content-based music recommendation scores based on deep audio semantic feature vectors and to calculate collaborative filtering prediction scores based on user latent factor matrices and song latent factor matrices in music recommendation scenarios.

[0084] The fusion recommendation module is used to fuse the calculated music recommendation score and the collaborative filtering prediction score to obtain a fusion recommendation score, and then make music recommendations based on the fusion recommendation score.

[0085] The beneficial effects achieved by this invention are as follows:

[0086] The system receives raw audio signals and preprocesses them. The preprocessed audio signals are then input into a pre-set deep neural network model to obtain deep audio semantic feature vectors. Music interaction behavior data from all users is collected, and a user-song interaction matrix is ​​constructed. A pre-set matrix factorization technique is used to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix. In a music recommendation scenario, a content-based music recommendation score is calculated based on the deep audio semantic feature vectors, and a collaborative filtering prediction score is calculated based on the user latent factor matrix and the song latent factor matrix. The music recommendation score and the collaborative filtering prediction score are then fused to obtain a fused recommendation score, which is used for music recommendation. By calculating a content-based music recommendation score based on the deep audio semantic feature vectors and a collaborative filtering prediction score based on the user latent factor matrix and the song latent factor matrix, the system achieves the fusion of deep audio semantic analysis and collaborative filtering technology. The resulting fused recommendation score is more accurate, improving the precision of music recommendations.

[0087] The raw audio signal is processed directly using a pre-built deep neural network model, avoiding the subjectivity of manual labeling and the limitations of manual features in traditional methods.

[0088] Even without any user behavior data, a deep audio semantic feature vector can still be extracted from a new song. By combining the deep audio semantic feature vector with each user's preference profile, a content-based music recommendation score can be calculated. The new song is then recommended to the user according to the music recommendation score, which fundamentally solves the problem that new songs cannot be recommended due to a lack of user behavior data, enabling new songs to quickly gain exposure opportunities.

[0089] For new users, a progressive recommendation strategy is adopted. First, an initial preference profile is built based on the initial songs selected by the new user, and an initial music recommendation score is calculated for content-based music recommendation, which solves the recommendation problem when there is no music interaction behavior data. After the new user generates initial music interaction behavior data, collaborative filtering technology is used to associate the new user with neighboring users with similar early behaviors, so as to achieve rapid iteration and optimization. The hybrid strategy enables new users to quickly obtain personalized recommendations, avoiding the problem that traditional methods require a large amount of historical data for effective recommendations. Attached Figure Description

[0090] Figure 1 is a flowchart of the music recommendation method of the present invention that integrates deep audio semantics and collaborative filtering;

[0091] Figure 2 is a structural diagram of the music recommendation system that integrates deep audio semantics and collaborative filtering according to the present invention. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0093] As shown in Figure 1, this embodiment of the invention provides a music recommendation method that integrates deep audio semantics and collaborative filtering, including:

[0094] 101. Receive the raw audio signal and preprocess it.

[0095] In this embodiment, the digitized original audio signal of the music file is read into memory, and preprocessing such as sampling rate conversion and amplitude normalization is performed to convert the original audio signal into a standard format suitable for a preset deep neural network model.

[0096] For example, if the sampling rates are inconsistent, resampling is performed; the original audio signal is normalized so that its amplitude range is between [-1, 1]; the audio is divided into segments of fixed length (e.g., 3 seconds or 10 seconds), and zero padding is performed if the audio length is insufficient.

[0097] 102. Input the preprocessed original audio signal into a preset deep neural network model to obtain a deep audio semantic feature vector;

[0098] In this embodiment, the pre-set deep neural network model can specifically be the MusicNN model. Using the pre-processed raw audio signal as input, it automatically learns and extracts high-level, abstract semantic feature vectors of music through structures such as multi-layer convolutional neural networks and recurrent neural networks. The role of the MusicNN model is to: bypass traditional manual feature engineering, learn directly from the most fundamental waveform data, avoiding information loss and bias during preprocessing; and extract semantic features that can be analogous to complex concepts such as "melody progression," "harmonic color," "timbre," and even "musical style" perceived by human musicians or experienced listeners through layer-by-layer abstraction, ultimately obtaining a deep audio semantic feature vector.

[0099] Specifically, the first layer of the MusicNN model is a convolutional layer, which uses multiple convolutional kernels of different sizes (such as 3×3 and 5×5) to extract local features, and the activation function is ReLU.

[0100] The intermediate layers of the MusicNN model consist of alternating convolutional and pooling layers, gradually extracting more abstract features.

[0101] The recurrent layers of the MusicNN model: use LSTM or GRU to process temporal information and capture the temporal features of music;

[0102] The fully connected layer of the MusicNN model maps features to a fixed-dimensional deep audio semantic feature vector (dimension d, usually 128 or 256), representing high-level semantic information such as melody, harmony, timbre, and style of the song.

[0103] 103. Collect music interaction behavior data from all users and construct a user-song interaction matrix;

[0104] In this embodiment, it should be noted that the order of steps 103, 101, and 102 is not important.

[0105] Collect music interaction behavior data from all users, including playback behavior data, favorite behavior data, skip behavior data, and rating behavior data; based on the data size of playback behavior data, favorite behavior data, skip behavior data, and rating behavior data, obtain the interaction intensity value R[i][j] between different users i and different songs j, i≤m, j≤n, where m represents the total number of users and n represents the total number of songs; construct the user-song interaction matrix R based on the interaction intensity values ​​between all users and all songs.

[0106] Specifically, music interaction behavior data includes:

[0107] Playback behavior data: Records data such as the duration of a user's song playback and whether it was played completely;

[0108] Collection behavior data: Records the songs that users collect;

[0109] Skip behavior data: Records songs that users quickly skip;

[0110] Rating behavior data: Records users' ratings of songs (e.g., 1-5 stars);

[0111] The process of constructing the user-song interaction matrix R:

[0112] Full playback: Add 2.0 to the interaction intensity value R[i][j];

[0113] Add 3.0 to the interaction strength value R[i][j];

[0114] Partial playback (playback duration exceeding 50% of the total song duration): Increase the interaction intensity value R[i][j] by 1.0;

[0115] Skip (playback duration less than 10 seconds): Subtract 1.0 from the interaction intensity value R[i][j];

[0116] Rating: Different ratings are represented by the interaction intensity value R[i][j] plus the corresponding numerical value. For example, 1-5 stars correspond to values ​​of 1.0-5.0.

[0117] 104. The user-song interaction matrix is ​​decomposed into a user latent factor matrix and a song latent factor matrix using a pre-defined matrix decomposition technique.

[0118] In this embodiment, a preset matrix factorization technique is used to optimize the objective function L, which is expressed as:

[0119] ;

[0120] in, This represents a set of user-song pairs representing interactive behaviors. Here, F represents the predefined regularization coefficient; F represents the regularization norm; and U represents the user latent factor matrix with dimension 1. k represents the dimension of the latent factors; Q represents the song's latent factor matrix, with dimension k. Each latent factor represents a feature value of a user or song on a certain hidden preference dimension;

[0121] The objective function L is solved using stochastic gradient descent (SGD) or alternating least squares (ALS) algorithms to obtain the user latent factor matrix U and the song latent factor matrix Q.

[0122] Specifically, each dimension of the user latent factor matrix U[u] represents the feature value of user u on a certain hidden preference dimension; for example, one factor may represent "liking the atmosphere of listening alone on a rainy afternoon", and another factor may represent "preferring fast-paced electronic music", etc.; these factors are learned from user behavior data through matrix factorization.

[0123] Each dimension of the song latent factor matrix Q[s] represents the feature value of song s on a certain latent attribute dimension, and it resides in the same latent factor space as the user latent factor matrix U[u], such that U[u]·Q[s] T It can predict user u's preference for song s.

[0124] 105. In the music recommendation scenario, a content-based music recommendation score is calculated based on the deep audio semantic feature vector, and a collaborative filtering prediction score is calculated based on the user latent factor matrix and the song latent factor matrix.

[0125] In this embodiment, there are three music recommendation scenarios: (i) regular recommendation scenario: all users are old users and all songs are old songs; (ii) new song cold start scenario: the target song is a new song and all users are old users; (iii) new user cold start scenario: the target user is a new user and all songs are old songs.

[0126] (i) When the music recommendation scenario is a regular recommendation scenario;

[0127] Content-based music recommendation scores are calculated based on deep audio semantic feature vectors, and collaborative filtering prediction scores are calculated based on user latent factor matrices and song latent factor matrices, including:

[0128] Determine the deep audio semantic feature vector V_song[s] for each song s;

[0129] Obtain the historical playback records of each user u, extract multiple historical songs from the historical playback records, obtain the deep audio semantic feature vector of each historical song through a pre-set deep neural network model, calculate the average value of the deep audio semantic feature vectors of all historical songs, and obtain the preference profile V_user[u] of each user u.

[0130] Based on the deep audio semantic feature vector V_song[s] and the preference profile V_user[u], the content-based music recommendation score Score_content(s, u) for each user u for each song s is calculated. The expression for the music recommendation score Score_content(s, u) is as follows:

[0131] Score_content(s,u)=cosine(V_song[s],V_user[u]);

[0132] Based on the latent factor matrix U[u] of each user u and the latent factor matrix Q[s] of each song s, the collaborative filtering prediction score Score_cf(s,u) for each user u on each song s is calculated. The expression for the collaborative filtering prediction score Score_cf(s,u) is as follows:

[0133] Score_cf(s, u) = U[u]·Q[s] T +bias_u+bias_s;

[0134] Where bias_u is the preset user bias term, and bias_s is the preset song bias term.

[0135] (ii) When the music recommendation scenario is a cold start scenario for new songs;

[0136] Content-based music recommendation scores are calculated based on deep audio semantic feature vectors, and collaborative filtering prediction scores are calculated based on user latent factor matrices and song latent factor matrices, including:

[0137] Determine the deep audio semantic feature vector V_song[new] of the new song new;

[0138] Obtain the historical playback records of each user u, extract multiple historical songs from the historical playback records, obtain the deep audio semantic feature vector of each historical song through a pre-set deep neural network model, calculate the average value of the deep audio semantic feature vectors of all historical songs, and obtain the preference profile V_user[u] of each user u.

[0139] Based on the deep audio semantic feature vector V_song[new] and the preference profile V_user[u], the content-based music recommendation score Score_content(new, u) for each user u on the new song new is calculated. The expression for the music recommendation score Score_content(new, u) is as follows:

[0140] Score_content(new,u)=cosine(V_song[new],V_user[u]);

[0141] If it is determined that there is no corresponding song latent factor matrix for the new song "new", the song latent factor matrix Q[new] for the new song "new" is set to 0. The collaborative filtering prediction score is calculated to be 0 based on the user latent factor matrix U[u] and the song latent factor matrix Q[new] for each user u.

[0142] Before implementing the music recommendation scenario (iii), it is also necessary to build a neighboring user network in advance, specifically as follows:

[0143] Based on the user latent factor matrix U, determine the user latent factor U[u1] of the first user u1 and the user latent factor U[u2] of the second user u2;

[0144] Calculate the similarity value sim_user(u1, u2) between the first user u1 and the second user u2. The expression for the similarity value sim_user(u1, u2) is:

[0145] sim_user(u1, u2)=cosine(U[u1],U[u2]) =(U[u1]·U[u2]) / (||U[u1]||×||U[u2]||);

[0146] Obtain the similarity value between a single user u and all users. Based on the principle of selecting K neighbor users from largest to smallest similarity value, construct the neighbor user network N(u).

[0147] (iii) When the music recommendation scenario is a new user cold start scenario;

[0148] Determine the deep audio semantic feature vector V_song[s] for each song s;

[0149] Get the initial song selection of the new user user_new (e.g., three songs: "Nocturne", "Moonlight", and "Serenade"). Obtain the deep audio semantic feature vector of the initial song selection through a pre-set deep neural network model. Calculate the average value of the deep audio semantic feature vector of the initial song selection to obtain the initial preference profile of the new user user_new, V_user_initial[new].

[0150] Based on the deep audio semantic feature vector V_song[s] and the initial preference profile V_user_initial[new], the content-based initial music recommendation score Score_content(s,user_new_initial) for each song s by the new user user_new is calculated. The expression for the initial music recommendation score Score_content(s, user_new) is as follows:

[0151] Score_content(s, user_new_initial)=cosine(V_song[s], V_user_initial[new]);

[0152] Based on the initial music recommendation scores Score_content(s, user_new_initial) sorted from largest to smallest, the top H songs are selected as the initial recommended songs; the specific value of H can be 20, that is, the initial recommendation is made when the new user has no music interaction behavior data;

[0153] Obtain the initial music interaction behavior data of the new user user_new for the initial recommended songs. Since the number of initial recommended songs is small, only a small amount of initial music interaction behavior data needs to be collected, such as 10. Update the initial preference profile V_user_initial[new] based on the initial music interaction behavior data to obtain the iterative preference profile V_user_updated[new]. The iterative preference profile makes the profile more accurate.

[0154] Based on the iterative preference profile V_user_updated[new] and the deep audio semantic feature vector V_song[s] of each song s, the content-based iterative music recommendation score Score_content(s, user_new_updated) for the new user user_new and each song s is calculated. The expression for the iterative music recommendation score Score_content(s, user_new_updated) is as follows:

[0155] Score_content(s, user_new_updated)=cosine(V_song[s], V_user_updated[new]);

[0156] Based on the initial music interaction behavior data, select G neighboring users user_g from the neighboring user network N(u) that are similar to the new user user_new; the value of G is specifically 20, that is, select 20 neighboring users whose preferences are similar to the new user.

[0157] Based on the latent factor matrix U[user_g] of the neighboring user user_g and the latent factor matrix Q[s] of each song s, the collaborative filtering prediction score Score_cf(s, user_g) for each song s is calculated. The expression for the collaborative filtering prediction score Score_cf(s, user_g) is as follows:

[0158] Score_cf(s,user_g)=U[user_g]·Q[s] T +bias_u+bias_s;

[0159] Where bias_u is the preset user bias term, and bias_s is the preset song bias term.

[0160] 106. The music recommendation score and the collaborative filtering prediction score are combined to obtain a fused recommendation score, and music recommendations are made based on the fused recommendation score.

[0161] In this embodiment, (i) when the music recommendation scenario is a regular recommendation scenario;

[0162] The calculated music recommendation score Score_content(s, u) and the collaborative filtering prediction score Score_cf(s, u) are fused together to obtain the fused recommendation score Score_final(s, u) for each user u for each song s. The expression for the fused recommendation score Score_final(s, u) is as follows:

[0163] Score_final(s,u) =α×Score_content(s,u) +β×Score_cf(s,u);

[0164] Where α and β are preset weight coefficients, and α+β=1;

[0165] For each user u, the top E songs are selected as the candidate song set according to the fusion recommendation score Score_final(s, u) sorted from largest to smallest; the specific value of E can be 10.

[0166] The candidate song collection is used as the music recommendation result for music recommendation.

[0167] (ii) When the music recommendation scenario is a cold start scenario for new songs;

[0168] Use the music recommendation score Score_content(new, u) as the fusion recommendation score Score_final(new, u);

[0169] Based on the fusion recommendation score Score_final(new, u) sorted from largest to smallest, the top F corresponding users are selected as recommended users; the specific value of F is 1000, which means that new songs are pushed to a large number of interested users;

[0170] New songs will be pushed to recommended users.

[0171] (iii) When the music recommendation scenario is a new user cold start scenario;

[0172] The iterative music recommendation score Score_content(s, user_new_updated) and the collaborative filtering prediction score Score_cf(s, user_g) are fused to obtain the fused recommendation score Score_final(s, user_new). The expression for the fused recommendation score Score_final(s, user_new) is as follows:

[0173] Score_final(s, user_new) =γ×Score_content(s, user_new_updated) +δ×Score_cf(s, user_g);

[0174] Here, γ and δ are dynamic weighting coefficients, γ+δ=1, and the value of δ increases as the music interaction data of new user user_new increases, while γ decreases accordingly.

[0175] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0176] This system calculates content-based music recommendation scores based on deep audio semantic feature vectors and collaborative filtering prediction scores based on user latent factor matrices and song latent factor matrices. This integration of deep audio semantic analysis and collaborative filtering techniques ensures the accuracy of recommendations, accurately capturing users' true preferences for music content. Collaborative filtering, by incorporating the wisdom of neighboring users, can discover users' potential interests, broaden their musical horizons, avoid the information cocoon effect, and enhance the novelty and exploratory nature of recommendations. The fusion of these two technologies enables the system to "understand music" and "understand people," ensuring novelty while maintaining accurate recommendations.

[0177] By directly processing the raw audio signal using a pre-built deep neural network model, the subjectivity of manual labeling and the limitations of manual features in traditional methods are avoided. Through multi-layer nonlinear transformations, the deep neural network can automatically learn the mapping relationship from low-level audio features (such as spectrum and temporal features) to high-level semantic features (such as melody, harmony, and style), thereby extracting high-level semantic features that can accurately represent the inherent artistic and emotional attributes of music. The end-to-end learning approach enables the model to capture the complex musical attributes that are difficult to express by traditional methods, achieving a deep understanding of the musical content.

[0178] Even without any user behavior data, a deep audio semantic feature vector can still be extracted from a new song. By combining the deep audio semantic feature vector with each user's preference profile, a content-based music recommendation score can be calculated. The new song is then recommended to the user according to the music recommendation score, which fundamentally solves the problem that new songs cannot be recommended due to a lack of user behavior data, enabling new songs to quickly gain exposure opportunities.

[0179] For new users, a progressive recommendation strategy is adopted. First, an initial preference profile is built based on the initial songs selected by the new user, and an initial music recommendation score is calculated for content-based music recommendation, which solves the recommendation problem when there is no music interaction behavior data. After the new user generates initial music interaction behavior data, collaborative filtering technology is used to associate the new user with neighboring users with similar early behaviors, enabling rapid iteration and optimization. The hybrid strategy enables new users to quickly obtain personalized recommendations, avoiding the problem of traditional methods that require a large amount of historical data for effective recommendations.

[0180] Deep audio semantic feature vectors serve as the initial input or regularization term for matrix factorization techniques, improving the performance of collaborative filtering in sparse data situations. User group preferences discovered by collaborative filtering can feed back into the content analysis model, helping to understand which audio features are more popular among specific user groups. This two-way feedback mechanism enables the two technologies to form a closed loop of complementary advantages and synergistic effects, achieving a 1+1>2 effect.

[0181] As shown in Figure 2, this embodiment of the invention provides a music recommendation system that integrates deep audio semantics and collaborative filtering, including:

[0182] Audio processing module 201 is used to receive the raw audio signal and preprocess the raw audio signal;

[0183] The audio feature extraction module 202 is used to input the preprocessed raw audio signal into a preset deep neural network model to obtain a deep audio semantic feature vector.

[0184] The user-song interaction matrix construction module 203 is used to collect music interaction behavior data of all users and construct the user-song interaction matrix.

[0185] The matrix decomposition module 204 is used to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix using a preset matrix decomposition technique.

[0186] The content recommendation and coordinated filtering module 205 is used to calculate a content-based music recommendation score based on the deep audio semantic feature vector and a coordinated filtering prediction score based on the user latent factor matrix and the song latent factor matrix when in a music recommendation scenario.

[0187] The fusion recommendation module 206 is used to fuse the music recommendation score and the collaborative filtering prediction score to obtain a fusion recommendation score, and to make music recommendations based on the fusion recommendation score.

[0188] The music recommendation system that integrates deep audio semantics and collaborative filtering in this embodiment of the invention can implement the method shown in the embodiment of Figure 1 above.

[0189] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0193] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A music recommendation method integrating deep audio semantics and collaborative filtering, characterized in that, include: Receive the raw audio signal and preprocess the raw audio signal; The preprocessed original audio signal is input into a preset deep neural network model to obtain a deep audio semantic feature vector; music interaction behavior data of all users are collected and a user-song interaction matrix is ​​constructed; the user-song interaction matrix is ​​decomposed into a user latent factor matrix and a song latent factor matrix using a preset matrix factorization technique. In a music recommendation scenario, a content-based music recommendation score is calculated based on the deep audio semantic feature vector, and a collaborative filtering prediction score is calculated based on the user latent factor matrix and the song latent factor matrix. The music recommendation scenario includes a regular recommendation scenario, a new song cold start scenario, and a new user cold start scenario. In the regular recommendation scenario, all users are existing users, and all songs are existing songs. In the new song cold start scenario, the target song is a new song, and all users are existing users. In the new user cold start scenario, the target user is a new user, and all songs are existing songs. The music recommendation score and the collaborative filtering prediction score are fused to obtain a fused recommendation score, and music recommendations are made based on this fused recommendation score. When the music recommendation scenario is the new user cold start scenario, the user latent factor U[u1] of the first user u1 and the user latent factor U[u2] of the second user u2 are determined based on the user latent factor matrix U. The similarity value sim_user(u1, u2) between the first user u1 and the second user u2 is calculated, and the expression for the similarity value sim_user(u1, u2) is: The process involves: obtaining the similarity value between a single user u and all users; selecting K neighboring users based on the principle of descending similarity values ​​to construct a neighboring user network N(u); calculating a content-based music recommendation score based on the deep audio semantic feature vector and a collaborative filtering prediction score based on the user latent factor matrix and the song latent factor matrix, including: determining the deep audio semantic feature vector V_song[s] for each song s; obtaining the initial song selection of a new user user_new; obtaining the deep audio semantic feature vector of the initial song selection through the pre-set deep neural network model; calculating the average value of the deep audio semantic feature vector of the initial song selection to obtain the initial preference profile V_user_initial[new] for the new user user_new; and calculating the content-based initial music recommendation score Score_content(s, user_new_initial) for each song s based on the deep audio semantic feature vector V_song[s] and the initial preference profile V_user_initial[new]. The expression for `user_new` is: `Score_content(s, user_new_initial) = cosine(V_song[s], V_user_initial[new])`. Based on the initial music recommendation scores `Score_content(s, user_new_initial)` sorted from largest to smallest, the top H songs are selected as the initial recommended songs. The initial music interaction behavior data of the new user `user_new` for the initially recommended songs is obtained, and the initial preference profile `V_user_initial[new]` is updated according to the initial music interaction behavior data to obtain the iterative preference profile `V_user_updated[new]`. Based on the iterative preference profile `V_user_updated[new]` and the deep audio semantic feature vector `V_song[s]` of each song `s`, the content-based iterative music recommendation score `Score_content(s, user_new_updated)` for the new user `user_new` and each song `s` is calculated. The expression is: Score_content(s, user_new_updated) = cosine(V_song[s], V_user_updated[new]);Based on the initial music interaction behavior data, select G neighboring users user_g from the neighboring user network N(u) that are similar to the new user user_new; based on the user latent factor matrix U[user_g] of the neighboring user user_g and the song latent factor matrix Q[s] of each song s, calculate the collaborative filtering prediction score Score_cf(s, user_g) of the neighboring user user_g for each song s, and the expression of the collaborative filtering prediction score Score_cf(s, user_g) is: Score_cf(s, user_g)=U[user_g]·Q[s]; T +bias_u+bias_s; where bias_u is a preset user bias term, and bias_s is a preset song bias term; the fusion calculation of the music recommendation score and the collaborative filtering prediction score to obtain a fused recommendation score, and the music recommendation based on the fused recommendation score, includes: fusing the iterative music recommendation score Score_content(s, user_new_updated) and the collaborative filtering prediction score Score_cf(s, user_g) to obtain a fused recommendation score Score_final(s, user_new), the expression of the fused recommendation score Score_final(s, user_new) is: Score_final(s, user_new) = γ×Score_content(s, user_new_updated) +δ×Score_cf(s, user_g); where γ and δ are dynamic weight coefficients, γ+δ=1, and the value of δ increases as the music interaction behavior data of the new user user_new increases.

2. The music recommendation method integrating deep audio semantics and collaborative filtering according to claim 1, characterized in that, The process of collecting music interaction behavior data from all users and constructing a user-song interaction matrix includes: collecting music interaction behavior data from all users, including playback behavior data, favorite behavior data, skip behavior data, and rating behavior data; obtaining the interaction intensity value R[i][j] between different users i and different songs j based on the data size of the playback behavior data, favorite behavior data, skip behavior data, and rating behavior data, where i≤m, j≤n, where m represents the total number of users and n represents the total number of songs; and constructing the user-song interaction matrix R based on the interaction intensity values ​​between all users and all songs.

3. The music recommendation method integrating deep audio semantics and collaborative filtering according to claim 2, characterized in that, The step of decomposing the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix using a pre-defined matrix factorization technique includes: optimizing the objective function L using a pre-defined matrix factorization technique, expressed as: ; wherein, the The set of user-song pairs representing interactive behaviors; The predefined regularization coefficient; F represents the regularization norm; U represents the user latent factor matrix with dimension . Where k is the dimension of the latent factors; Q represents the song latent factor matrix, with dimension 1. The objective function L is solved using stochastic gradient descent or alternating least squares algorithms to obtain the user latent factor matrix U and the song latent factor matrix Q.

4. The music recommendation method integrating deep audio semantics and collaborative filtering according to claim 1, characterized in that, When the music recommendation scenario is the regular recommendation scenario; the step of calculating a content-based music recommendation score based on the deep audio semantic feature vector and calculating a collaborative filtering prediction score based on the user latent factor matrix and the song latent factor matrix includes: determining the deep audio semantic feature vector V_song[s] for each song s; obtaining the historical playback records of each user u, extracting multiple historical songs from the historical playback records, obtaining the deep audio semantic feature vector of each historical song through the pre-set deep neural network model, calculating the average value of the deep audio semantic feature vectors of all historical songs, and obtaining the preference profile V_user[u] for each user u; calculating the content-based music recommendation score Score_content(s, u) for each user u for each song s based on the deep audio semantic feature vector V_song[s] and the preference profile V_user[u]. The expression is: Score_content(s, u) = cosine(V_song[s], V_user[u]); Based on the user latent factor matrix U[u] for each user u and the song latent factor matrix Q[s] for each song s, the collaborative filtering prediction score Score_cf(s, u) for each user u on each song s is calculated, and the expression for the collaborative filtering prediction score Score_cf(s, u) is: Score_cf(s, u) = U[u]·Q[s] T +bias_u+bias_s; where bias_u is a preset user bias term and bias_s is a preset song bias term.

5. The music recommendation method integrating deep audio semantics and collaborative filtering according to claim 4, characterized in that, The process of fusing the music recommendation score and the collaborative filtering prediction score to obtain a fused recommendation score, and then recommending music based on the fused recommendation score, includes: fusing the music recommendation score Score_content(s, u) and the collaborative filtering prediction score Score_cf(s, u) to obtain a fused recommendation score Score_final(s, u) for each user u for each song s, wherein the expression for the fused recommendation score Score_final(s, u) is: Score_final(s, u) = α × Score_content(s, u) + β × Score_cf(s, u); where α and β are preset weight coefficients, and α + β = 1; for each user u, selecting the top E corresponding songs as a candidate song set according to the fused recommendation score Score_final(s, u) sorted from largest to smallest; and using the candidate song set as the music recommendation result for music recommendation.

6. The music recommendation method integrating deep audio semantics and collaborative filtering according to claim 1, characterized in that, When the music recommendation scenario is a cold start scenario for a new song; the step of calculating a content-based music recommendation score based on the deep audio semantic feature vector and calculating a collaborative filtering prediction score based on the user latent factor matrix and the song latent factor matrix includes: determining the deep audio semantic feature vector V_song[new] for the new song; obtaining the historical playback records of each user u, extracting multiple historical songs from the historical playback records, obtaining the deep audio semantic feature vector of each historical song through the pre-set deep neural network model, calculating the average value of the deep audio semantic feature vectors of all historical songs, and obtaining the preference profile V_user[u] for each user u; calculating the content-based music recommendation score Score_content(new, u) for each user u on the new song new based on the deep audio semantic feature vector V_song[new] and the preference profile V_user[u], wherein the music recommendation score Score_content(new, u) is... The expression is: Score_content(new, u) = cosine(V_song[new], V_user[u]); It is determined that the new song 'new' does not have a corresponding latent factor matrix, so the latent factor matrix Q[new] of the new song 'new' is set to 0. The collaborative filtering prediction score is calculated as 0 based on the latent factor matrix U[u] of each user u and the latent factor matrix Q[new]. The fusion calculation of the music recommendation score and the collaborative filtering prediction score yields a fused recommendation score. Music recommendation is performed based on the fused recommendation score, including: using the music recommendation score Score_content(new, u) as the fused recommendation score Score_final(new, u); selecting the top F corresponding users as recommended users according to the descending order of the fused recommendation score Score_final(new, u); and pushing the new song 'new' to the recommended users.

7. A music recommendation system integrating deep audio semantics and collaborative filtering, characterized in that, include: An audio processing module is used to receive raw audio signals and preprocess the raw audio signals; The audio feature extraction module is used to input the preprocessed original audio signal into a preset deep neural network model to obtain a deep audio semantic feature vector. The user-song interaction matrix construction module is used to collect music interaction behavior data of all users and construct the user-song interaction matrix. The matrix factorization module is used to decompose the user-song interaction matrix into a user latent factor matrix and a song latent factor matrix using a preset matrix factorization technique. The content recommendation and coordinated filtering module is used to calculate a content-based music recommendation score based on the deep audio semantic feature vector and a coordinated filtering prediction score based on the user latent factor matrix and the song latent factor matrix when in a music recommendation scenario. The music recommendation scenarios include a regular recommendation scenario, a new song cold start scenario, and a new user cold start scenario. In the regular recommendation scenario, all users are existing users, and all songs are existing songs. In the new song cold start scenario, the target song is a new song, and all users are existing users. In the new user cold start scenario, the target user is a new user, and all songs are existing songs. The fusion recommendation module is used to fuse the music recommendation score and the collaborative filtering prediction score to obtain a fusion recommendation score, and to make music recommendations based on the fusion recommendation score; when the music recommendation scenario is the new user cold start scenario; the music recommendation system further includes: a neighbor user network construction module; the neighbor user network construction module is used to determine the user latent factor U[u1] of the first user u1 and the user latent factor U[u2] of the second user u2 based on the user latent factor matrix U; and to calculate the similarity value sim_user(u1, u2) between the first user u1 and the second user u2, wherein the expression of the similarity value sim_user(u1, u2) is: ; Obtain the similarity value between a single user u and all users, and select K neighboring users according to the principle of descending similarity value to construct a neighboring user network N(u); The content recommendation and coordination filtering module is specifically used to determine the deep audio semantic feature vector V_song[s] of each song s; Obtain the initial selected song initial of the new user user_new, obtain the deep audio semantic feature vector of the initial selected song initial through the pre-set deep neural network model, calculate the average value of the deep audio semantic feature vector of the initial selected song initial, and obtain the initial preference profile V_user_initial[new] of the new user user_new; According to the deep audio semantic feature vector V_song[s] and the initial preference profile V_user_initial[new], calculate the content-based initial music recommendation score Score_content(s, user_new_initial) of the new user user_new for each song s, and the expression of the initial music recommendation score Score_content(s, user_new) is: Score_content(s, The algorithm calculates the initial music recommendation score Score_content(s, user_new_initial) as cosine(V_song[s], V_user_initial[new]). It then selects the top H songs as initial recommendation songs based on the initial music recommendation scores Score_content(s, user_new_initial) sorted from largest to smallest. The algorithm obtains the initial music interaction behavior data of the new user user_new for the initial recommendation songs and updates the initial preference profile V_user_initial[new] based on this data, resulting in an iterative preference profile V_user_updated[new]. Finally, it calculates the content-based iterative music recommendation score Score_content(s, user_new_updated) for the new user user_new and each song s based on the iterative preference profile V_user_updated[new] and the deep audio semantic feature vector V_song[s] of each song s. The expression is: Score_content(s, user_new_updated) = cosine(V_song[s], V_user_updated[new]); Based on the initial music interaction behavior data, select G neighboring users user_g from the neighboring user network N(u) that are similar to the new user user_new;Based on the user latent factor matrix U[user_g] of the neighboring user user_g and the song latent factor matrix Q[s] of each song s, the collaborative filtering prediction score Score_cf(s, user_g) of the neighboring user user_g for each song s is calculated. The expression of the collaborative filtering prediction score Score_cf(s, user_g) is: Score_cf(s, user_g)=U[user_g]·Q[s]; T +bias_u+bias_s; where bias_u is a preset user bias term and bias_s is a preset song bias term; the fusion recommendation module is specifically used to fuse the iterative music recommendation score Score_content(s, user_new_updated) and the collaborative filtering prediction score Score_cf(s, user_g) to obtain the fusion recommendation score Score_final(s, user_new), the expression of which is: Score_final(s, user_new) = γ×Score_content(s, user_new_updated) +δ×Score_cf(s, user_g); where γ and δ are dynamic weight coefficients, γ+δ=1, and the value of δ increases as the music interaction behavior data of the new user user_new increases.

Images