Method, device and storage medium for song recommendation
By using a cross-modal representation model and a preference feature extraction model, personalized recommendations are made using the audio and tag information of new songs, which solves the problem of lack of personalized recommendations in new song recommendations and achieves accurate recommendations without historical user behavior.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
- Filing Date
- 2024-08-23
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies lack personalized recommendation methods for new song recommendations and cannot effectively utilize the audio and tag information of new songs for recommendations.
A cross-modal representation model and a preference feature extraction model are adopted. The audio fusion features and tag fusion features of the songs to be recommended are extracted through audio feature representation branches and tag feature representation branches. Similarity analysis is performed by combining the preference features of the user's favorite songs to achieve personalized recommendations.
It does not rely on historical user behavior for new songs, but makes personalized recommendations based on audio and tag information, improving the accuracy of new song recommendations and user satisfaction.
Smart Images
Figure CN119025703B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio technology, and in particular to a method, apparatus and storage medium for song recommendation. Background Technology
[0002] Song recommendations are one of the important channels for users to listen to music, and music recommendations are also an important way to promote songs.
[0003] Currently, song recommendations mainly use collaborative filtering, which recommends songs to other users with similar preferences based on the target user's historical user behavior, such as playing or saving the song. This achieves song recommendations based on user preferences.
[0004] However, for new songs, since they haven't been promoted before, there's no historical user behavior data. In this case, the above song recommendation methods are no longer applicable. Therefore, there is an urgent need for a method that can provide personalized recommendations for new songs. Summary of the Invention
[0005] This application provides a method, device, and storage medium for song recommendation, enabling personalized song recommendations to be given to suitable users. The technical solution is as follows:
[0006] Firstly, a method for song recommendation is provided, the method comprising:
[0007] Obtain the audio and tag information of the songs to be recommended;
[0008] Based on the audio and the tag information, determine the audio fusion features and tag fusion features of the song to be recommended;
[0009] Based on the audio fusion features and the tag fusion features, the preference features of the songs to be recommended are determined;
[0010] If the similarity between the song preference characteristics of the target user account and the song preference characteristics of the song to be recommended meets the recommendation criteria, the song to be recommended is recommended to the target user account.
[0011] In one possible implementation,
[0012] The step of determining the audio fusion features and tag fusion features of the song to be recommended based on the audio and the tag information includes:
[0013] A cross-modal representation model is determined, which includes an audio feature representation branch and a label feature representation branch;
[0014] The audio is input into the audio feature representation branch to obtain the audio fusion features of the song to be recommended;
[0015] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.
[0016] In one possible implementation, before inputting the audio into the audio feature representation branch and the tag information into the tag feature representation branch, the method further includes:
[0017] Obtain multiple sets of first training samples, wherein each set of first training samples includes the audio of the first sample song, the label information of the first sample song, and the label information of at least one other song;
[0018] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least a pair of negative examples.
[0019] Based on the positive examples and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain the audio feature representation branch and the label feature representation branch.
[0020] In one possible implementation, determining the preference features of the song to be recommended based on the audio fusion features and the tag fusion features includes:
[0021] The audio fusion features and the tag fusion features are input into the preference feature extraction model to obtain the preference features of the song to be recommended.
[0022] In one possible implementation, before inputting the audio fusion features and the tag fusion features into the preference feature extraction model, the method further includes:
[0023] Obtain multiple sets of second training samples, wherein each set of second training samples includes the audio and tag information of multiple second sample songs, and the multiple second sample songs are the intersection of the songs collected in the collection playlists of multiple users;
[0024] For each second sample song in each group of second training samples, the audio and label information of the second sample song are input into the cross-modal representation model to obtain the audio fusion feature and the label fusion feature of the second sample song.
[0025] The audio fusion features and tag fusion features of the second sample song are input into the preference feature extraction model to obtain the predicted preference features of the second sample song.
[0026] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, the central predicted preference features corresponding to the second training samples are determined, and the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples is calculated.
[0027] Based on the similarity between the predicted preference features of the center and the predicted preference features of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.
[0028] In one possible implementation, determining the center prediction preference features corresponding to the second training sample based on the prediction preference features of the second sample songs included in the second training sample includes:
[0029] Select the center song from the second sample songs included in the second training sample;
[0030] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.
[0031] In one possible implementation, selecting the center song in the second training sample includes:
[0032] The second sample song with the highest popularity in the second training sample is taken as the center song of the second training sample, wherein the popularity is used to represent at least one of the song's play count, download count, and collection count.
[0033] In one possible implementation, determining the center prediction preference features corresponding to the second training sample based on the prediction preference features of the second sample songs included in the second training sample includes:
[0034] Calculate the average feature of the predicted preference features of the second sample songs included in the second training sample;
[0035] The average feature is used as the center prediction preference feature corresponding to the second training sample.
[0036] In one possible implementation, the initial preference feature extraction model is trained based on the similarity between the predicted preference features of the center and the predicted preference features of at least one second sample song included in the second training samples to obtain the preference feature extraction model, which includes:
[0037] For each group of second training samples, the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of at least one second sample song included in the second training sample is accumulated to obtain the accumulated result.
[0038] The accumulated result is multiplied by the weight corresponding to the second training sample to obtain the product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;
[0039] Based on the product result, calculate the loss value;
[0040] Based on the loss value, the initial preference feature extraction model is trained.
[0041] In one possible implementation, the tag information of the song to be recommended includes at least one of the following: the artist's name, lyrics, release date, awards, creative background, and genre.
[0042] Secondly, a song recommendation device is provided, the device comprising:
[0043] The acquisition module is used to acquire the audio and tag information of the songs to be recommended;
[0044] The inference module is used to determine the audio fusion features and tag fusion features of the song to be recommended based on the audio and tag information; and to determine the preference features of the song to be recommended based on the audio fusion features and tag fusion features.
[0045] The recommendation module is used to recommend the song to be recommended to the target user account if the similarity between the preference features of the songs collected by the target user account and the preference features of the song to be recommended meets the recommendation conditions.
[0046] In one possible implementation, the cross-modal representation model includes an audio feature representation branch and a label feature representation branch, and the inference module is used for:
[0047] A cross-modal representation model is determined, which includes an audio feature representation branch and a label feature representation branch;
[0048] The audio is input into the audio feature representation branch to obtain the audio fusion features of the song to be recommended;
[0049] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.
[0050] In one possible implementation, the device further includes a training module for:
[0051] Obtain multiple sets of first training samples, wherein each set of first training samples includes the audio of the first sample song, the label information of the first sample song, and the label information of at least one other song;
[0052] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least a pair of negative examples.
[0053] Based on the positive examples and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain the audio feature representation branch and the label feature representation branch.
[0054] In one possible implementation, the inference module is used for:
[0055] The audio fusion features and the tag fusion features are input into the preference feature extraction model to obtain the preference features of the song to be recommended.
[0056] In one possible implementation, the device further includes a training module for:
[0057] Obtain multiple sets of second training samples, wherein each set of second training samples includes the audio and tag information of multiple second sample songs, and the multiple second sample songs are the intersection of the songs collected in the collection playlists of multiple users;
[0058] For each second sample song in each group of second training samples, the audio and label information of the second sample song are input into the cross-modal representation model to obtain the audio fusion feature and the label fusion feature of the second sample song.
[0059] The audio fusion features and tag fusion features of the second sample song are input into the preference feature extraction model to obtain the predicted preference features of the second sample song.
[0060] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, the central predicted preference features corresponding to the second training samples are determined, and the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples is calculated.
[0061] Based on the similarity between the predicted preference features of the center and the predicted preference features of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.
[0062] In one possible implementation, the training module is used for:
[0063] Select the center song from the second sample songs included in the second training sample;
[0064] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.
[0065] In one possible implementation, the training module is used for:
[0066] The second sample song with the highest popularity in the second training sample is taken as the center song of the second training sample, wherein the popularity is used to represent at least one of the song's play count, download count, and collection count.
[0067] In one possible implementation, the training module is used for:
[0068] Calculate the average feature of the predicted preference features of the second sample songs included in the second training sample;
[0069] The average feature is used as the center prediction preference feature corresponding to the second training sample.
[0070] In one possible implementation, the training module is used for:
[0071] For each group of second training samples, the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of at least one second sample song included in the second training sample is accumulated to obtain the accumulated result.
[0072] The accumulated result is multiplied by the weight corresponding to the second training sample to obtain the product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;
[0073] Based on the product result, calculate the loss value;
[0074] Based on the loss value, the initial preference feature extraction model is trained.
[0075] In one possible implementation, the tag information of the song to be recommended includes at least one of the following: the artist's name, lyrics, release date, awards, creative background, and genre.
[0076] Thirdly, a computing device is provided, the terminal including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform operations as described in the first aspect and any possible implementation of the song recommendation method described in the first aspect.
[0077] Fourthly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the instruction being loaded and executed by a processor to perform the operations performed as described in the first aspect and any possible implementation of the song recommendation method described in the first aspect.
[0078] Fifthly, a computer program product is provided, the computer program product storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed as described in the first aspect and any possible implementation of the song recommendation method described in the first aspect.
[0079] The beneficial effects of the technical solution provided in this application are:
[0080] In the technical solution provided in this application, when recommending songs, regardless of whether the song is new or old, it is not necessary to obtain the historical user behavior of the song to be recommended. Only the audio and tag information of the song itself needs to be obtained, and the preference features of the song to be recommended are derived based on this. Then, the preference features of the song to be recommended are compared with the preference features of songs collected by each user. If the similarity between the preference features of the target user's collected songs and the song to be recommended meets the recommendation criteria, it indicates that the target user is very likely to like the song to be recommended. Therefore, the song to be recommended can be recommended to the target user, achieving personalized recommendations of new songs without the need for historical user behavior. Attached Figure Description
[0081] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0082] Figure 1 This is a schematic flowchart of a song recommendation method provided in an embodiment of this application;
[0083] Figure 2 This is a schematic diagram of the network structure of a cross-modal representation model provided in an embodiment of this application;
[0084] Figure 3 This is a schematic diagram of the interface of a music application provided in an embodiment of this application;
[0085] Figure 4 This is a schematic diagram of a model training method provided in an embodiment of this application;
[0086] Figure 5 This is a schematic diagram illustrating an embodiment of obtaining training samples provided in this application;
[0087] Figure 6 This is a schematic flowchart of a song recommendation method provided in an embodiment of this application;
[0088] Figure 7 This is a schematic diagram of a music recommendation device provided in an embodiment of this application;
[0089] Figure 8 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation
[0090] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0091] This application provides a method for song recommendation, implemented by a computing device, such as a server or server cluster. After a new song is created, it needs to be promoted. In this case, the song recommendation method provided in this application can be used to recommend the new song to target users who may like it. Thus, when a target user listens to music through a music application installed on their terminal, if they want to listen to the recommended song, the computing device can send the audio of the new song to the terminal for playback.
[0092] The method for song recommendation provided in this application embodiment will be described below. This method can be applied to computing devices; see [link to relevant documentation]. Figure 1 The method may include the following steps:
[0093] Step 101: Obtain the audio and tag information of the song to be recommended.
[0094] The songs to be recommended can be newly created songs by the singer that have not yet been promoted, songs that have already been promoted, or songs generated by artificial intelligence. The tag information for the songs to be recommended can include at least one of the following: the singer, lyrics, release date, awards, creative background, and genre. The release date can be the year of release.
[0095] In practice, for songs to be recommended, relevant personnel can transmit the audio and tag information of the songs to a computing device.
[0096] Alternatively, relevant personnel can send a recommendation instruction for a song to be recommended to the computing device. This instruction can include the song's tags. The computing device then retrieves the audio and tag information of the song from its music library. The music library can store song-related information such as audio, tag information, lyrics, play count, and download count.
[0097] Step 102: Based on the audio and tag information of the song to be recommended, determine the audio fusion features and tag fusion features of the song to be recommended.
[0098] In practice, the tag information of the song to be recommended can be in text form, described in one sentence, for example: The singer of song 1 is singer A, it was created in 2024, won a first prize in a music competition in 2024, and the background of its creation is heartbreak.
[0099] The computing device can perform feature extraction based on the audio and tag information of the song to be recommended. Specifically, it can use feature extraction algorithms, such as at least one of time-domain analysis, frequency-domain analysis, and MFCC (Mel Frequency Cepstrum Coefficient) analysis, to extract features from the audio of the song to be recommended, obtaining audio fusion features. Similarly, it can use at least one of PAC (Principal Component Analysis) and word embedding algorithms to extract features from the tag information of the song to be recommended, obtaining tag fusion features. In some embodiments, feature extraction can also be performed using a machine learning model. Specifically, the computing device can input the audio and tag information of the song to be recommended into a cross-modal representation model. The cross-modal representation model extracts features from the audio and tag information of the song to be recommended, and then outputs the audio fusion features and tag fusion features of the song to be recommended. Here, the cross-modal representation model is a machine learning model.
[0100] In one possible implementation, the cross-modal representation model can be a dual-tower input structure, including an audio feature representation branch and a label feature representation branch. Accordingly, the processing in step 102 above can be as follows:
[0101] The audio of the song to be recommended is input into the audio feature representation branch, which extracts features from the audio and outputs the audio fusion features of the song to be recommended. The tag information of the song to be recommended is input into the tag feature representation branch, which extracts features from the tag information of the song to be recommended, obtaining the tag fusion features of the song to be recommended. The specific network structures of the audio feature representation branch and the tag feature representation branch can be varied. The following example illustrates the network structures of the audio feature representation branch and the tag feature representation branch:
[0102] See Figure 2 The audio feature representation branch can include an audio encoding module and a first MLP (Multilayer Perceptron) module. The audio encoding module can be a Mert (Music Understanding Model with Large-Scale Self-supervised Training) model. The label feature representation branch can include a text encoding module and a second MLP module. The text encoding module can be a BERT (Bidirectional Encoder Representations from Transformers) model. The first MLP and the second MLP can have the same network structure or different network structures.
[0103] exist Figure 2 Based on the audio feature representation branch shown, the audio of the song to be recommended is input into the audio encoding module, which outputs the audio features of the song to be recommended. The output of the audio encoding module serves as the input to the first MLP module; that is, the audio features of the song to be recommended are input into the first MLP module, which outputs the audio fusion features of the song to be recommended. Both the audio features and the audio fusion features of the song to be recommended can be embedding features.
[0104] exist Figure 2 Based on the label feature representation branch shown, the label information of the song to be recommended is input into the text encoding module, which outputs the label features of the song to be recommended. The output of the text encoding module serves as the input to the second MLP module; that is, the label features of the song to be recommended are input into the second MLP module, which outputs the label fusion features of the song to be recommended. Both the label features and the label fusion features of the song to be recommended can be embedding features.
[0105] Furthermore, the audio fusion features output by the first MLP module and the tag fusion features output by the second MLP module can have the same dimension. For example, both the audio fusion features and the tag fusion features are vectors with a dimension of 1×512.
[0106] In one possible implementation, the cross-modal representation model of the aforementioned dual-tower structure can be trained using contrastive learning, enabling the fusion of cross-modal information between audio fusion features and label fusion features. The training process can be as follows: Multiple sets of first training samples are obtained. Each set of first training samples includes the audio of a first sample song, the label information of the first sample song, and the label information of at least one other song. The first sample song can be any song in the music library, and the at least one other song can be at least one other song in the music library besides the first sample song. Different sets of training samples include different first sample songs. For each set of first training samples, the audio of the first sample song and the label information of the first sample song included in the first training sample are taken as a pair of positive examples, and the audio of the first sample song and the label information of at least one other song included in the first training sample are taken as at least one pair of negative examples. Based on the positive examples and at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain the audio feature representation branch and the label feature representation branch.
[0107] Specifically, for each group of first training samples, the audio of the first sample song is input into the initial audio feature representation branch to obtain the positive example predicted audio features of the first sample song, and the label information of the first sample song is input into the initial label feature representation branch to obtain the positive example predicted label features of the first sample song. The label information of the K other songs included in the negative examples is input into the initial label feature representation branch to obtain K negative example predicted label features. Based on the above positive example predicted audio features, positive example predicted label features, K negative example predicted label features and the loss function of contrastive learning, the loss value corresponding to the first training sample is calculated. Based on the loss value, the initial audio feature representation branch and the initial label feature representation branch are trained. The training objective is to make the positive example predicted audio features and the positive example predicted label features closer together and make the positive example predicted audio features and the negative example predicted label features farther apart. The loss function can be shown in the following formula (1):
[0108]
[0109] in, Let represent the loss value corresponding to the first training sample, log represent the logarithm, exp() represent the natural exponential function, q represent the positive example predicted audio features, and k represent the loss value. + The positive example predicts the label feature. τ is a parameter configured by technicians according to actual needs, used to control the range of values for the loss function. k i Let K represent the predicted label feature of the i-th negative example, and K represent the number of negative examples included in the first training sample.
[0110] Step 103: Determine the preference features of the songs to be recommended based on the audio fusion features and tag fusion features of the songs to be recommended.
[0111] In implementation, after obtaining the audio fusion features and tag fusion features of the song to be recommended through the cross-modal representation model, the audio fusion features and tag fusion features of the song to be recommended are input into the preference feature extraction model, and the preference feature extraction model outputs the preference features of the song to be recommended.
[0112] The preference feature extraction model is a neural network model, specifically, it can be a ResNet (residual neural network). The training of the preference feature extraction model is described in the following examples.
[0113] Step 104: If the similarity between the preference features of the songs collected by the target user account and the preference features of the songs to be recommended meets the recommendation criteria, recommend the songs to be recommended to the target user account.
[0114] In implementation, for songs in the music library, the audio and tag information of the songs can be pre-input into a cross-modal representation model to obtain the audio fusion features and tag fusion features of the songs. Then, the audio fusion features and tag fusion features of the songs are input into a preference feature extraction model to obtain the reference preference features of the songs. In this way, for each user account registered in the music application, the reference preference features of each song in the user account's favorite playlist can be obtained.
[0115] After obtaining the preference features of the songs to be recommended, the similarity between these preference features and the reference preference features of each song in the music library can be calculated. The similarity can be represented by cosine distance, Euclidean distance, etc.
[0116] If the similarity between the reference preference features of each song in the target user account's playlist and the preference features of the song to be recommended meets the recommendation criteria, then the target user account will be used as the recommending user account, and the song to be recommended will be recommended to the target user account.
[0117] There can be multiple recommendation criteria; some of them are illustrated below:
[0118] Recommendation condition one:
[0119] If the similarity between the reference preference features of each song in a user's playlist and the preference features of the song to be recommended is greater than a first threshold, then the recommendation criteria are met. Otherwise, the criteria are not met. The first threshold can be configured by technical personnel according to actual needs.
[0120] Recommendation condition two:
[0121] If the average similarity between the reference preference features of each song in a user's favorite playlist and the preference features of the song to be recommended is greater than a first threshold, then the recommendation criteria are met. Otherwise, the criteria are not met.
[0122] Recommendation condition three:
[0123] If the proportion of songs in a user's favorite playlist whose similarity to the reference preference features and the preference features of the songs to be recommended is greater than a threshold, and this proportion is also greater than a second threshold, then the recommendation criteria are met. Otherwise, the criteria are not met. The second threshold can be configured by technical personnel according to actual needs.
[0124] The following description, in conjunction with the accompanying drawings, provides an exemplary illustration of how a computing device recommends the aforementioned songs to a target user account.
[0125] See Figure 3 The image shows the main interface of a music application, which includes options such as a daily recommendation function and a music radar function. A user opens the music application, logs in to their account, and then clicks the daily recommendation function. The terminal then sends a request to the computing device to retrieve the daily recommended songs, and the computing device sends the audio, lyrics, and other relevant information of the recommended songs to the terminal for playback. Alternatively, the user opens the music application, clicks the music radar function, and the terminal sends a request to the computing device to retrieve radar recommended songs. The computing device then sends the audio, lyrics, and other relevant information of the recommended songs to the terminal for playback.
[0126] In one possible implementation, the training of the above preference feature extraction model can employ a center loss function, see [link to relevant documentation]. Figure 4 Training a preference feature extraction model can include the following steps:
[0127] Step 401: Obtain multiple sets of second training samples.
[0128] Each set of second training samples includes the audio and tag information of multiple second sample songs. These multiple second sample songs are the intersection of the songs collected in the playlists of two users.
[0129] In practice, multiple users' favorite playlists can be obtained, each playlist containing at least one favorite song. For multiple users' favorite playlists, the intersection of any two playlists can be taken.
[0130] For example, such as Figure 5As shown, there are user 1's favorite playlist 1, user 2's favorite playlist 2, and user 3's favorite playlist 3. We can take the intersection 1 of the favorite songs included in favorite playlist 1 and favorite playlist 2, the intersection 2 of the favorite songs included in favorite playlist 1 and favorite playlist 3, and the intersection 3 of the favorite songs included in favorite playlist 2 and favorite playlist 3.
[0131] When finding the intersection, taking playlist 1 and playlist 2 as an example, assuming playlist 1 includes song A, song B, song C and song D, and playlist 2 includes song A, song C, song D and song E, then the intersection of the songs included in playlist 1 and playlist 2 is [song A, song C, song D].
[0132] After finding the intersection, you can further deduplicate the intersection. Specifically, if the first intersection is a subset of the second intersection, then the first intersection is deleted. For example, if the first intersection is [Collected Song A, Collected Song C, Collected Song D], and the second intersection is [Collected Song A, Collected Song C, Collected Song D, Collected Song F, Collected Song K], and the first intersection is a subset of the second intersection, then the first intersection is deleted, and the second intersection is retained.
[0133] Furthermore, to improve the robustness of training, intersections containing fewer than a third threshold of songs in the collection are removed. This third threshold can be configured by technicians according to actual needs. For example, the third threshold could be 2.
[0134] After the above processing, the remaining intersections can be obtained. For each remaining intersection, each collected song included in the intersection is taken as a second sample song, and the audio and tag information of each second sample song is obtained. In this way, for each intersection, the audio and tag information of the second sample songs included in the intersection can be used as a set of second training samples.
[0135] Step 402: For each second sample song in each group of second training samples, input the audio and label information of the second sample song into the cross-modal representation model to obtain the audio fusion feature and label fusion feature of the second sample song.
[0136] In practice, the processing of step 402 is similar to that of step 102 above, and will not be described again here.
[0137] Step 403: Input the audio fusion features and tag fusion features of the second sample song into the preference feature extraction model to obtain the predicted preference features of the second sample song.
[0138] In practice, the processing of step 403 is similar to that of step 103 above, and will not be described again here.
[0139] Step 404: For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine the central predicted preference features corresponding to the second training samples, and calculate the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples.
[0140] In implementation, for each group of second training samples, the center prediction preference features corresponding to the second training samples can be determined first. There are several methods for determining the center prediction preference features; some are illustrated below:
[0141] Method 1:
[0142] Select a center song from the songs in the second training sample. Use the predicted preference features of this center song as the center predicted preference features for the second training sample.
[0143] When selecting the center song, the song with the highest popularity among the songs in the second training sample can be used as the center song. Popularity is used to represent at least one of the following: number of plays, downloads, favorites, and comments.
[0144] Method 2:
[0145] Calculate the average feature of the predicted preference features of the second sample songs included in the second training sample, and use this average feature as the central predicted preference feature corresponding to the second training sample.
[0146] After determining the center prediction preference feature corresponding to the second training sample, the similarity between the center prediction preference feature and the prediction preference feature of at least one second sample song included in the second training sample is calculated. The following explains this process in relation to Method 1 and Method 2 respectively:
[0147] Corresponding method one:
[0148] The similarity between the predicted preference features of the center song and the predicted preference features of the second training samples (excluding the center song) is calculated. The similarity can be Euclidean distance, cosine distance, etc.
[0149] Corresponding method two:
[0150] The similarity between the predicted preference features of the calculation center and the predicted preference features of each second sample song included in the second training sample is calculated.
[0151] Step 405: Based on the similarity between the center predicted preference features corresponding to the second training sample and the predicted preference features of at least one second sample song included in the second training sample, train the initial preference feature extraction model to obtain the preference feature extraction model.
[0152] In implementation, for each second training sample, the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of at least one second sample song included in the second training sample can be accumulated, and the accumulated result can be used as the loss value. The loss function can be shown in the following formula (2):
[0153]
[0154] Where l represents the loss value corresponding to the second training sample, and emb center Emb represents the center prediction preference feature corresponding to the second training sample. i Dis(emb) represents the predicted preference feature of the i-th second sample song in the second training sample. center emb i The expression represents the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of the i-th second sample song in the second training sample. The value of n can be one of the following two cases:
[0155] Case 1: Corresponding to Method 1 above, n represents the number of second sample songs included in the second training sample, excluding the center song.
[0156] Case 2: Corresponding to Method 2 above, n represents the number of second sample songs included in the second training sample.
[0157] Compared with the ternary loss function in related technologies, the central loss function used in this embodiment does not require the selection of negative examples (songs that users do not like) in the training samples, thus avoiding the problems of difficulty in obtaining negative examples and small data volume, resulting in better training effect.
[0158] In one possible implementation, when calculating the loss value, multiple sets of second training samples can be treated as a batch, and a loss value can be calculated for each batch. Accordingly, the loss function can be shown in the following formula (3):
[0159]
[0160] Where L represents the loss value corresponding to a batch, l j This represents the loss value corresponding to the second training sample in the j-th group, and m represents the number of second training samples included in a batch. The value of m can be configured by technical personnel according to actual needs. For example, m can be 5.
[0161] In one possible scenario, when calculating the loss value corresponding to each batch, the loss value corresponding to the second training sample can also be weighted. Accordingly, based on the above formula (3), the loss function can be updated to the following formula (4):
[0162]
[0163] Where, β j This represents the weight corresponding to the second training sample in the j-th group. The weight of the second training sample is positively correlated with the number of second sample songs included in the second training sample. For example, the weight of the second training sample can be the number of second sample songs included in the second training sample. Alternatively, the weight of the second training sample can be a normalized value of the number of second sample songs included in the second training sample.
[0164] When calculating the loss value corresponding to a batch, the number of second sample songs included in the second training sample is taken into account. The larger the number, the better the stability of the corresponding loss value, and the greater its weight can be given in the loss value corresponding to the batch.
[0165] After calculating the loss value corresponding to the second training sample or the batch, the initial preference feature extraction model can be trained based on the loss value. The training goal is to make the predicted preference features of the second sample song closer to the predicted preference features of the center song.
[0166] Figure 6 This is a schematic flowchart of a song recommendation method provided in an embodiment of this application. The following is in conjunction with… Figure 6 The method for song recommendation provided in the embodiments of this application will be described.
[0167] The song recommendation method provided in this application embodiment employs a cross-modal representation model and a preference feature extraction model (such as...). Figure 6 These two neural network models (the new song cold start recommendation model in the text) can be trained before the cross-modal representation model and preference feature extraction model are deployed. See also Figure 6 The cross-modal representation model can have a dual-tower input structure. One branch is the audio feature representation branch, which includes an audio encoding module (as shown in the Mert model) and a first MLP module. The other branch is the label feature representation branch, which includes a text encoding module (as shown in the BERT model) and a second MLP module. When training the cross-modal representation model, the audio of the sample song is input into the Mert model, and the Mert model outputs audio features (such as...). Figure 6The audio embedding is input to the first MLP module, and the first MLP module outputs the audio fusion features (such as audio embedding). Figure 6 The audio embedding 1 in the sample song is input into the BERT model, and the BERT model outputs the label features (e.g., audio embedding 1). Figure 6 The text embedding is input into the second MLP module, and the second MLP module outputs the label fusion features (such as text embedding). Figure 6 The text embedding 2 is used. Based on the audio embedding 1 and the text embedding 2, a contrastive learning method is used to train the two branches. For training the new song cold start recommendation model, multiple users' favorite playlists are obtained, such as... Figure 6 In the context of user 1, user 2, ..., user n, we can filter these playlists and find their intersections to obtain multiple intersecting playlists, such as... Figure 6 The model uses intersecting playlists 1, 2, ..., m to obtain the song audio and tag information of the songs in each playlist. This information is then input into the trained cross-modal representation model to obtain the corresponding fusion features. Finally, based on these fusion features, a weighted loss method is used to train the ResNet to obtain the new song cold start recommendation model. For specific training methods of the new song cold start recommendation model, please refer to [link to relevant documentation]. Figure 4 The training methods shown will not be elaborated further here.
[0168] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0169] In the technical solution provided in this application embodiment, when recommending songs, regardless of whether the song to be recommended is a new song or an old song, it is not necessary to obtain the historical user behavior of the song to be recommended. It is only necessary to obtain the audio and tag information of the song to be recommended itself, and obtain the preference features of the song to be recommended through a cross-modal representation model and a preference feature extraction model. Then, the preference features of the song to be recommended are compared with the preference features of each user's favorite songs. If the similarity between the preference features of the target user's favorite songs and the song to be recommended meets the recommendation conditions, it means that the target user is very likely to like the song to be recommended. Therefore, the song to be recommended can be recommended to the target user, realizing personalized recommendation of new songs without historical user behavior.
[0170] Based on the same technical concept, embodiments of this application also provide a song recommendation device, which can be applied to a computing device, see [link to relevant documentation]. Figure 7 The device may include an acquisition module 610, an inference module 620, and a recommendation module 630, wherein:
[0171] The acquisition module 610 is used to acquire the audio and tag information of the song to be recommended;
[0172] The inference module 620 is used to determine the audio fusion features and tag fusion features of the song to be recommended based on the audio and the tag information; and to determine the preference features of the song to be recommended based on the audio fusion features and the tag fusion features.
[0173] The recommendation module 630 is used to recommend the song to be recommended to the target user account if the similarity between the preference features of the songs collected by the target user account and the preference features of the song to be recommended meets the recommendation conditions.
[0174] In one possible implementation, the cross-modal representation model includes an audio feature representation branch and a label feature representation branch, and the inference module 620 is used for:
[0175] A cross-modal representation model is determined, which includes an audio feature representation branch and a label feature representation branch;
[0176] The audio is input into the audio feature representation branch to obtain the audio fusion features of the song to be recommended;
[0177] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.
[0178] In one possible implementation, the device further includes a training module for:
[0179] Obtain multiple sets of first training samples, wherein each set of first training samples includes the audio of the first sample song, the label information of the first sample song, and the label information of at least one other song;
[0180] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least a pair of negative examples.
[0181] Based on the positive examples and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain the audio feature representation branch and the label feature representation branch.
[0182] In one possible implementation, the inference module is used for:
[0183] The audio fusion features and the tag fusion features are input into the preference feature extraction model to obtain the preference features of the song to be recommended.
[0184] In one possible implementation, the device further includes a training module for:
[0185] Obtain multiple sets of second training samples, wherein each set of second training samples includes the audio and tag information of multiple second sample songs, and the multiple second sample songs are the intersection of the songs collected in the collection playlists of multiple users;
[0186] For each second sample song in each group of second training samples, the audio and label information of the second sample song are input into the cross-modal representation model to obtain the audio fusion feature and the label fusion feature of the second sample song.
[0187] The audio fusion features and tag fusion features of the second sample song are input into the preference feature extraction model to obtain the predicted preference features of the second sample song.
[0188] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, the central predicted preference features corresponding to the second training samples are determined, and the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples is calculated.
[0189] Based on the similarity between the predicted preference features of the center and the predicted preference features of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.
[0190] In one possible implementation, the training module is used for:
[0191] Select the center song from the second sample songs included in the second training sample;
[0192] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.
[0193] In one possible implementation, the training module is used for:
[0194] The second sample song with the highest popularity in the second training sample is taken as the center song of the second training sample, wherein the popularity is used to represent at least one of the song's play count, download count, and collection count.
[0195] In one possible implementation, the training module is used for:
[0196] Calculate the average feature of the predicted preference features of the second sample songs included in the second training sample;
[0197] The average feature is used as the center prediction preference feature corresponding to the second training sample.
[0198] In one possible implementation, the training module is used for:
[0199] For each group of second training samples, the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of at least one second sample song included in the second training sample is accumulated to obtain the accumulated result.
[0200] The accumulated result is multiplied by the weight corresponding to the second training sample to obtain the product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;
[0201] Based on the product result, calculate the loss value;
[0202] Based on the loss value, the initial preference feature extraction model is trained.
[0203] In one possible implementation, the tag information of the song to be recommended includes at least one of the following: the artist's name, lyrics, release date, awards, creative background, and genre.
[0204] In the technical solution provided in this application embodiment, when recommending songs, regardless of whether the song to be recommended is a new song or an old song, it is not necessary to obtain the historical user behavior of the song to be recommended. Only the audio and tag information of the song to be recommended itself needs to be obtained. Based on this, the preference features of the song to be recommended are obtained. Then, the preference features of the song to be recommended are compared with the preference features of each user's favorite songs. If the similarity between the preference features of the target user's favorite songs and the song to be recommended meets the recommendation conditions, it means that the target user is very likely to like the song to be recommended. Therefore, the song to be recommended can be recommended to the target user, thus realizing personalized recommendation of new songs without historical user behavior.
[0205] It should be noted that the song recommendation device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. In addition, the song recommendation device and the song recommendation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0206] Figure 8 This is a schematic diagram of a computing device 1000 provided in an embodiment of this application. The computing device 1000 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 1001 and one or more memories 1002. The memories 1002 store at least one instruction, which is loaded and executed by the processors 1001 to implement the methods provided in the various method embodiments described above. Of course, the computing device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computing device may also include other components for implementing device functions, which will not be elaborated upon here.
[0207] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the song recommendation method described above. This computer-readable storage medium may be non-transitory. For example, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.
[0208] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the user's favorite playlists, favorite songs, audio, tag information, etc. involved in this application were all obtained with full authorization.
[0209] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0210] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for song recommendation, characterized in that, The method includes: Obtain multiple sets of second training samples, wherein each set of second training samples includes the audio and tag information of multiple second sample songs, and the multiple second sample songs are the intersection of the songs collected in the collection playlists of multiple users; For each second sample song in each group of second training samples, the audio and label information of the second sample song are input into the cross-modal representation model to obtain the audio fusion feature and the label fusion feature of the second sample song. The audio fusion features and tag fusion features of the second sample song are input into the preference feature extraction model to obtain the predicted preference features of the second sample song. For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, the central predicted preference features corresponding to the second training samples are determined, and the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples is calculated. For each group of second training samples, the similarity between the center prediction preference feature corresponding to the second training sample and the prediction preference feature of at least one second sample song included in the second training sample is accumulated to obtain the accumulated result. The accumulated result is multiplied by the weight corresponding to the second training sample to obtain the product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample; Based on the product result, calculate the loss value; Based on the loss value, the initial preference feature extraction model is trained to obtain the preference feature extraction model; Obtain the audio and tag information of the songs to be recommended; Based on the audio and the tag information, determine the audio fusion features and tag fusion features of the song to be recommended; The audio fusion features and the tag fusion features are input into the preference feature extraction model to obtain the preference features of the song to be recommended. If the similarity between the song preference characteristics of the target user account and the song preference characteristics of the song to be recommended meets the recommendation criteria, the song to be recommended is recommended to the target user account.
2. The method according to claim 1, characterized in that, The step of determining the audio fusion features and tag fusion features of the song to be recommended based on the audio and the tag information includes: A cross-modal representation model is determined, which includes an audio feature representation branch and a label feature representation branch; The audio is input into the audio feature representation branch to obtain the audio fusion features of the song to be recommended; The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.
3. The method according to claim 2, characterized in that, Before inputting the audio into the audio feature representation branch and the label information into the label feature representation branch, the method further includes: Obtain multiple sets of first training samples, wherein each set of first training samples includes the audio of the first sample song, the label information of the first sample song, and the label information of at least one other song; For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least a pair of negative examples. Based on the positive examples and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain the audio feature representation branch and the label feature representation branch.
4. The method according to claim 1, characterized in that, The step of determining the central prediction preference features corresponding to the second training sample based on the prediction preference features of the second sample songs included in the second training sample includes: Select the center song from the second sample songs included in the second training sample; The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.
5. The method according to claim 4, characterized in that, The step of selecting the center song in the second training sample includes: The second sample song with the highest popularity in the second training sample is taken as the center song of the second training sample, wherein the popularity is used to represent at least one of the song's play count, download count, and collection count.
6. The method according to claim 1, characterized in that, The step of determining the central prediction preference features corresponding to the second training sample based on the prediction preference features of the second sample songs included in the second training sample includes: Calculate the average feature of the predicted preference features of the second sample songs included in the second training sample; The average feature is used as the center prediction preference feature corresponding to the second training sample.
7. The method according to any one of claims 1-6, characterized in that, The tag information of the song to be recommended includes at least one of the following: the artist's name, lyrics, release date, awards, creative background, and genre.
8. A computing device, characterized in that, The computing device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operations performed by the song recommendation method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operations of the song recommendation method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded and executed by a processor to perform the operations of the song recommendation method as described in any one of claims 1 to 7.