Song recommendation method and device and storage medium

Through the cross-modal representation model and preference feature extraction model, song features are extracted and compared with user preference features, the problem of new song recommendation is solved, personalized recommendation is achieved, and the accuracy and user experience of recommendations are improved.

CN119025703AActive Publication Date: 2024-11-26TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411168924.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-26
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

It is difficult to personalize the recommendation method of existing songs without historical user behavior in new songs.

Method used

By obtaining the audio and tag information of the song to be recommended, using the cross-modal representation model and preference feature extraction model, the audio fusion and tag fusion characteristics of the song are extracted, the preference characteristics of the song are calculated, and the preference characteristics of the user's favorite songs are compared with the preference characteristics of the user's favorite songs Yes, recommended to users who may like the song.

Benefits of technology

It realizes personalized recommendations of new songs without historical user behavior, improving the accuracy and user experience of song recommendations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025703A_ABST
    Figure CN119025703A_ABST
Patent Text Reader

Abstract

The invention provides a song recommendation method and device and a storage medium, and relates to the technical field of audio, and the method comprises the steps: determining an audio fusion feature and a label fusion feature of a to-be-recommended song based on the audio and label information of the to-be-recommended song, determining a preference feature of the to-be-recommended song based on the audio fusion feature and the label fusion feature, and recommending the to-be-recommended song according to the preference feature. And calculating the similarity between the preference characteristics of the song to be recommended and the preference characteristics of other songs in the song library. And under the condition that the similarity between the preference characteristics of the collected songs of the target user and the preference characteristics of the to-be-recommended songs meets recommendation conditions, recommending the to-be-recommended songs to the target user. By adopting the method and the device, personalized recommendation of songs can be realized, and the songs can be recommended to users who possibly like the songs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a song recommendation method, device and storage medium. Background Art

[0002] Song recommendation is one of the important channels for users to listen to songs, and music recommendation is also an important way to promote songs.

[0003] At present, song recommendations mainly adopt collaborative filtering, which recommends songs to other users with similar preferences as the target user based on the target user's historical user behaviors such as playing and collecting the songs, so as to achieve song recommendations based on user preferences.

[0004] However, for new songs, since they have not been promoted before, there is no historical user behavior. In this case, the above song recommendation method is no longer applicable. Based on this, a method for personalized recommendation of new songs is urgently needed. Summary of the invention

[0005] The embodiment of the present application provides a song recommendation method, device and storage medium, which can realize personalized recommendation of songs and recommend songs to appropriate users. The technical solution is as follows:

[0006] In a first aspect, a method for song recommendation is provided, the method comprising:

[0007] Get the audio and tag information of the song to be recommended;

[0008] Based on the audio and the tag information, determine the audio fusion feature and the tag fusion feature of the song to be recommended;

[0009] Determining the preference features of the song to be recommended based on the audio fusion features and the tag fusion features;

[0010] When the similarity between the preference characteristics of the favorite songs of the target user account and the preference characteristics of the song to be recommended meets the recommendation condition, the song to be recommended is recommended to the target user account.

[0011] In one possible implementation,

[0012] The step of determining the audio fusion feature and the tag fusion feature of the song to be recommended based on the audio and the tag information includes:

[0013] Determine a cross-modal representation model, wherein the cross-modal representation model includes an audio feature representation branch and a label feature representation branch;

[0014] Input the audio into the audio feature representation branch to obtain the audio fusion feature of the song to be recommended;

[0015] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.

[0016] In a possible implementation, before the audio is input into the audio feature characterization branch and the tag information is input into the tag feature characterization branch, the method further includes:

[0017] Acquire multiple groups of first training samples, wherein each group of first training samples includes audio of a first sample song, label information of the first sample song, and label information of at least one other song;

[0018] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least one pair of negative examples;

[0019] Based on the positive example and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through comparative learning to obtain the audio feature representation branch and the label feature representation branch.

[0020] In a possible implementation, determining the preference feature of the song to be recommended based on the audio fusion feature and the tag fusion feature includes:

[0021] The audio fusion feature and the label fusion feature are input into a preference feature extraction model to obtain the preference feature of the song to be recommended.

[0022] In a possible implementation, before inputting the audio fusion feature and the label fusion feature into the preference feature extraction model, the method further includes:

[0023] Acquire multiple groups of second training samples, wherein each group of second training samples includes audio and label information of multiple second sample songs, and the multiple second sample songs are intersections of favorite songs included in favorite song lists of multiple users;

[0024] For each second sample song in each group of second training samples, input the audio of the second sample song and the label information of the second sample song into the cross-modal representation model to obtain the audio fusion feature of the second sample song and the label fusion feature of the second sample song;

[0025] Inputting the audio fusion feature of the second sample song and the label fusion feature of the second sample song into the preference feature extraction model to obtain the predicted preference feature of the second sample song;

[0026] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine a central predicted preference feature corresponding to the second training samples, and calculate a similarity between the central predicted preference feature and a predicted preference feature of at least one second sample song included in the second training samples;

[0027] Based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.

[0028] In a possible implementation, determining the central predicted preference feature corresponding to the second training sample based on the predicted preference feature of the second sample song included in the second training sample includes:

[0029] Selecting a central song from the second sample songs included in the second training sample;

[0030] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.

[0031] In a possible implementation, the selecting a central song in the second training sample includes:

[0032] The second sample song with the highest popularity in the second training sample is used as the central song of the second training sample, wherein the popularity is used to characterize at least one of the number of plays, the number of downloads, and the number of collections of the song.

[0033] In a possible implementation, determining the central predicted preference feature corresponding to the second training sample based on the predicted preference feature of the second sample song included in the second training sample includes:

[0034] Calculating an average feature of the predicted preference features of the second sample songs included in the second training sample;

[0035] The average feature is used as the central prediction preference feature corresponding to the second training sample.

[0036] In a possible implementation, the training of the initial preference feature extraction model based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample to obtain the preference feature extraction model includes:

[0037] For each group of second training samples, accumulating the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample to obtain an accumulated result;

[0038] Multiplying the accumulated result by the weight corresponding to the second training sample to obtain a product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;

[0039] Based on the multiplication result, calculating the loss value;

[0040] Based on the loss value, an initial preference feature extraction model is trained.

[0041] In a possible implementation, the tag information of the song to be recommended includes at least one of the singer's name, lyrics, release time, award information, creation background, and style of the song to be recommended.

[0042] In a second aspect, a device for recommending songs is provided, the device comprising:

[0043] The acquisition module is used to obtain the audio and tag information of the song to be recommended;

[0044] An inference module, used to determine the audio fusion feature and the label fusion feature of the song to be recommended based on the audio and the label information; and determine the preference feature of the song to be recommended based on the audio fusion feature and the label fusion feature;

[0045] The recommendation module is used to recommend the song to be recommended to the target user account when the similarity between the preference characteristics of the favorite songs of the target user account and the preference characteristics of the song to be recommended meets the recommendation conditions.

[0046] In a possible implementation, the cross-modal representation model includes an audio feature representation branch and a label feature representation branch, and the reasoning module is used to:

[0047] Determine a cross-modal representation model, wherein the cross-modal representation model includes an audio feature representation branch and a label feature representation branch;

[0048] Input the audio into the audio feature representation branch to obtain the audio fusion feature of the song to be recommended;

[0049] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.

[0050] In a possible implementation, the apparatus further includes a training module, configured to:

[0051] Acquire multiple groups of first training samples, wherein each group of first training samples includes audio of a first sample song, label information of the first sample song, and label information of at least one other song;

[0052] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least one pair of negative examples;

[0053] Based on the positive example and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through comparative learning to obtain the audio feature representation branch and the label feature representation branch.

[0054] In a possible implementation, the reasoning module is used to:

[0055] The audio fusion feature and the label fusion feature are input into a preference feature extraction model to obtain the preference feature of the song to be recommended.

[0056] In a possible implementation, the apparatus further includes a training module, configured to:

[0057] Acquire multiple groups of second training samples, wherein each group of second training samples includes audio and label information of multiple second sample songs, and the multiple second sample songs are intersections of favorite songs included in favorite song lists of multiple users;

[0058] For each second sample song in each group of second training samples, input the audio of the second sample song and the label information of the second sample song into the cross-modal representation model to obtain the audio fusion feature of the second sample song and the label fusion feature of the second sample song;

[0059] Inputting the audio fusion feature of the second sample song and the label fusion feature of the second sample song into the preference feature extraction model to obtain the predicted preference feature of the second sample song;

[0060] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine a central predicted preference feature corresponding to the second training samples, and calculate a similarity between the central predicted preference feature and a predicted preference feature of at least one second sample song included in the second training samples;

[0061] Based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.

[0062] In a possible implementation, the training module is used to:

[0063] Selecting a central song from the second sample songs included in the second training sample;

[0064] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.

[0065] In a possible implementation, the training module is used to:

[0066] The second sample song with the highest popularity in the second training sample is used as the central song of the second training sample, wherein the popularity is used to characterize at least one of the number of plays, the number of downloads, and the number of collections of the song.

[0067] In a possible implementation, the training module is used to:

[0068] Calculating an average feature of the predicted preference features of the second sample songs included in the second training sample;

[0069] The average feature is used as the central prediction preference feature corresponding to the second training sample.

[0070] In a possible implementation, the training module is used to:

[0071] For each group of second training samples, accumulating the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample to obtain an accumulated result;

[0072] Multiplying the accumulated result by the weight corresponding to the second training sample to obtain a product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;

[0073] Based on the multiplication result, calculating the loss value;

[0074] Based on the loss value, an initial preference feature extraction model is trained.

[0075] In a possible implementation, the tag information of the song to be recommended includes at least one of the singer's name, lyrics, release time, award information, creation background, and style of the song to be recommended.

[0076] According to a third aspect, a computing device is provided, wherein the terminal includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the song recommendation method as described in the first aspect and any possible implementation of the first aspect.

[0077] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the song recommendation method as described in the first aspect and any possible implementation of the first aspect.

[0078] In a fifth aspect, a computer program product is provided, wherein at least one instruction is stored in the computer program product, and the instruction is loaded and executed by a processor to implement the operations performed by the song recommendation method as described in the first aspect and any possible implementation of the first aspect.

[0079] The beneficial effects of the technical solution provided by this application are:

[0080] In the technical solution provided by the present application, when recommending songs to be recommended, no matter whether the songs to be recommended are new songs or old songs, there is no need to obtain the historical user behavior of the songs to be recommended. It is only necessary to obtain the audio and tag information of the songs to be recommended, and based on this, the preference characteristics of the songs to be recommended are obtained. Then, the preference characteristics of the songs to be recommended are compared with the preference characteristics of the songs collected by each user. If the similarity between the preference characteristics of the songs collected by the target user and the songs to be recommended meets the recommendation conditions, it means that the target user is likely to like the songs to be recommended. Then, the songs to be recommended can be recommended to the target user, realizing personalized recommendation of new songs without historical user behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 This is a flowchart of a song recommendation method provided by an embodiment of the present application;

[0083] Figure 2 is a schematic diagram of a network structure of a cross-modal representation model provided in an embodiment of the present application;

[0084] Figure 3 is a schematic diagram of an interface of a music application provided in an embodiment of the present application;

[0085] Figure 4 It is a flow chart of a method for model training provided in an embodiment of the present application;

[0086] Figure 5 is a schematic diagram of obtaining a training sample provided in an embodiment of the present application;

[0087] Figure 6 This is a flowchart of a song recommendation method provided by an embodiment of the present application;

[0088] Figure 7 is a schematic diagram of a music recommendation device structure provided in an embodiment of the present application;

[0089] Figure 8 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0091] The embodiment of the present application provides a song recommendation method, which is implemented by a computing device, which may be a server, a server cluster, etc. After a new song is created, it is necessary to promote the new song. In this case, the song recommendation method provided by the embodiment of the present application can be used to recommend the new song to target users who may like it. In this way, when the target user listens to a song through a music application installed on a terminal, if he wants to listen to a recommended song, the computing device can send the audio of the new song to the terminal for playback.

[0092] The following is an explanation of the song recommendation method provided in the embodiment of the present application. The method can be applied to a computing device, see Figure 1 , the method may include the following steps:

[0093] Step 101: Obtain the audio and tag information of the song to be recommended.

[0094] The songs to be recommended may be newly created songs by singers that have not been promoted yet, songs that have been promoted, or songs generated by artificial intelligence. The tag information of the songs to be recommended may include at least one of the singer, lyrics, release time, award information, creation background, and style of the songs to be recommended. The release time may be the year of release.

[0095] During implementation, for songs to be recommended, relevant personnel can transmit the audio and tag information of the songs to be recommended to a computing device.

[0096] Alternatively, a relevant person may send a recommendation instruction of a song to be recommended to a computing device, wherein the recommendation instruction may carry a tag of the song to be recommended. Then, the computing device obtains the audio and tag information of the song to be recommended from the music library. The music library may store song-related information such as the audio, tag information, lyrics, playback volume, and download volume of the song.

[0097] Step 102: Based on the audio and tag information of the song to be recommended, determine the audio fusion features and tag fusion features of the song to be recommended.

[0098] In implementation, the label information of the songs to be recommended can be in text form and described in one sentence, for example: Song 1 is sung by singer A, created in 2024, won the first prize in a certain music award in 2024, and the background of its creation is a broken heart.

[0099] The computing device can perform feature extraction based on the audio and label information of the song to be recommended, specifically based on a feature extraction algorithm, such as at least one of time domain analysis, frequency domain analysis, MFCC (Mel Frequency Cepstrum Coefficient) analysis and other algorithms, to perform feature extraction on the audio of the song to be recommended, and obtain the audio fusion feature of the song to be recommended; perform feature extraction on the label information of the song to be recommended by at least one of PAC (Principal Component Analysis), word embedding and other algorithms, and obtain the label fusion feature of the song to be recommended. In some embodiments, feature extraction can also be performed through a machine learning model, and specifically the computing device can input the audio and label information of the song to be recommended into a cross-modal representation model, and the cross-modal representation model performs feature extraction on the audio and label information of the song to be recommended, and then outputs the audio fusion feature and label fusion feature of the song to be recommended. Among them, the cross-modal representation model is a machine learning model.

[0100] In a possible implementation, the cross-modal representation model may be a dual-tower input structure, including an audio feature representation branch and a label feature representation branch. Accordingly, the processing of step 102 may be as follows:

[0101] The audio of the song to be recommended is input into the audio feature representation branch, which extracts features from the audio of the song to be recommended and outputs the audio fusion features of the song to be recommended. The tag information of the song to be recommended is input into the tag feature representation branch, which extracts features from the tag information of the song to be recommended and obtains the tag fusion features of the song to be recommended. There can be many specific network structures for the audio feature representation branch and the tag feature representation branch. The following is an example of the network structure of the audio feature representation branch and the tag feature representation branch:

[0102] See also Figure 2 The audio feature representation branch may include an audio encoding module and a first MLP (Multilayer Perceptron) module, wherein the audio encoding module may be a MERTS (Music Understanding Model with Large-Scale Self-supervised Training). The label feature representation branch may include a text encoding module and a second MLP module, wherein the text encoding module may be a BERT (Bidirectional Encoder Representations from Transformers) model, and the first MLP and the second MLP may have the same network structure or different network structures.

[0103] exist Figure 2 Based on the audio feature characterization branch shown, the audio of the song to be recommended is input into the audio encoding module, and the audio encoding module outputs the audio features of the song to be recommended. The output of the audio encoding module is used as the input of the first MLP module, that is, the audio features of the song to be recommended are input into the first MLP module, and the first MLP module outputs the audio fusion features of the song to be recommended. Among them, the audio features and audio fusion features of the song to be recommended can both be embedding features.

[0104] exist Figure 2 Based on the label feature representation branch shown, the label information of the song to be recommended is input into the text encoding module, and the text encoding module outputs the label features of the song to be recommended. The output of the text encoding module is used as the input of the second MLP module, that is, the label features of the song to be recommended are input into the second MLP module, and the second MLP module outputs the label fusion features of the song to be recommended. Among them, the label features and label fusion features of the song to be recommended can both be embedding features.

[0105] In addition, the audio fusion features output by the first MLP module and the label fusion features output by the second MLP module may have the same dimension. Exemplarily, both the audio fusion features and the label fusion features are vector dimensions of 1×512.

[0106] In a possible implementation, for the cross-modal representation model of the above-mentioned dual-tower structure, contrastive learning can be used for training, so that cross-modal information is blended between audio fusion features and label fusion features. The training process can be as follows: obtain multiple groups of first training samples, wherein each group of first training samples includes the audio of a first sample song, the label information of the first sample song, and the label information of at least one other song, the first sample song can be any song in the music library, and the at least one other song can be at least one other song in the music library except the first sample song, and the first sample songs included in different groups of training samples are different. For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are used as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are used as at least one pair of negative examples. Based on the positive examples and at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through contrastive learning to obtain an audio feature representation branch and a label feature representation branch.

[0107] Specifically, for each group of first training samples, the audio of the first sample song is input into the initial audio feature characterization branch to obtain the positive example predicted audio features of the first sample song, and the label information of the first sample song is input into the initial label feature characterization branch to obtain the positive example predicted label features of the first sample song. The label information of the K other songs included in the negative examples are respectively input into the initial label feature characterization branch to obtain K negative example predicted label features. Based on the above-mentioned positive example predicted audio features, positive example predicted label features, K negative example predicted label features and the loss function of contrastive learning, the loss value corresponding to the first training sample is calculated. Based on the loss value, the initial audio feature characterization branch and the initial label feature characterization branch are trained, and the training goal is to make the positive example predicted audio features and the positive example predicted label features closer, and to make the positive example predicted audio features and the negative example predicted label features farther. The loss function can be shown as follows in formula (1):

[0108]

[0109] in, represents the loss value corresponding to the first training sample, log represents the logarithm, exp() represents the natural exponential function, q represents the audio feature of the positive example prediction, and k + represents the positive example prediction label feature, τ is a parameter configured by the technician according to actual needs to control the value range of the loss function, and k i represents the i-th negative example prediction label feature, and K represents the number of negative examples included in the first training sample.

[0110] Step 103: Determine the preference features of the song to be recommended based on the audio fusion features and label fusion features of the song to be recommended.

[0111] In implementation, after obtaining the audio fusion features and label fusion features of the song to be recommended through the cross-modal representation model, the audio fusion features and label fusion features of the song to be recommended are input into the preference feature extraction model, and the preference feature extraction model outputs the preference features of the song to be recommended.

[0112] The preference feature extraction model is a neural network model, specifically, a resnet (residual neural network). The training of the preference feature extraction model is described in the following embodiment.

[0113] Step 104: When the similarity between the preference characteristics of the favorite songs of the target user account and the preference characteristics of the songs to be recommended meets the recommendation conditions, the songs to be recommended are recommended to the target user account.

[0114] In implementation, for songs in the music library, the audio and tag information of the songs can be input into the cross-modal representation model in advance to obtain the audio fusion features and tag fusion features of the songs. Then, the audio fusion features and tag fusion features of the songs are input into the preference feature extraction model to obtain the reference preference features of the songs. In this way, for each user account registered in the music application, the reference preference features of each favorite song in the favorite playlist of the user account can be obtained.

[0115] After obtaining the preference features of the songs to be recommended, the similarity between the preference features of the songs to be recommended and the reference preference features of each song in the music library can be calculated. The similarity can be expressed by cosine distance, Euclidean distance, etc.

[0116] If the similarity between the reference preference features of each favorite song in the favorite playlist of the target user account and the preference features of the song to be recommended meets the recommendation conditions, the target user account will be used as the recommended user account, and the song to be recommended will be recommended to the target user account.

[0117] There may be multiple recommendation conditions, and several of them are listed below for illustration:

[0118] Recommended condition 1:

[0119] If it is determined that the similarity between the reference preference characteristics of each favorite song in the favorite song list of the user account and the preference characteristics of the song to be recommended is greater than the first threshold, it is determined that the recommendation condition is met. Otherwise, it is determined that the condition is not met. The first threshold can be configured by the technician according to actual needs.

[0120] Recommended condition 2:

[0121] If it is determined that the average value of the similarity between the reference preference features of each favorite song in the favorite song list of the user account and the preference features of the song to be recommended is greater than the first threshold, it is determined that the recommendation condition is met. Otherwise, it is determined that the condition is not met.

[0122] Recommended condition three:

[0123] If it is determined that the proportion of the collection songs whose similarity between the corresponding reference preference features in the collection song list of the user account and the preference features of the songs to be recommended is greater than the threshold and is greater than the second threshold, it is determined that the recommendation condition is met. Otherwise, it is determined that the condition is not met. The second threshold can be configured by the technician according to actual needs.

[0124] The following is an exemplary description of the manner in which the computing device recommends the above-mentioned songs to be recommended to the target user account in conjunction with the accompanying drawings.

[0125] See also Figure 3 , shows the main interface of a music application, which includes daily recommendation function options, music radar function options, etc. The user opens the music application and logs in to the target account, then clicks the daily recommendation function option in the music application, and then the terminal sends a daily recommended song acquisition request to the computing device, and the computing device can send the audio, lyrics and other related information of the above-mentioned songs to be recommended to the terminal for playback. Alternatively, the user opens the music application and clicks the music radar function option in the music application, and then the terminal sends a radar recommended song acquisition request to the computing device, and the computing device can send the audio, lyrics and other related information of the above-mentioned songs to be recommended to the terminal for playback.

[0126] In one possible implementation, the training of the above preference feature extraction model can adopt the central loss function, see Figure 4 , the training of the preference feature extraction model may include the following steps:

[0127] Step 401: Obtain multiple groups of second training samples.

[0128] Each group of second training samples includes audio and label information of multiple second sample songs, and the multiple second sample songs are the intersection of the collection songs included in the collection playlists of two users.

[0129] In implementation, the favorite playlists of multiple users may be obtained, each favorite playlist including at least one favorite song. For the favorite playlists of multiple users, the intersection of every two favorite playlists may be taken.

[0130] For example, Figure 5As shown, there are favorite song list 1 of user 1, favorite song list 2 of user 2 and favorite song list 3 of user 3. You can take the intersection 1 of the favorite songs included in favorite song list 1 and favorite song list 2, take the intersection 2 of the favorite songs included in favorite song list 1 and favorite song list 3, and take the intersection 3 of the favorite songs included in favorite song list 2 and favorite song list 3.

[0131] When taking the intersection, taking favorite song list 1 and favorite song list 2 as an example, assuming that favorite song list 1 includes favorite song A, favorite song B, favorite song C and favorite song D, and favorite song list 2 includes favorite song A, favorite song C, favorite song D and favorite song E, then the intersection of the favorite songs included in favorite song list 1 and favorite song list 2 is [favorite song A, favorite song C, favorite song D].

[0132] After the intersection is taken, the intersection can also be deduplicated. Specifically, if the first intersection is a subset of the second intersection, the first intersection is deleted. For example, if the first intersection is [Collected Songs A, Collected Songs C, Collected Songs D], and the second intersection is [Collected Songs A, Collected Songs C, Collected Songs D, Collected Songs F, Collected Songs K], and the first intersection is a subset of the second intersection, the first intersection is deleted and the second intersection is retained.

[0133] In addition, in order to improve the robustness of the training, the intersections whose number of favorite songs is less than the third threshold are deleted, wherein the third threshold can be configured by the technician according to actual needs. For example, the third threshold is 2.

[0134] After the above processing, the remaining intersections can be obtained. For each remaining intersection, each collection song included in the intersection is taken as a second sample song, and the audio and label information of each second sample song is obtained. In this way, for each intersection, the audio and label information of the second sample song included in the intersection can be taken as a set of second training samples.

[0135] Step 402: For each second sample song in each group of second training samples, the audio and label information of the second sample song are input into the cross-modal representation model to obtain the audio fusion feature and label fusion feature of the second sample song.

[0136] In practice, the processing of step 402 is similar to the processing of step 102 described above, and will not be described in detail herein.

[0137] Step 403: Input the audio fusion features and label fusion features of the second sample song into the preference feature extraction model to obtain the predicted preference features of the second sample song.

[0138] In practice, the processing of step 403 is similar to the processing of step 103 above, and will not be described in detail here.

[0139] Step 404: For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine the central predicted preference features corresponding to the second training samples, and calculate the similarity between the central predicted preference features and the predicted preference features of at least one second sample song included in the second training samples.

[0140] In implementation, for each group of second training samples, the center prediction preference feature corresponding to the second training sample may be determined first. There may be multiple methods for determining the center prediction preference feature, and several of them are listed below for illustration:

[0141] Method 1:

[0142] A central song is selected from the second sample songs included in the second training sample, and the predicted preference feature of the central song is used as the central predicted preference feature corresponding to the second training sample.

[0143] When selecting the core song, the song with the highest popularity among the second sample songs included in the second training sample can be used as the core song. The popularity is used to represent at least one of the number of plays, downloads, collections, and comments of the song.

[0144] Method 2:

[0145] The average feature of the predicted preference features of the second sample songs included in the second training sample is calculated, and the average feature is used as the central predicted preference feature corresponding to the second training sample.

[0146] After determining the central prediction preference feature corresponding to the second training sample, the similarity between the central prediction preference feature and the prediction preference feature of at least one second sample song included in the second training sample is calculated. The following is an explanation of the processing corresponding to the above method 1 and method 2 respectively:

[0147] Corresponding method 1:

[0148] The similarity between the central predicted preference feature and the predicted preference features of the second sample songs included in the second training sample other than the central song is calculated, wherein the similarity may be a Euclidean distance, a cosine distance, or the like.

[0149] Corresponding method 2:

[0150] The similarity between the central predicted preference feature and the predicted preference feature of each second sample song included in the second training sample is calculated.

[0151] Step 405: Based on the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain a preference feature extraction model.

[0152] In implementation, for each second training sample, the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample can be accumulated, and the accumulated result is used as the loss value. The loss function can be shown as the following formula (2):

[0153]

[0154] Among them, l represents the loss value corresponding to the second training sample, emb center represents the central prediction preference feature corresponding to the second training sample, emb i represents the predicted preference feature of the i-th second sample song in the second training sample, dis(emb center ,emb i ) represents the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of the i-th second sample song in the second training sample. The value of n can be as follows:

[0155] Case 1: Corresponding to the above method 1, n represents the number of second sample songs included in the second training sample except the central song.

[0156] Case 2: Corresponding to the above method 2, n represents the number of second sample songs included in the second training sample.

[0157] Compared with the ternary loss function in the related art, the central loss function adopted in the embodiment of the present application does not need to include negative examples (songs that users do not like) in the selected training samples, thus avoiding the difficulty in obtaining negative examples and the problem of small data volume, thereby achieving better training effect.

[0158] In a possible implementation, when calculating the loss value, multiple groups of second training samples can be taken as a batch, and a loss value is calculated for each batch. Accordingly, the loss function can be shown as the following formula (3):

[0159]

[0160] Among them, L represents the loss value corresponding to a batch, l j represents the loss value corresponding to the jth group of second training samples, m represents the number of groups of second training samples included in a batch, and the value of m can be configured by technical personnel according to actual needs. For example, m is 5.

[0161] In one possible case, when calculating the loss value corresponding to each batch, the loss value corresponding to the second training sample can also be weighted. Accordingly, based on the above formula (3), the loss function can be updated to the following formula (4):

[0162]

[0163] Among them, β j represents the weight corresponding to the jth group of second training samples, and the weight corresponding to the second training sample is positively correlated with the number of second sample songs included in the second training sample. For example, the value of the weight corresponding to the second training sample can be the number of second sample songs included in the second training sample. For another example, the value of the weight corresponding to the second training sample can be the normalized value of the number of second sample songs included in the second training sample.

[0164] When calculating the loss value corresponding to the batch, the number of second sample songs included in the second training sample is taken into account. The larger the number, the better the stability of the corresponding loss value, which can make it occupy a larger weight in the loss value corresponding to the batch.

[0165] After calculating the loss value corresponding to the second training sample or the loss value corresponding to the batch, the initial preference feature extraction model can be trained based on the loss value. The training goal is to make the predicted preference features of the second sample song closer to the predicted preference features of the central song.

[0166] Figure 6 This is a flowchart of a song recommendation method provided by an embodiment of the present application. Figure 6 The song recommendation method provided in the embodiment of the present application is described.

[0167] The song recommendation method provided in the embodiment of the present application adopts a cross-modal representation model and a preference feature extraction model (such as Figure 6 The two neural network models are the new song cold start recommendation model in . Before the cross-modal representation model and the preference feature extraction model are put into use, they can be trained first. Figure 6 The cross-modal representation model can be a dual-tower input structure, in which one branch is an audio feature representation branch, including an audio encoding module (such as the Mert model in the figure) and a first MLP module, and the other branch is a label feature representation branch, including a text encoding module (such as the Bert model in the figure) and a second MLP module. When training the cross-modal representation model, the audio of the song as a sample is input to the Mert model, and the Mert model outputs audio features (such as Figure 6The audio embedding in the audio is input to the first MLP module, and the first MLP module outputs the audio fusion feature (such as Figure 6 The label information of the song as a sample is input into the BERT model, and the BERT model outputs label features (such as Figure 6 The text embedding in the text is input to the second MLP module, and the second MLP module outputs the label fusion feature (such as Figure 6 According to audio embedding1 and text embedding2, the two branches are trained using contrastive learning. For the training of the new song cold start recommendation model, the favorite song lists of multiple users are obtained, such as Figure 6 In the example, user 1 collects a playlist, user 2 collects a playlist, and user n collects a playlist. Then, these playlists are filtered and their intersections are taken to obtain multiple intersection playlists, such as Figure 6 Then, the song audio and label information of the songs in the intersection playlists are obtained and input into the trained cross-modal representation model to obtain the corresponding fusion features. Then, according to the obtained fusion features, the weighted loss method is used to train Resnet to obtain the new song cold start recommendation model. Here, the specific training method of the new song cold start recommendation model can be referred to Figure 4 The training method shown will not be repeated here.

[0168] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0169] In the technical solution provided by the embodiment of the present application, when recommending songs to be recommended, no matter whether the songs to be recommended are new songs or old songs, there is no need to obtain the historical user behavior of the songs to be recommended. It is only necessary to obtain the audio and tag information of the songs to be recommended, and obtain the preference features of the songs to be recommended through the cross-modal representation model and the preference feature extraction model. Then, the preference features of the songs to be recommended are compared with the preference features of the songs collected by each user. When the similarity between the preference features of the target user's collected songs and the songs to be recommended meets the recommendation conditions, it means that the target user is likely to like the songs to be recommended. Then, the songs to be recommended can be recommended to the target user, realizing personalized recommendation of new songs without historical user behavior.

[0170] Based on the same technical concept, the embodiment of the present application also provides a device for song recommendation, which can be applied to a computing device, see Figure 7 The apparatus may include an acquisition module 610, a reasoning module 620, and a recommendation module 630, wherein:

[0171] The acquisition module 610 is used to acquire the audio and tag information of the song to be recommended;

[0172] The reasoning module 620 is used to determine the audio fusion feature and the label fusion feature of the song to be recommended based on the audio and the label information; and determine the preference feature of the song to be recommended based on the audio fusion feature and the label fusion feature;

[0173] The recommendation module 630 is used to recommend the song to be recommended to the target user account when the similarity between the preference characteristics of the favorite songs of the target user account and the preference characteristics of the song to be recommended meets the recommendation condition.

[0174] In a possible implementation, the cross-modal representation model includes an audio feature representation branch and a label feature representation branch, and the reasoning module 620 is used to:

[0175] Determine a cross-modal representation model, wherein the cross-modal representation model includes an audio feature representation branch and a label feature representation branch;

[0176] Input the audio into the audio feature representation branch to obtain the audio fusion feature of the song to be recommended;

[0177] The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.

[0178] In a possible implementation, the apparatus further includes a training module, configured to:

[0179] Acquire multiple groups of first training samples, wherein each group of first training samples includes audio of a first sample song, label information of the first sample song, and label information of at least one other song;

[0180] For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least one pair of negative examples;

[0181] Based on the positive example and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through comparative learning to obtain the audio feature representation branch and the label feature representation branch.

[0182] In a possible implementation, the reasoning module is used to:

[0183] The audio fusion feature and the label fusion feature are input into a preference feature extraction model to obtain the preference feature of the song to be recommended.

[0184] In a possible implementation, the apparatus further includes a training module, configured to:

[0185] Acquire multiple groups of second training samples, wherein each group of second training samples includes audio and label information of multiple second sample songs, and the multiple second sample songs are intersections of favorite songs included in favorite song lists of multiple users;

[0186] For each second sample song in each group of second training samples, input the audio of the second sample song and the label information of the second sample song into the cross-modal representation model to obtain the audio fusion feature of the second sample song and the label fusion feature of the second sample song;

[0187] Inputting the audio fusion feature of the second sample song and the label fusion feature of the second sample song into the preference feature extraction model to obtain the predicted preference feature of the second sample song;

[0188] For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine a central predicted preference feature corresponding to the second training samples, and calculate a similarity between the central predicted preference feature and a predicted preference feature of at least one second sample song included in the second training samples;

[0189] Based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.

[0190] In a possible implementation, the training module is used to:

[0191] Selecting a central song from the second sample songs included in the second training sample;

[0192] The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.

[0193] In a possible implementation, the training module is used to:

[0194] The second sample song with the highest popularity in the second training sample is used as the central song of the second training sample, wherein the popularity is used to characterize at least one of the number of plays, the number of downloads, and the number of collections of the song.

[0195] In a possible implementation, the training module is used to:

[0196] Calculating an average feature of the predicted preference features of the second sample songs included in the second training sample;

[0197] The average feature is used as the central prediction preference feature corresponding to the second training sample.

[0198] In a possible implementation, the training module is used to:

[0199] For each group of second training samples, accumulating the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample to obtain an accumulated result;

[0200] Multiplying the accumulated result by the weight corresponding to the second training sample to obtain a product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample;

[0201] Based on the multiplication result, calculating the loss value;

[0202] Based on the loss value, an initial preference feature extraction model is trained.

[0203] In a possible implementation, the tag information of the song to be recommended includes at least one of the singer's name, lyrics, release time, award information, creation background, and style of the song to be recommended.

[0204] In the technical solution provided by the embodiment of the present application, when recommending songs to be recommended, no matter whether the songs to be recommended are new songs or old songs, there is no need to obtain the historical user behavior of the songs to be recommended. It is only necessary to obtain the audio and tag information of the songs to be recommended, based on which the preference characteristics of the songs to be recommended are obtained. Then, the preference characteristics of the songs to be recommended are compared with the preference characteristics of the songs collected by each user. If the similarity between the preference characteristics of the songs collected by the target user and the songs to be recommended meets the recommendation conditions, it means that the target user is likely to like the songs to be recommended. Then, the songs to be recommended can be recommended to the target user, realizing personalized recommendation of new songs without historical user behavior.

[0205] It should be noted that: the song recommendation device provided in the above embodiment only uses the division of the above functional modules as an example when recommending songs. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. In addition, the song recommendation device provided in the above embodiment and the song recommendation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0206] Figure 8 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. The computing device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the computing device may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the computing device may also include other components for implementing device functions, which will not be described in detail here.

[0207] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the song recommendation method in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0208] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the user's favorite playlists, favorite songs, audio, tag information, etc. involved in this application are all obtained with full authorization.

[0209] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0210] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A song recommendation method, characterized in that: The method comprises: Get the audio and tag information of the song to be recommended; Based on the audio and the tag information, determine the audio fusion feature and the tag fusion feature of the song to be recommended; Determining the preference features of the song to be recommended based on the audio fusion features and the tag fusion features; When the similarity between the preference characteristics of the favorite songs of the target user account and the preference characteristics of the song to be recommended meets the recommendation condition, the song to be recommended is recommended to the target user account.

2. The method according to claim 1, characterized in that The step of determining the audio fusion feature and the tag fusion feature of the song to be recommended based on the audio and the tag information includes: Determine a cross-modal representation model, wherein the cross-modal representation model includes an audio feature representation branch and a label feature representation branch; Input the audio into the audio feature representation branch to obtain the audio fusion feature of the song to be recommended; The tag information is input into the tag feature representation branch to obtain the tag fusion feature of the song to be recommended.

3. The method according to claim 2, characterized in that Before inputting the audio into the audio feature characterization branch and inputting the tag information into the tag feature characterization branch, the method further includes: Acquire multiple groups of first training samples, wherein each group of first training samples includes audio of a first sample song, label information of the first sample song, and label information of at least one other song; For each group of first training samples, the audio of the first sample song included in the first training sample and the label information of the first sample song are taken as a pair of positive examples, and the audio of the first sample song included in the first training sample and the label information of at least one other song are taken as at least one pair of negative examples; Based on the positive example and the at least one pair of negative examples, the initial audio feature representation branch and the initial label feature representation branch are trained through comparative learning to obtain the audio feature representation branch and the label feature representation branch.

4. The method according to claim 1, characterized in that: The step of determining the preference feature of the song to be recommended based on the audio fusion feature and the tag fusion feature includes: The audio fusion feature and the label fusion feature are input into a preference feature extraction model to obtain the preference feature of the song to be recommended.

5. The method according to claim 4, characterized in that Before inputting the audio fusion feature and the label fusion feature into the preference feature extraction model, the method further includes: Acquire multiple groups of second training samples, wherein each group of second training samples includes audio and label information of multiple second sample songs, and the multiple second sample songs are intersections of favorite songs included in favorite song lists of multiple users; For each second sample song in each group of second training samples, input the audio of the second sample song and the label information of the second sample song into the cross-modal representation model to obtain the audio fusion feature of the second sample song and the label fusion feature of the second sample song; Inputting the audio fusion feature of the second sample song and the label fusion feature of the second sample song into the preference feature extraction model to obtain the predicted preference feature of the second sample song; For each group of second training samples, based on the predicted preference features of the second sample songs included in the second training samples, determine a central predicted preference feature corresponding to the second training samples, and calculate a similarity between the central predicted preference feature and a predicted preference feature of at least one second sample song included in the second training samples; Based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample, the initial preference feature extraction model is trained to obtain the preference feature extraction model.

6. The method according to claim 5, characterized in that The step of determining the central predicted preference feature corresponding to the second training sample based on the predicted preference feature of the second sample song included in the second training sample includes: Selecting a central song from the second sample songs included in the second training sample; The predicted preference features of the central song are used as the central predicted preference features corresponding to the second training sample.

7. The method according to claim 6, characterized in that The selecting a central song in the second training sample comprises: The second sample song with the highest popularity in the second training sample is used as the central song of the second training sample, wherein the popularity is used to characterize at least one of the number of plays, the number of downloads, and the number of collections of the song.

8. The method according to claim 5, characterized in that The step of determining the central predicted preference feature corresponding to the second training sample based on the predicted preference feature of the second sample song included in the second training sample includes: Calculating an average feature of the predicted preference features of the second sample songs included in the second training sample; The average feature is used as the central prediction preference feature corresponding to the second training sample.

9. The method according to any one of claims 5 to 8, characterized in that: The method of training the initial preference feature extraction model based on the similarity between the central predicted preference feature and the predicted preference feature of at least one second sample song included in the second training sample to obtain the preference feature extraction model comprises: For each group of second training samples, accumulating the similarity between the central predicted preference feature corresponding to the second training sample and the predicted preference feature of at least one second sample song included in the second training sample to obtain an accumulated result; Multiplying the accumulated result by the weight corresponding to the second training sample to obtain a product result, wherein the number of second sample songs included in the second training sample is positively correlated with the weight corresponding to the second training sample; Based on the multiplication result, calculating the loss value; Based on the loss value, the initial preference feature extraction model is trained to obtain the preference feature extraction model.

10. The method according to any one of claims 1 to 9, characterized in that The tag information of the song to be recommended includes at least one of the singer name, lyrics, release time, award information, creation background, and style of the song to be recommended.

11. A computing device, characterized in that: The computing device includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the song recommendation method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the song recommendation method as described in any one of claims 1 to 10.

13. A computer program product, characterized in that The computer program product stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the song recommendation method as described in any one of claims 1 to 10.

Citation Information

Patent Citations

  • Song recommendation method and mobile equipment

    CN107798107A

  • Music recommendation method based on time residence and state residence mixed model

    CN113220929A

  • Recommendation model training method, song recommendation method and related device

    CN115934996A

  • Supervised metric learning for music structure features

    US20230121764A1