Music recommendation method based on knowledge graph and reinforcement learning

Abstract

The invention relates to the technical field of recommendation systems, in particular to a music recommendation method based on a knowledge graph and reinforcement learning. The method comprises the following steps: S1, extracting information of users and songs and interaction records of the users and the songs from a music data set, and performing knowledge extraction; s2, constructing nodes and edges of a knowledge graph according to entities, relationships and attributes of knowledge extraction, storing and managing the knowledge graph by using a graphic database, and performing knowledge representation; s3, designing a reasonable prompt template and generating prompt input according to behavior characteristics and requirements of the user, and constructing a user music preference model after model training; s4, constructing a user preference music knowledge graph according to the user music preference model and actual historical interaction behaviors of the user and the music; and S5, proposing a recommendation strategy based on reinforcement learning, and updating and optimizing the recommendation strategy according to user feedback.

Description

Technical Field

[0001] The present invention belongs to the technical field of recommendation methods, and specifically relates to a music recommendation method based on knowledge graph and reinforcement learning. Background Art

[0002] With the rapid development of Internet technology, mobile technology and Internet of Things technology, the problem of information overload has emerged accordingly. With the rise of new media, music creation has reached an unprecedented peak, and the number of songs has exploded. This makes it extremely challenging for listeners to find songs that meet their personal music aesthetics in the ocean of music. From various music genres to regional music styles, diversified music content provides listeners with a wider range of choices, but it also makes them face the problem of information overload to some extent. General music recommendation systems use collaborative filtering (CF) to learn more accurate user interest representations based on users' historical music interaction records, so as to recommend music of interest to users. However, in real life, the recommendation accuracy of collaborative filtering is not high and lacks a certain degree of interpretability. Applying knowledge graphs to recommendation systems can alleviate this problem to a certain extent by navigating paths in the graph. In the construction of the relationship model between users and music, prompt learning is further integrated to accurately capture users' user music preferences, making the recommendation effect more significant.

[0003] Reinforcement learning simulates the reward and punishment mechanism in the learning process and analyzes how to act to maximize the cumulative reward. It allows the agent to learn the optimal strategy in the interaction with the environment, so that it can make decisions with more long-term benefits and better solve the decision-making problem in the recommendation process. Reinforcement learning algorithms are used for training to predict the music content that users may be interested in. This helps to solve the shortcomings of traditional recommendation methods in terms of music subjectivity and diversity, thereby providing music recommendations that are more in line with user preferences. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and propose a music recommendation method based on knowledge graph and reinforcement learning. On the basis of constructing user and music knowledge graph, the model incorporates prompt learning and introduces reinforcement learning strategy, so as to make the recommendation model more targeted and accurate.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0006] A music recommendation method based on knowledge graph and reinforcement learning includes the following steps:

[0007] S1: Extract information about users, songs, and their interaction records from the music dataset and perform knowledge extraction;

[0008] S2: Based on the entities, relationships and attributes extracted above, the nodes and edges of the knowledge graph are constructed, and the knowledge graph is stored and managed using a graph database and knowledge representation is performed;

[0009] S3: Based on the user's behavioral characteristics and needs, a reasonable prompt template is designed and prompt input is generated. After model training, a user music preference model is constructed;

[0010] S4: Construct a user preference music knowledge graph based on the user music preference model and the user's actual historical interaction behavior with music;

[0011] S5: Design a recommendation strategy based on reinforcement learning, and update and optimize the recommendation strategy based on user feedback.

[0012] Further, as a preferred technical solution of the present invention, step S1 includes the following steps: extracting information about users and songs and their interaction records from the music data set. Clean and preprocess the data to ensure the accuracy and consistency of the data: remove duplicate data, data with missing key fields, and data with obvious errors; deduplicate and normalize user information to ensure that each user has only one unique identifier; standardize song information, such as unifying the format of song names and singer names. Then, knowledge extraction is performed on the processed data set, including entity extraction, relationship extraction, and attribute extraction. Among them, entity recognition and extraction need to identify entities such as users and songs, and determine their boundaries and types; relationship extraction needs to extract semantic relationships between two or more entities from text or data, such as user-favorite songs, songs-genres, etc.; attribute extraction extracts its attributes and attribute values ​​from unstructured text through a given entity (such as a user or song) to form structured data.

[0013] Further, as a preferred technical solution of the present invention, step S2 includes the following steps: mapping the extracted entities (user u and song v) to nodes in the knowledge graph. Each node represents a unique entity and contains the unique identifier and attribute p of the entity. The entity sets of users and songs are represented by U and V respectively, and the attribute set of songs is represented by P. The extracted entity, relationship and attribute data are imported into a graph database, and the nodes and edges of the knowledge graph are constructed based on the extracted entities, relationships and attributes. Then, representation learning is performed on the constructed knowledge graph so as to map the entities and relationships in the knowledge graph into vector representations to capture the semantics and associations between entities and relationships. The present invention selects a TransE model based on a vector space and uses a stochastic gradient descent (SGD) optimization algorithm to gradually reduce the loss function by continuously iteratively updating parameters. Its characteristic is that the gradients of some training samples are used to update the model parameters, which greatly reduces the computational overhead and accelerates the convergence speed of the model. The update rules of the SGD algorithm are as follows:

[0014]

[0015] Among them, θ t is the parameter at the tth iteration, η is the learning rate, J(θ t ;x i ,y i ) is the loss function, For the parameter θ t In each iteration, a training sample is selected and the gradient of the loss function with respect to the parameters is calculated, and then the parameters are updated. This process is iterated continuously until the loss function converges or the preset number of iterations is reached.

[0016] Further, as a preferred technical solution of the present invention, step S3 includes the following steps: design a reasonable prompt template according to the user's behavioral characteristics and needs. For example, design a prompt template about the user's music preference, such as "User X likes to listen to music of the type <mask>". The <mask> part in the template will be used for subsequent filling and prediction. Then, the user's historical behavior data is converted into an input format suitable for prompt learning. For example, the user's playback records and evaluations and ratings can be converted into a series of sentences describing the user's music preferences. These sentences are combined with the prompt template to generate a complete prompt input. The generated prompt input is sent to the pre-trained deep learning model. The present invention selects the Fine-tuning method based on ERNIE3.0 to perform model training, uses the model to predict the <mask> part, and generates music types or specific songs that the user may be interested in. Based on the user's historical behavior data and the user's interest preferences predicted by prompt learning, a collaborative filtering algorithm is used to construct a user model, which contains personalized features such as the user's music preferences and emotional tendencies.

[0017] Further, as a preferred technical solution of the present invention, step S4 includes the following steps: using the actual historical interaction behavior between the user and the music, plus the user's music preferences learned by prompting, connecting them by defining entities and establishing relationships and giving corresponding emotional preference labels, to construct a user music preference knowledge graph It consists of entity set ε and relationship set Composition, which is defined as The triple (e h ,r,e t ) represents the head entity e h With the tail entity e t There is a preference relationship r between them.

[0018] Further, as a preferred technical solution of the present invention, step S5 includes the following steps: by designing a reinforcement learning strategy, from each user entity e u Start with all possible paths to recommended music and find e u A collection of music entities that may be liked. Since the experiment requires the agent to navigate in the knowledge graph, an MDP environment is constructed, and its reinforcement learning related settings are as follows:

[0019] State: The state s at step t is defined as a tuple (u, e t ,h t ), u represents the entity embedding vector of the initial state user, e t represents the entity embedding vector that the agent reaches at step t, h t represents the set of all entities and relations visited by the agent before t steps.

[0020] The experiment defines n historical interaction records, which are the collection of all entities and relationships traversed in the past n steps.

[0021] That is {e t-n , r t-n+1 ,…,e t-1 r t}. Taking user u as an example, the initial state When reaching step T, the state is s T =(u,e r ,h T ).

[0022] Action space: In state s t The action space A t That is, entity e t , except for all connected entities and relationships that have been traversed

[0023] Bonus: As a music recommendation system approach, an excellent recommendation path is to connect users with potential music that they may be interested in. To achieve this goal, the final state s T =(u,e T ,h T ) can be expressed as:

[0024]

[0025] The agent learns the relevant action strategy based on the above MDP environment, denoted as π(·|s t , A t ), which outputs in the current state s t And the set A of all possible action spaces t The ultimate goal is to maximize the expected reward for a given user u, as shown below:

[0026]

[0027] Where γ represents the loss factor and θ represents the model parameter set of the policy network π. The gradient calculation of the policy network is as follows:

[0028]

[0029] Among them, G represents the discounted cumulative reward at step t.

[0030] After training the policy network, the agent can perform a probabilistic search starting from the specified user to eventually find the music recommendation candidate set with the highest cumulative reward and recommend it to the user.

[0031] The music recommendation method based on knowledge graph and reinforcement learning described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0032] (1) The present invention constructs a knowledge graph of user music preferences to make the final recommendation results more explainable.

[0033] (2) The present invention utilizes a reinforcement learning strategy network to make recommendations more logical and accurate.

[0034] (3) The present invention uses prompt learning to better capture the user's music interests and preferences, thereby providing more personalized recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present invention;

[0036] Figure 2 Pre-training language model framework for ERNIE3.0;

[0037] Figure 3 This is the principle of the reinforcement learning strategy in the present invention. DETAILED DESCRIPTION

[0038] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0039] like Figure 1 As shown, the present invention consists of a user music preference model, a user music preference knowledge graph and reinforcement learning, wherein the user music preference model is obtained through prompt learning after model training, and the user music preference knowledge graph consists of the user music preference model and actual interaction records.

[0040] A music recommendation method based on knowledge graph and reinforcement learning includes the following steps:

[0041] S1: Extract information about users, songs, and their interaction records from the music dataset and perform knowledge extraction;

[0042] S2: Based on the entities, relationships and attributes extracted above, the nodes and edges of the knowledge graph are constructed, and the knowledge graph is stored and managed using a graph database and knowledge representation is performed;

[0043] S3: Based on the user's behavioral characteristics and needs, a reasonable prompt template is designed and prompt input is generated. After model training, a user music preference model is constructed;

[0044] S4: Construct a user preference music knowledge graph based on the user music preference model and the user's actual historical interaction behavior with music;

[0045] S5: Design a recommendation strategy based on reinforcement learning, and update and optimize the recommendation strategy based on user feedback.

[0046] S1 is specifically: extracting information about users, songs, and their interaction records from the music data set. Clean and preprocess the data to ensure data accuracy and consistency: remove duplicate data, data with missing key fields, and data with obvious errors; deduplicate and normalize user information to ensure that each user has only one unique identifier; standardize song information, such as unifying the format of song names and singer names. Then perform knowledge extraction on the processed data set, including entity extraction, relationship extraction, and attribute extraction. Among them, entity recognition and extraction need to identify entities such as users and songs, and determine their boundaries and types; relationship extraction needs to extract semantic relationships between two or more entities from text or data, such as songs that users like, songs to which genres they belong, etc.; attribute extraction extracts attributes and their attribute values ​​from unstructured text through given entities (such as users or songs) to form structured data.

[0047] In S2, the entities extracted in S1 (user u and song v) are mapped to nodes in the knowledge graph. Each node represents a unique entity and contains the unique identifier and attribute p of the entity. The entity sets of users and songs are represented by U and V respectively, and the attribute set of songs is represented by users. The extracted entity, relationship and attribute data are imported into the graph database, and the nodes and edges of the knowledge graph are constructed based on the extracted entities, relationships and attributes. Then, representation learning is performed on the constructed knowledge graph so that the entities and relationships in the knowledge graph are mapped into vector representations to capture the semantics and associations between entities and relationships. The present invention selects the TransE model based on vector space and uses the Stochastic Gradient Descent (SGD) optimization algorithm to gradually reduce the loss function by continuously iteratively updating the parameters. Its characteristic is that the gradient of some training samples is used to update the model parameters, which greatly reduces the computational overhead and accelerates the convergence speed of the model. The update rules of the SGD algorithm are as follows:

[0048]

[0049] Among them, θ t is the parameter at the tth iteration, η is the learning rate, J(θ t ;x i ,y i ) is the loss function, For the parameter θ tIn each iteration, a training sample is selected and the gradient of the loss function with respect to the parameters is calculated, and then the parameters are updated. This process is iterated continuously until the loss function converges or the preset number of iterations is reached.

[0050] S3 specifically designs a reasonable prompt template according to the user's behavioral characteristics and needs. For example, design a prompt template about the user's music preference, such as "User X likes to listen to music of type <mask>". The <mask> part in the template will be used for subsequent filling and prediction. Next, the user's historical behavior data is converted into an input format suitable for prompt learning. For example, the user's playback records, comments and ratings can be converted into a series of sentences describing the user's music preferences. These sentences are combined with the prompt template to generate a complete prompt input. The generated prompt input is sent to the pre-trained deep learning model. The present invention selects the Fine-tuning method based on ERNIE3.0 to perform model training, and its pre-trained model framework is as follows: Figure 2 As shown, the model is used to predict the <mask> part and generate the music type or specific song that the user may be interested in. Based on the user's historical behavior data and the user's interest preferences predicted by prompt learning, a collaborative filtering algorithm is used to build a user model, which contains personalized features such as the user's music preferences and emotional tendencies.

[0051] In S4, the actual historical interaction between users and music, combined with the user music preferences learned from the prompts in S3, are used to build a user music preference knowledge graph by defining entities and establishing relationships to connect them and give corresponding emotional preference labels. It consists of entity set ε and relationship set Composition, which is defined as The triple (e h ,r,e t ) represents the head entity e h With the tail entity e t There is a preference relationship r between them.

[0052] S5 specifically designs a reinforcement learning strategy that simulates the reward and punishment mechanism in the learning process and analyzes how to act to maximize the cumulative reward. The principle is as follows Figure 3 As shown. From each user entity e u Start with all possible paths to recommended music and find e u A collection of music entities that may be liked. Since the experiment requires the agent to navigate in the knowledge graph, an MDP environment is constructed, and its reinforcement learning related settings are as follows:

[0053] State: The state s at step t is defined as a tuple (u, e t ,h t), u represents the entity embedding vector of the initial state user, e t represents the entity embedding vector that the agent reaches at step t, h t represents the set of all entities and relations visited by the agent before step t. The experiment defines n historical interaction records as the set of all entities and relations traversed in the past n steps, that is, {e t-n , r t-n+1 ,…,e t-1 , r t}. Taking user u as an example, the initial state When reaching step T, the state is s T =(u,e r ,h T ).

[0054] Action space: In state s t The action space A t That is, entity e t , except for all connected entities and relationships that have been traversed

[0055] Bonus: As a music recommendation system approach, an excellent recommendation path is to connect users with potential music that they may be interested in. To achieve this goal, the final state s T =(u,e r ,h T ) can be expressed as:

[0056]

[0057] The agent learns the relevant action strategy based on the above MDP environment, denoted as π(·|s t , A t ), which outputs in the current state s t And the set A of all possible action spaces t The ultimate goal is to maximize the expected reward for a given user u, as shown below:

[0058]

[0059] Where γ represents the loss factor and θ represents the model parameter set of the policy network π. The gradient calculation of the policy network is as follows:

[0060]

[0061] Among them, G t , represents the discounted cumulative reward at step t.

[0062] After training the policy network, the agent can perform a probabilistic search starting from the specified user to eventually find the music recommendation candidate set with the highest cumulative reward and recommend it to the user.

[0063] This paper proposes a music recommendation system method based on knowledge graph and reinforcement learning. By constructing a knowledge graph of user music preferences, the final recommendation results are made more explainable. At the same time, the strategy network of reinforcement learning is used to make the recommendation more logical and accurate, and prompt learning is introduced to better capture the user's music interests and preferences, thereby providing more personalized recommendations.

[0064] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A music recommendation method based on knowledge graph and reinforcement learning, characterized in that: The following steps are involved: S1, extract user, song information and their interaction records from the music dataset and perform knowledge extraction; S2. Construct nodes and edges of knowledge graphs based on entities, relationships and attributes extracted from knowledge, and use graph databases to store and manage knowledge graphs and perform knowledge representation; S3. Design a reasonable prompt template and generate prompt input according to the user's behavioral characteristics and needs. After model training, build a user music preference model; S4. Construct a user preference music knowledge graph based on the user music preference model and the user's actual historical interaction behavior with music; S5. Propose a recommendation strategy based on reinforcement learning, and update and optimize the recommendation strategy based on user feedback.

2. The music recommendation method based on knowledge graph and reinforcement learning according to claim 1, characterized in that: The step S1 includes the following steps: extracting information about users and songs and their interaction records from a music data set; cleaning and preprocessing the data to ensure the accuracy and consistency of the data: removing duplicate data, data with missing key fields, and data with obvious errors; deduplicating and normalizing user information to ensure that each user has only one unique identifier; standardizing song information; and then performing knowledge extraction on the processed data set, including entity extraction, relationship extraction, and attribute extraction; wherein entity recognition and extraction require identifying user and song entities and determining their boundaries and types; relationship extraction requires extracting semantic relationships between two or more entities from text or data; attribute extraction extracts attributes and their attribute values ​​from unstructured text through a given entity to form structured data.

3. The music recommendation method based on knowledge graph and reinforcement learning according to claim 2 is characterized in that: The step S2 includes the following steps: mapping the extracted entities including user u and song v to nodes in the knowledge graph; each node represents a unique entity and contains the unique identifier and attribute p of the entity, the entity sets of users and songs are represented by U and V respectively, and the attribute set of songs is represented by P; importing the extracted entity, relationship and attribute data into a graph database, and constructing nodes and edges of the knowledge graph according to the extracted entities, relationships and attributes; then, performing representation learning on the constructed knowledge graph so as to map the entities and relationships in the knowledge graph into vector representations to capture the semantics and relevance between entities and relationships; selecting a TransE model based on a vector space, and using a stochastic gradient descent SGD optimization algorithm, and gradually reducing the loss function by continuously iteratively updating parameters, and the update rules of the SGD optimization algorithm are as follows: Among them, θ t is the parameter at the tth iteration, η is the learning rate, J(θ t ;x i ,y i ) is the loss function, For the parameter θ t In each iteration, a training sample is selected and the gradient of the loss function with respect to the parameters is calculated, and then the parameters are updated. This process is iterated continuously until the loss function converges or the preset number of iterations is reached.

4. The music recommendation method based on knowledge graph and reinforcement learning according to claim 3 is characterized in that: The step S3 includes the following steps: designing a reasonable prompt template according to the user's behavior characteristics and needs; designing a prompt template about the user's music preference: user X likes to listen to <mask>Type of music; template <mask>The part will be used for subsequent filling and prediction; then, the user's historical behavior data is converted into an input format suitable for prompt learning, and the user's play records, comments and ratings are converted into a series of sentences describing the user's music preferences; the sentence is combined with the prompt template to generate a complete prompt input; the generated prompt input is sent to the pre-trained deep learning model; the Fine-tuning method based on ERNIE3.0 is selected for model training, and the model is used to <mask> The system predicts the types of music or specific songs that users may be interested in. It also uses collaborative filtering algorithms to build a user model based on the user's historical behavior data and prompt learning to predict user interest preferences, which includes personalized features such as the user's music preferences and emotional tendencies.< / mask> < / mask> < / mask> 5. The music recommendation method based on knowledge graph and reinforcement learning according to claim 4 is characterized in that: The step S4 comprises the following steps: using the actual historical interaction behavior between the user and the music, plus the user's music preferences learned by prompting, connecting them by defining entities and establishing relationships and giving corresponding emotional preference labels, to construct a user music preference knowledge graph It consists of entity set ε and relationship set Composition, which is defined as The triple (e h ,r,e t ) represents the head entity e h With the tail entity e t There is a preference relationship r between them.

6. The music recommendation method based on knowledge graph and reinforcement learning according to claim 5, characterized in that: The step S5 comprises the following steps: By designing a reinforcement learning strategy, from each user entity e u Start with all possible paths to recommended music and find e u A collection of music entities that may be liked; since the experiment requires the agent to navigate in the knowledge graph, an MDP environment is constructed, and its reinforcement learning related settings are as follows: State: The state s at step t is defined as a tuple (u, e t ,h t ), u represents the entity embedding vector of the initial state user, e t represents the entity embedding vector that the agent reaches at step t, h t represents the set of all entities and relations visited by the agent before t steps; The experiment defines n historical interaction records, which are the collection of all entities and relationships traversed in the past n steps. That is {e t-n ,r t-n+1 ,...,e t-1 ,r t }; User u, initial state When reaching step T, the state is s T =(u,e T ,h T ); Action space: In state s t The action space A t That is, entity e t , except for all connected entities and relationships that have been traversed Reward: An excellent recommendation path connects users with potential music that they may be interested in; the final state is T =(u,e T ,h T ) is expressed as: The agent learns the relevant action strategy based on the above MDP environment, denoted as π(·|s t , A t ), which outputs in the current state s t And the set A of all possible action spaces t The probability distribution when maximizing the expected reward for a given user u is as follows: Where γ represents the loss factor, θ represents the model parameter set of the policy network π; the gradient calculation of the policy network is as follows: Among them, G t , represents the cumulative reward at step t; After training the policy network, the agent can perform a probabilistic search starting from the specified user to eventually find the music recommendation candidate set with the highest cumulative reward and recommend it to the user.

Images