User multi-behavior-oriented learning path recommendation method and system based on reinforcement learning

Abstract

The invention discloses a user multi-behavior-oriented learning path recommendation method and system based on reinforcement learning, and the method comprises the steps: taking different types of interactive behaviors as entity nodes of a knowledge graph, carrying out the connection with user nodes and learning resource nodes, obtaining a curriculum knowledge graph after fusion and enhancement, and carrying out the learning path recommendation of the curriculum knowledge graph; the curriculum knowledge graph embedding representation after fusion enhancement is obtained; based on the curriculum knowledge graph after fusion enhancement, performing data enhancement from a behavior perspective and a learning resource perspective, and obtaining curriculum knowledge graph embedded representation after data enhancement from the behavior perspective and the learning resource perspective; on the basis of the obtained embedded representation, feature extraction is carried out by adopting a graph convolutional neural network with an attention mechanism; and regarding the output of the graph convolutional neural network as the state of the user, regarding the deep Q network as an agent, jointly performing reward feedback on the agent at the knowledge point level and the path level, and performing personalized learning path recommendation on the user. The method has a better recommendation result.

Description

Technical Field

[0001] This invention relates to the field of intelligent recommendation technology, and in particular to a learning path recommendation method and system based on reinforcement learning for multiple user behaviors. Background Technology

[0002] Recommender systems are key to solving online information overload and improving the relationship between users and resources. By modeling and analyzing historical user data, they accurately capture user needs and recommend the resources they require. This personalized service has made recommender systems an indispensable part of modern information technology, applicable to all industries, including online education.

[0003] With the increasing popularity of online learning, artificial intelligence is driving the development of online learning platforms, and numerous scholars are researching learning path recommendation. Learning path recommendation comprehensively considers the differences in learning ability, knowledge background, and learning goals of target learners, tailoring a learning path that conforms to educational principles and achieves learning objectives. Generally, learning path recommendation is divided into two categories: globally optimal path recommendation and locally iterative path recommendation. The former provides a complete learning path based on the learning objectives all at once, while the latter iterates and generates a learning path gradually based on the learning progress. Currently, commonly used methods in this field include those based on traditional machine learning, graph theory, deep learning, and reinforcement learning. These technologies are driving the development of learning path recommendation, automatically and efficiently identifying online learner characteristics, efficiently organizing and allocating online learning resources, and planning personalized learning paths for each learner, helping them achieve their learning goals with minimal learning costs and maximum efficiency.

[0004] Artificial intelligence is driving the development of learning path recommendation in online learning models. However, the sparsity of interaction data between students and learning resources, as well as the high heterogeneity of student groups, often lead to systems planning inappropriate learning paths. To address this issue, existing research focuses on meeting the frequently changing learning states of students during the learning process. While this has achieved some success, it still faces two challenges:

[0005] 1. The problem of sparse data in learner-learning resource interaction. Although the internet offers a wealth of online learning resources, the online learning format is unique. Most learning platforms lack the ability to dynamically generate new practice questions after students complete exercises, and most learners do not repeat the material in the short term after completing a course. Furthermore, to protect learner privacy, online platforms do not publicly disclose learning data, which is another reason for the data sparsity problem. Therefore, in online learning models, most learners can only generate a single and limited learning path.

[0006] 2. The problem of insufficient personalization in recommendation results. Current learning path recommendation systems still have significant limitations. For example, they lack fine-grained modeling of learners' multidimensional characteristics (such as learning style, learning preferences, learning goals, learning abilities, and learning scenarios), resulting in poor personalized recommendation performance. Furthermore, most existing research has failed to fully explore the rich interactive behaviors between learners and learning resources (such as answering exercises, watching videos, and participating in community discussions). This interactive data contains crucial information about learning patterns, but it has not been effectively utilized. This also leads to poor accuracy and personalization of recommendation results. Summary of the Invention

[0007] This invention addresses the problems of sparse learner interaction data and lack of personalization in learning path recommendations in current online learning platforms. It proposes a learning path recommendation method and system based on reinforcement learning, focusing on multiple user behaviors. First, we categorize various user interactions with learning resources into four types, treating them as entity nodes and fusing them with the learning resource knowledge graph for enhancement. We then use a graph convolutional neural network with an attention mechanism to accurately capture their multi-source heterogeneous features. Next, we propose three data augmentation strategies from both behavioral and learning resource perspectives, comparing and learning from the additional data generated and incorporating it into reinforcement learning as a supplement to state information. Finally, we fully consider the personalized characteristics of users, setting up a dual-level reward function mechanism (path and knowledge point) in reinforcement learning to assist the model in selection, planning the most suitable learning resources for users based on their different personalized characteristics.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] This invention proposes a learning path recommendation method based on reinforcement learning for multiple user behaviors, comprising:

[0010] Step 1: Classify the user's interaction with learning resources; treat different types of behavior as entity nodes of the knowledge graph, connect them with user nodes and learning resource nodes to obtain the fused and enhanced course knowledge graph, and use TransR technology to obtain the embedded representation of the fused and enhanced course knowledge graph;

[0011] Step 2: Based on the fused and enhanced course knowledge graph, perform data augmentation from the perspectives of behavior and learning resources, and use TransR technology to obtain the embedded representation of the course knowledge graph after data augmentation from the perspectives of behavior and learning resources;

[0012] Step 3: Based on the embedding representations obtained in Step 1 and Step 2, feature extraction is performed using a graph convolutional neural network with an attention mechanism.

[0013] Step 4: Treat the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. Provide reward feedback to the agent at both the knowledge point level and the path level to recommend personalized learning paths for the user.

[0014] Furthermore, step 1 includes: classifying user interaction behaviors with learning resources into four categories: entry, learning, marking, and submission.

[0015] Furthermore, in step 2, data augmentation is performed from a behavioral perspective in the following manner:

[0016] With probability as Modify the interaction between users and learning resources.

[0017] Furthermore, in step 2, data augmentation is performed from the perspective of learning resources in the following manner:

[0018] The interaction sequence of each user with the learning resources is arranged in a certain proportion. Random blocking is implemented;

[0019] Partial exchange of learning resources belonging to the same chapter.

[0020] Furthermore, prior to step 4, the method includes: comparative learning of the features corresponding to the embedded representation of the course knowledge graph based on behavioral and learning resource data enhancement.

[0021] Furthermore, the loss function for contrastive learning is:

[0022] L CL =L I +L B +L C ,

[0023] In the formula

[0024]

[0025] Where L CL L represents the loss function for contrastive learning. I L B L C These represent the contrast loss at the learning resource level, the contrast loss at the behavior node level, and the contrast loss at the knowledge point level, respectively; sim() represents the similarity function. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the first augmented view, respectively. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the second enhanced view, respectively; Z I,- , ZB,- , Z C,- These are negative samples from the same batch of data.

[0026] Furthermore, the user's status also includes: the user's learning goals, learning progress, and learning preferences.

[0027] Furthermore, in step 4, the agent is rewarded at both the knowledge point level and the path level in the following manner:

[0028] R(s t ,a t )=(1-α)·R points (s t ,a t )+α·R path (s t ,a t ),

[0029] In the formula,

[0030]

[0031] Where R represents the reward function, R points Represents the knowledge-point level reward function, R path This represents the path-level reward function, where α is a weight parameter, and s t In the state at time t, a t In state s t The actions to be taken are β1 and β2, which are weighting coefficients, T represents the number of knowledge points, and φ (point) t ,point t+1 (a knowledge point) t with point t+1 The function of the dependency relationship between them and φ(·)∈{0,1}, emb rec and emb hi This represents the embedding of predictions with actual knowledge points, where k is the penalty weight, and Edit(P) rec ,P hi ) represents the number of edits required to convert the predicted learning path into the actual learning path, where max(|P) represents the number of edits required. rec |,|P his |) represents the maximum length of the two learning paths after normalization, R path (s t ,a t The range of ) is [-γ, 0], where γ is the discount factor.

[0032] Another aspect of this invention proposes a learning path recommendation system based on reinforcement learning for user multi-behavior scenarios, comprising:

[0033] The knowledge graph enhancement module is used to classify the interaction behaviors between users and learning resources; different types of behaviors are regarded as entity nodes of the knowledge graph, which are connected with user nodes and learning resource nodes to obtain the fused and enhanced course knowledge graph, and the TransR technology is used to obtain the embedded representation of the fused and enhanced course knowledge graph.

[0034] The data augmentation module is used to perform data augmentation from the perspectives of behavior and learning resources based on the fused and augmented course knowledge graph, and uses TransR technology to obtain the embedded representation of the course knowledge graph data augmented from the perspectives of behavior and learning resources.

[0035] The knowledge graph feature extraction module extracts features based on the embedded representations obtained from the knowledge graph augmentation module and the data augmentation module, using a graph convolutional neural network with an attention mechanism.

[0036] The reinforcement learning-based learning path recommendation module treats the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. It provides reward feedback to the agent at both the knowledge point level and the path level, and recommends personalized learning paths for the user.

[0037] Furthermore, it also includes:

[0038] The contrastive learning module is used to embed the course knowledge graph, which is augmented based on behavioral and learning resource data, into the corresponding features for contrastive learning.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. Expanding Data Volume. In traditional learning path recommendation research, most methods rely solely on learners' real historical interaction data for modeling and analysis. However, this single data source has significant limitations: on the one hand, learners typically do not repeat the course in the short term after completion; on the other hand, most online learning platforms have fixed and non-dynamically updated after-class exercises, resulting in sparse, partial, and singular historical interaction data. To overcome this data bottleneck, this invention innovatively proposes a data augmentation strategy employing behavioral data reconstruction and learning resource interaction sequence reorganization. By generating two diversified enhanced learning paths within the learner's capabilities, it effectively expands the data foundation for user path planning, thereby providing richer and more reliable data support for personalized learning recommendations.

[0041] 2. Planning more personalized learning paths. In real-world learning scenarios, learners exhibit significant individual differences, manifested in multiple dimensions such as learning goals, cognitive preferences, application scenarios, and ability levels. This diverse learning characteristic requires personalized recommendation systems to accurately identify and adapt to the unique needs of each learner. Addressing this key requirement, this invention innovatively employs a reinforcement learning framework to construct a personalized learning path recommendation system. This system, through deep modeling and multi-dimensional analysis of learners' historical interaction data, accurately identifies personalized information such as users' learning abilities, learning preferences, and learning goals. Based on these characteristics, it dynamically optimizes and adjusts learning paths in real time, thereby providing users with truly personalized optimal learning paths. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a reinforcement learning-based learning path recommendation method for user multi-behavior scenarios provided in an embodiment of the present invention.

[0043] Figure 2 A schematic diagram of the TransR process provided in an embodiment of the present invention;

[0044] Figure 3 This is a data enhancement diagram provided for an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of feature acquisition based on attention mechanism and graph convolutional network provided in an embodiment of the present invention;

[0046] Figure 5 This is one of the architectural diagrams of a reinforcement learning-based learning path recommendation system for user multi-behavior scenarios provided in an embodiment of the present invention;

[0047] Figure 6 This is the second schematic diagram of the architecture of a user-oriented multi-behavior learning path recommendation system based on reinforcement learning, provided as an embodiment of the present invention. Detailed Implementation

[0048] For ease of understanding, the following explanations are provided for some of the terms used in the specific embodiments of this invention:

[0049] Multi-behavior: refers to the various heterogeneous behavioral data generated by users during the interaction process, which can comprehensively reflect user preferences and interaction patterns.

[0050] Reinforcement learning: a machine learning paradigm based on Markov decision processes that optimizes decision-making strategies through continuous interaction between an agent and its environment, aiming to maximize the long-term cumulative reward function.

[0051] Learning path recommendation: Based on the user's learning goals, learning style, learning preferences and other characteristics, an optimal knowledge point path is planned for the user to reach the target knowledge node.

[0052] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0053] like Figure 1 As shown, a reinforcement learning-based learning path recommendation method for user multi-behavior applications includes:

[0054] Step 1: Classify the user's interaction with learning resources; treat different types of behavior as entity nodes of the knowledge graph, connect them with user nodes and learning resource nodes to obtain the fused and enhanced course knowledge graph, and use TransR technology to obtain the embedded representation of the fused and enhanced course knowledge graph;

[0055] Step 2: Based on the fused and enhanced course knowledge graph, perform data augmentation from the perspectives of behavior and learning resources, and use TransR technology to obtain the embedded representation of the course knowledge graph after data augmentation from the perspectives of behavior and learning resources;

[0056] Step 3: Based on the embedding representations obtained in Step 1 and Step 2, feature extraction is performed using a graph convolutional neural network with an attention mechanism.

[0057] Step 4: Treat the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. Provide reward feedback to the agent at both the knowledge point level and the path level to recommend personalized learning paths for the user.

[0058] The method specifically includes:

[0059] 1. Problem Description

[0060] We use KG c This represents the course knowledge graph of a particular course. When a user studies this course, they interact with its learning resources. For each user, we use l. i To represent his historical learning path, l s Indicates the initial learning resource, l e This represents the target learning resource. Our goal is to plan a learning path leading to the target learning resource based on the user's personalized factors such as learning ability, knowledge background, and learning goals. LP = {l s ,…,l j ,…,l e}, where l j These are the optimal knowledge points selected from the course knowledge graph.

[0061] 2. Heterogeneous Knowledge Graph Enhancement

[0062] Based on semantic information and behavioral timing, this invention categorizes online learning behaviors into four types: entry, learning, marking, and submission. Different categories of behavior reflect users' varying willingness to interact with learning resources. Capturing the potential relationship between users and behaviors can uncover their characteristics; for example, entry and submission behaviors always occur earliest and latest, respectively, while marking behavior reflects a user's willingness to continue learning. For each user's historical interaction sequence with a series of learning resources... i We will derive a sequence l containing these four behaviors. b This process is illustrated in the behavior classification diagram within the overall framework.

[0063] This invention treats four types of behavioral nodes as entity nodes in a knowledge graph, connecting them with user nodes and learning resource nodes. The triplet relationships between them are: (user, generate, behavioral node); (behavioral node, act on, learning resource). This process is illustrated in the knowledge graph enhancement section of the overall framework.

[0064] For ease of description, this invention uses a set of triples to represent the course knowledge graph, i.e., KG. c ={(h1,r1,t1),……(h n ,r n ,t n )}, where h i For the head entity, t i For the tail entity, r i Let n be the number of triples representing the relationships between them. If m action nodes are added, then 2m relations (i.e., 2m triples) need to be added. The enhanced knowledge graph is then represented as: KG′ c =Kg c ∪{(h′1,r′1,t′1),……,(h′ 2m ,r′ 2m ,t′ 2m )}

[0065] In course knowledge graphs, the relationships between learning resources at various levels are highly complex. For example, a course may correspond to multiple chapters, a chapter to multiple videos and exercises (interactive learning resources), and videos and exercises may correspond to multiple knowledge points, with many-to-many relationships between these knowledge points. Therefore, this invention uses TransR technology for knowledge graph embedding. It maps entities and relations to different vector spaces to solve the problem of relation representation. Because it can separate the representation space, each relation has its own projection matrix, which can effectively handle one-to-many and many-to-many relationships. The TransR process is as follows: Figure 2 As shown.

[0066] Let the triple (h,r,t) be represented as h,t∈R k , r∈R dWhere k and d are different dimensions. Each relation r has a corresponding mapping matrix M. r ∈R k×d Through M r Map the head and tail entities h, t to h in the relation space. r and t r Then use h r +r≈t r The approach method brings related entities closer together and excludes unrelated entities, where h r =h·M r , t r =t·M r The loss function is:

[0067]

[0068] in The squared value of the distance is represented by the right-hand index, which indicates the L2 norm, representing the linear distance between two vectors in space. Based on this, the knowledge graph representation embedding can be obtained using the tans R technique, expressed as:

[0069] u,h i ,h b ,h c =trans R(KG′) c (2)

[0070] Where u is the user's feature vector, h i ,h b ,h c For learning resource node embedding vectors, behavior node embedding vectors, and knowledge point node embedding vectors, h i ,h b ,h c ∈R d .

[0071] 3. Knowledge Graph Feature Extraction Based on Graph Convolutional Networks and Contrastive Learning

[0072] 3.1 Data Augmentation

[0073] To expand the amount of data generated from user interactions with learning resources, this invention proposes data augmentation operators from both behavioral and learning resource perspectives: behavioral data reconstruction and learning resource interaction reorganization. A schematic diagram of our proposed data augmentation is shown below. Figure 3 As shown.

[0074] (1) Behavioral data reconstruction. Behavioral data reconstruction refers to using probability as the basis for data reconstruction. Modifying user interactions with learning resources includes altering user tagging behaviors such as deleting comments, deleting likes, and changing ratings. It also includes modifying video viewing speed, the number of times a question can be answered, and the time taken to complete a question. In heterogeneous knowledge graphs, behavioral data reconstruction is considered as deleting or modifying attributes of behavioral nodes.

[0075] (2) Learning Resource Interaction Reorganization. This invention proposes an improved random masking method for processing user-learning resource interaction sequences. Specifically, we reorganize each user's interaction sequence according to a certain proportion. By performing random masking, this method enhances the original interaction records and avoids biases caused by data sparsity.

[0076] Furthermore, the interaction order of learning resources is highly sensitive for learning path recommendation, determining the recommended order. To ensure the relevance between learning resources and the overall knowledge progression order are not severely disrupted, some learning resources belonging to the same chapter are partially swapped to increase data robustness. The probability of order modification is considered as... For example, the order of a set of learning resources for user interaction is: l i ={p1,…p i ,p i+1 ,…,p n If p i p i+1 If they belong to the same superior learning resource, then they are determined by probability. The order of interaction for exchanging learning resources.

[0077] 3.2 Feature Extraction Based on Attention Mechanism and Graph Convolutional Networks

[0078] After embedding the enhanced (heterogeneous) knowledge graph into TransR, we use an attention-based graph convolutional neural network to perform semantic propagation within the heterogeneous knowledge graph, capturing its multi-source heterogeneous features. Feature acquisition based on the attention mechanism and graph convolutional network is as follows: Figure 4 As shown.

[0079] In our enhanced course knowledge graph, entity nodes include user nodes, behavior nodes, learning resource nodes at all levels, and knowledge point nodes. The neighborhood set of each entity h in the heterogeneous knowledge graph is:

[0080]

[0081] Where h is the head entity, t is the tail entity connected to h, and r is the relation. Each entity uses a fixed-size neighborhood, i.e., |N|. (h)| = L, where L represents the number of propagation layers. We use triples (h, r, t) to represent heterogeneous knowledge graphs, and for each triple, we employ an attention mechanism to calculate the weights of entities in the neighborhood set of the tail entity t (the target entity). We use the distance in the embedding space to calculate the attention scores of neighboring entities related to the target entity:

[0082]

[0083] Where e t The embedding vector representing the target entity. and e r Let represent the neighbor entity vector and relation vector corresponding to the target entity, respectively. The attention score depends on the sum of these two vectors. We use the softmax function to normalize the attention scores (weights) of the neighbor entities:

[0084]

[0085] Based on this, we can select suitable neighbor entities using attention scores. We extract the structure of neighbor entities by calculating a linear set of the self-networks of the target entity t:

[0086]

[0087] Where e represents the embedding representation of the neighboring entities of the target entity t. Finally, we use an aggregator to aggregate information, combining the target entity vector with a linear set of its self-networks into a single vector. For simplicity, we use the GCNAggregator aggregator to obtain the final embedding of the target entity as follows:

[0088]

[0089] Where W is the transformation weight matrix and W∈R d×d′ d′ represents the size of the transformation. Based on this, we obtain the result of processing by a single-layer graph convolutional network. After aggregating information through L layers, we can obtain the user's final embedding vector U, and further construct the learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence:

[0090]

[0091] in n1, n2, and n3 represent their quantities, respectively.

[0092] We use a negative sampling loss function with self-adversarial training for training.

[0093]

[0094]

[0095] Where γ is the fixed margin, σ is the sigmoid function, and (h′) i ,r,t′ i ) is the i-th negative triplet, α is the sampled temperature, and f(·) is the scoring function.

[0096] We perform data augmentation on the original data to obtain two augmented views. After propagation through GCN, we obtain the final feature embeddings of the two augmented views of the same user's historical interaction data:

[0097] U′,H′ i ,H′ b ,h′ c =GCN(augment1(u,h) i ,h b ,h c (13)

[0098] U″,H″ i ,H″ b ,H″ c =GCN(augment2(u,h) i ,h b ,h c (14)

[0099] Where augment1(·) and augment2(·) represent functions that use three data augmentation strategies in different proportions.

[0100] 3.3 Comparative Learning

[0101] To obtain more supporting data, we introduce contrastive learning to mine supervisory signals from the data itself, providing additional data support for reinforcement learning. We generate two relevant views for the same user combination using data augmentation strategies, treating them as positive sample pairs, and then process them using contrastive learning to provide additional data and state information for reinforcement learning. First, the interaction sequences of the augmented views are passed through an average pooling function to obtain a more compact embedding representation:

[0102] Here, pool(·) is the average pooling function. For each positive sample pair (U′, U″), we instantiate it as a sequence of learned resource embeddings. Behavior node embedding sequence and knowledge point embedding sequence Then the negative samples in the same batch of data are z. I,- ∈Z I,- , z B,- ∈Z B,- and z C,-∈Z C,- Their loss functions are:

[0103]

[0104] Where L I L B L C These represent the comparison loss at the learning resource level, the behavior node level, and the knowledge point level, respectively; sim() represents the similarity function. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the first augmented view, respectively. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the second enhanced view, respectively; Z I,- , Z B,- , Z C,- These are negative samples from the same batch of data.

[0105] Therefore, the loss function for contrastive learning is L. CL Where L CL =L I +L B +L C .

[0106] 4. Learning path recommendation based on reinforcement learning

[0107] This invention describes the learning path recommendation task based on reinforcement learning as a Markov decision. Specifically, it consists of (state space, action space, transition probabilities, reward function, and discount factor).

[0108] State: Represents the user's learning state space.

[0109] Action space: The discrete action space of unlearned knowledge points in the current state.

[0110] Reward: You will receive some feedback after recommending a knowledge point, which will be recorded as a reward.

[0111] Transition probability: The probability of transitioning from the current state to the next state.

[0112] Discount factor: Represented by the symbol γ, with a value of [0,1], it reflects the agent's trade-off between current rewards and future rewards.

[0113] 4.1 User Status

[0114] This invention treats the output of a graph convolutional neural network as the user's state. Furthermore, it incorporates personalized factors such as the user's learning goals, learning progress, and learning preferences into the state consideration. This personalized information is represented by a context vector q. Therefore, the user's three state components at time t are: s t 1 =join(U,H) b H i H c ,q), s t 2 =join(U′,H′) b ,H′ i ,H′ c ,q), s t 3 =join(U″,H″) b ,H″ i ,H″ c ,q).

[0115] The `join(·)` function is used to concatenate vectors. Each user has different learning goals, learning styles, learning preferences, learning progress, and other information, all of which are represented by a context vector `q`.

[0116] 4.2 Deep Q-Network

[0117] We view the deep Q-network as an intelligent agent that makes the most appropriate decision for the user. The entire Q-network is divided into two types: the present value deep Q-network and the target value deep Q-network.

[0118] The present-value deep Q-network is used to calculate the Q-value resulting from the current decision, with the goal of selecting the knowledge points that maximize the Q-value. This invention considers the linear sum of the Q-values ​​of the three states as the Q-value at time t:

[0119] Q(s t ,a t )=[Q(s t 1 ,a t ),Q(s t 2 ,a t ),Q(s t 3 ,a t (18)

[0120] Where s t In the state at time t, a t In state s t The actions taken below. The target depth Q-function is an independent neural network that provides a stable target Q-value, avoiding training instability caused by frequent changes in the target value during the training of the current depth Q-function.

[0121]

[0122] Where r is the immediate reward obtained from the current action a, γ is the discount factor, s′ is the next state transitioned to after performing action a, and max a′ Q target (s′,a′,θ - ) is the maximum Q-value prediction of the target network for all possible actions a' in the next state s′, θ - y represents the parameters of the target network, which are fixed and updated periodically, and y represents the updated target Q-value.

[0123] 4.3 Experience Replay Pool

[0124] To break the correlation between data, this invention sets up an experience replay pool. During the interaction between the agent and the environment, the state transitions experienced are stored in the experience replay buffer. During each training session, a small batch of samples is randomly drawn from the buffer for training.

[0125] Initially, a random seed is set to initialize the experience replay pool, the parameters θ of the current Q-network are randomly initialized, and the parameters are copied to the target Q-network so that θ - =θ, and then training began.

[0126] 4.4 Reward Function

[0127] Learning path recommendation involves numerous resource choices and a vast action space, making it a task with long-term feedback characteristics, and it generally suffers from the inherent problem of reward sparsity. This invention provides reward feedback to the agent at both the knowledge point level and the path level.

[0128] R(s t ,a t )=(1-α)·β points (s t ,a t )+α·R path (s t ,a t (20)

[0129] Among them, R points Represents the knowledge-point level reward function, R pat This represents the path-level reward function, where α is a weight parameter and a represents the action to be performed in the current state.

[0130] The reward function at the knowledge point level measures the accuracy of the next recommended knowledge point. Predicted knowledge points and actual interactive knowledge points are mapped to a low-dimensional vector space, and their cosine similarity is calculated to determine similarity. Furthermore, some knowledge points also exhibit certain pre-recommendation / successor relationships. The reward function at the knowledge point level is shown below.

[0131]

[0132] Where β1 and β2 are weighting coefficients, T represents the number of knowledge points, and φ(point) t ,point t+1 (a knowledge point) t with point t+1 The function of the dependency relationship between them and φ(·)∈{0,1}, emb rec and emb hi This indicates the embedding of predictions with actual knowledge points.

[0133] At the path level, paths are mapped to a low-dimensional vector space and treated as sequences of knowledge points. The similarity of paths is quantified by calculating the minimum number of edit operations required to transform the predicted sequence into an actual interaction sequence. The reward function at the path level is shown below.

[0134]

[0135] Where κ is the penalty weight, Edit(P) rec ,P his ) represents the number of edits required to convert the predicted learning path into the actual learning path, max(|P rec |,|P his |) represents the maximum length of the two learning paths after normalization. R path (s t ,a t The range of ) is [-γ, 0]. The closer the two paths are, the closer the value is to 0. The greater the difference between the paths, the closer it is to -γ.

[0136] 4.5 Loss Function

[0137] The loss function for learning path recommendation based on reinforcement learning is:

[0138]

[0139] Where γ is the discount factor, the expected value E represents uniform sampling of all samples in the replay pool, s′ represents the state of the target Q network, s represents the state of the current Q network, and a represents the action to be performed.

[0140] 5. Objective function

[0141] The overall objective function of the framework proposed in this invention is L. T L G L CL L R The linear sum:

[0142] L = L T +L G +L CL +L R (twenty four)

[0143] Based on the above embodiments, such as Figure 5 As shown, this invention also proposes a learning path recommendation system based on reinforcement learning for user multi-behavior scenarios, comprising:

[0144] The knowledge graph enhancement module is used to classify the interaction behaviors between users and learning resources; different types of behaviors are regarded as entity nodes of the knowledge graph, which are connected with user nodes and learning resource nodes to obtain the fused and enhanced course knowledge graph, and the TransR technology is used to obtain the embedded representation of the fused and enhanced course knowledge graph.

[0145] The data augmentation module is used to perform data augmentation from the perspectives of behavior and learning resources based on the fused and augmented course knowledge graph, and uses TransR technology to obtain the embedded representation of the course knowledge graph data augmented from the perspectives of behavior and learning resources.

[0146] The knowledge graph feature extraction module extracts features based on the embedded representations obtained from the knowledge graph augmentation module and the data augmentation module, using a graph convolutional neural network with an attention mechanism.

[0147] The reinforcement learning-based learning path recommendation module treats the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. It provides reward feedback to the agent at both the knowledge point level and the path level, and recommends personalized learning paths for the user.

[0148] Furthermore, such as Figure 6 As shown, the system also includes:

[0149] The contrastive learning module is used to embed the course knowledge graph, which is augmented based on behavioral and learning resource data, into the corresponding features for contrastive learning.

[0150] This invention proposes a knowledge graph-based learning behavior modeling method. By systematically deconstructing the interaction process between users and learning resources, complex learning behaviors are divided into four semantically distinct categories: entry, learning, marking, and submission. These behavior nodes, along with user entity nodes and learning resource entity nodes, constitute a three-dimensional interaction relationship. Based on this, this invention innovatively constructs a triple-association knowledge graph model of "user-behavior-learning resource," achieving fine-grained representation of the learning process and semantic enhancement of the knowledge graph through in-depth mining of the relationships between nodes.

[0151] This invention innovatively proposes a data augmentation strategy based on two dimensions: behavior data reconstruction and learning resource interaction reorganization. By combining data augmentation strategies on the original view, we obtain two augmented views of the same user's historical interaction data. We then utilize an attention-based graph convolutional neural network to extract hierarchical semantic features and introduce a cross-view contrastive learning mechanism to effectively capture the relevant information between the two augmented views, providing additional supplementary information for the reinforcement learning state. This technical solution not only significantly expands the diversity of training samples but also provides more comprehensive data support for the reinforcement learning model, thereby greatly improving the accuracy and adaptability of personalized recommendations.

[0152] This invention proposes a personalized learning path recommendation method based on deep reinforcement learning. It dynamically selects the optimal learning resources and plans a personalized learning path for each learner using a deep Q-network. We deeply consider the individual characteristics of different users, such as their learning styles, preferences, goals, and abilities, and dynamically allocate the most suitable learning resources accordingly. Furthermore, to address the inherent reward sparsity problem in learning path recommendation, we innovatively design a dual reward mechanism at both the knowledge point and path levels to aid our model in planning and convergence.

[0153] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A learning path recommendation method based on reinforcement learning for user multi-behavior scenarios, characterized in that, include: Step 1: Classify user interactions with learning resources; Different types of behaviors are treated as entity nodes in the knowledge graph, connected with user nodes and learning resource nodes to obtain a fused and enhanced course knowledge graph, and the TransR technology is used to obtain the embedded representation of the fused and enhanced course knowledge graph. Step 2: Based on the fused and enhanced course knowledge graph, perform data augmentation from the perspectives of behavior and learning resources, and use TransR technology to obtain the embedded representation of the course knowledge graph after data augmentation from the perspectives of behavior and learning resources; Step 3: Based on the embedding representations obtained in Step 1 and Step 2, feature extraction is performed using a graph convolutional neural network with an attention mechanism. Step 4: Treat the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. Provide reward feedback to the agent at both the knowledge point level and the path level to recommend personalized learning paths for the user.

2. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 1, characterized in that, Step 1 includes: classifying user interactions with learning resources into four categories: entry, learning, marking, and submission.

3. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 1, characterized in that, In step 2, data augmentation is performed from a behavioral perspective in the following manner: With probability as Modify the interaction between users and learning resources.

4. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 1, characterized in that, In step 2, data augmentation is performed from the perspective of learning resources in the following manner: The interaction sequence of each user with the learning resources is arranged in a certain proportion. Random blocking is implemented; Partial exchange of learning resources belonging to the same chapter.

5. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 1, characterized in that, Before step 4, the method further includes: embedding the course knowledge graph, which is augmented based on behavioral and learning resource data, into the corresponding features for comparative learning.

6. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 5, characterized in that, The loss function for contrastive learning is: L CL =L I +L B +L C , In the formula Where L CL L represents the loss function for contrastive learning. I L B L C These represent the contrast loss at the learning resource level, the contrast loss at the behavior node level, and the contrast loss at the knowledge point level, respectively; sim() represents the similarity function. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the first augmented view, respectively. These represent the average pooled learning resource embedding sequence, behavior node embedding sequence, and knowledge point embedding sequence in the second enhanced view, respectively; Z I,- Z B,- Z C,- These are negative samples from the same batch of data.

7. The learning path recommendation method based on reinforcement learning for user multi-behavior learning according to claim 1, characterized in that, The user's status also includes: the user's learning goals, learning progress, and learning preferences.

8. A learning path recommendation method based on reinforcement learning for user multi-behavior scenarios according to claim 1, characterized in that, In step 4, the agent is rewarded at both the knowledge point level and the path level in the following manner: R(s t ,a t )=(1-a)·R points (s t ,a t )+α·R path (s t ,a t ), In the formula, Where R represents the reward function, R points R represents the knowledge-point level reward function. path This represents the path-level reward function, where α is a weight parameter, and s t In the state at time t, a t In state s t The actions to be taken are β1 and β2, which are weighting coefficients, T represents the number of knowledge points, and φ (point) t ,point t+1 (a knowledge point) t With pint t+1 The function of the dependency relationship between them and φ(·)∈{0,1}, emb rec and emb his This represents the embedding of predictions with actual knowledge points, where κ is the penalty weight, and Edit(P) rec ,P his ) represents the number of edits required to convert the predicted learning path into the actual learning path, where max(|P) represents the number of edits required. rec |,|P his |) represents the maximum length of the two learning paths after normalization, R path (s t ,a t The range of ) is [-γ, 0], where γ is the discount factor.

9. A learning path recommendation system based on reinforcement learning for user multi-behavior scenarios, characterized in that, include: The knowledge graph enhancement module is used to classify user interactions with learning resources; Different types of behaviors are treated as entity nodes in the knowledge graph, connected with user nodes and learning resource nodes to obtain a fused and enhanced course knowledge graph, and the TransR technology is used to obtain the embedded representation of the fused and enhanced course knowledge graph. The data augmentation module is used to perform data augmentation from the perspectives of behavior and learning resources based on the fused and augmented course knowledge graph, and uses TransR technology to obtain the embedded representation of the course knowledge graph data augmented from the perspectives of behavior and learning resources. The knowledge graph feature extraction module extracts features based on the embedded representations obtained from the knowledge graph augmentation module and the data augmentation module, using a graph convolutional neural network with an attention mechanism. The reinforcement learning-based learning path recommendation module treats the output of the graph convolutional neural network as the user's state and the deep Q-network as the agent. It provides reward feedback to the agent at both the knowledge point level and the path level, and recommends personalized learning paths for the user.

10. A learning path recommendation system based on reinforcement learning for user multi-behavior scenarios according to claim 9, characterized in that, Also includes: The contrastive learning module is used to embed the course knowledge graph, which is augmented based on behavioral and learning resource data, into the corresponding features for contrastive learning.

Images