Personalized video learning recommendation method fusing hierarchical reinforcement learning and knowledge graph

Abstract

The invention relates to the technical field of online education recommendation, in particular to a personalized video learning recommendation method fusing hierarchical reinforcement learning and a knowledge graph, and the method comprises the steps: obtaining video data and a user portrait vector; wherein the video data comprises video resources and knowledge points associated with the video resources; constructing a target knowledge graph based on the video resources and the knowledge points associated with the video resources, and generating a knowledge graph embedding vector based on a graph neural network; and generating a personalized recommendation result of the user through a knowledge grouping aggregation mechanism and an attention mechanism based on the user portrait vector and the knowledge graph embedding vector, and generating a self-adaptive learning path of the user based on the user portrait vector and a target knowledge graph when the user selects a learning target. According to the method, the technical problems of noise historical interference, weak knowledge relevance, inaccurate path planning and the like in a traditional recommendation system can be solved.

Description

Technical Field

[0001] The embodiments of this application relate to the field of online education recommendation technology, and in particular to a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs. Background Technology

[0002] With the rapid development of internet technology and mobile devices, online education has become an important part of the education system. Platforms such as Massive Open Online Courses (MOOCs) provide users with a wealth of diverse learning resources, breaking through the time and space limitations of traditional education. However, with the rapid growth in the number of courses, these platforms face new challenges.

[0003] Existing platforms generally employ recommendation methods based on collaborative filtering, content feature matching, or sequence modeling. While these methods achieve resource filtering to some extent, they still have significant limitations. Collaborative filtering methods have limited effectiveness in handling sparse data and cold-start scenarios; content-based recommendations often struggle to capture deep semantic relationships within knowledge; and sequence recommendation methods are sensitive to noisy data and lack effective differentiation of learning behavior weights.

[0004] More importantly, the current platform's recommendation mechanism has the following shortcomings: The knowledge correlation mining is insufficient, and a logical relationship network between knowledge points has not been systematically constructed; the recommendation granularity is too coarse, mainly staying at the course level, and lacking fine recommendations at the knowledge unit level; the degree of personalization is limited, and it is difficult to dynamically adjust the recommendation strategy according to the user's real-time learning status; for users with clear learning goals, the path planning function is missing, and it is unable to provide users with a systematic learning plan from the current level to the achievement of the goal.

[0005] While existing knowledge graph augmentation methods have improved semantic understanding to some extent, there is still room for improvement in terms of relationship mining depth and personalized adaptation. Furthermore, reinforcement learning-based recommendation methods often oversimplify the decision-making process and struggle to handle complex, multi-layered learning scenarios. Summary of the Invention

[0006] To address the aforementioned technical issues, embodiments of this application propose a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs, aiming to solve technical problems in traditional recommendation systems such as historical noise interference, weak knowledge relevance, and inaccurate path planning.

[0007] To achieve the above objectives, embodiments of this application propose a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs, the method comprising the following steps: Acquire video data and user profile vectors; the video data includes video resources and related knowledge points. Based on video resources and the knowledge points associated with them, a target knowledge graph is constructed, and knowledge graph embedding vectors are generated based on graph neural networks. The target knowledge graph is used to represent the heterogeneous graph structure of video resource nodes, knowledge point nodes, and relation edges, and the knowledge graph embedding vectors include video embedding vectors and knowledge point embedding vectors. Based on user profile vectors and knowledge graph embedding vectors, personalized recommendation results are generated for users through knowledge grouping and aggregation mechanisms and attention mechanisms. Furthermore, when a user selects a learning target, an adaptive learning path is generated for the user based on the user profile vector and the target knowledge graph.

[0008] To achieve the above objectives, embodiments of this application also propose a personalized video learning recommendation device that integrates hierarchical reinforcement learning and knowledge graphs, the device comprising: The acquisition module is used to acquire video data and user profile vectors; the video data includes video resources and knowledge points associated with the video resources. The construction module is used to build a target knowledge graph based on video resources and knowledge points associated with video resources, and to generate knowledge graph embedding vectors based on graph neural networks. The target knowledge graph is used to represent a heterogeneous graph structure of video resource nodes, knowledge point nodes and relation edges, and the knowledge graph embedding vectors include video embedding vectors and knowledge point embedding vectors. The generation module is used to generate personalized recommendation results for users based on user profile vectors and knowledge graph embedding vectors, through knowledge grouping and aggregation mechanisms and attention mechanisms. When a user selects a learning target, it generates an adaptive learning path for the user based on user profile vectors and target knowledge graphs.

[0009] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph as described above.

[0010] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs as described above.

[0011] This application proposes a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs. First, video data and user profile vectors are acquired. The video data includes video resources and related knowledge points. Then, a target knowledge graph is constructed based on the video resources and related knowledge points, and a knowledge graph embedding vector is generated using a graph neural network. Next, based on the user profile vector and the knowledge graph embedding vector, a personalized recommendation result is generated for the user through a knowledge grouping and aggregation mechanism and an attention mechanism. Finally, when the user selects a learning objective, an adaptive learning path is generated for the user based on the user profile vector and the target knowledge graph. Compared to traditional fixed-path methods, this application can accurately generate personalized recommendation results and adaptive learning paths for users based on their own circumstances, video resources, and related knowledge points, thereby reducing historical noise interference in traditional recommendation systems and improving the accuracy of knowledge relevance and path planning.

[0012] Optionally, acquiring video data and user profile vectors includes: acquiring user historical behavior data and a target database; wherein, user historical behavior data includes user browsing history data and user static features, user static features include user personal information, and the target database is used to represent the mapping relationship between video resources and knowledge point data; based on the mapping relationship between video resources and knowledge point data, acquiring video resources and knowledge point data associated with video resources; and based on a hierarchical reinforcement learning algorithm, correcting the browsing history data and static features to obtain a corrected user profile vector; wherein, the high-level agent in the hierarchical reinforcement algorithm is used to filter key historical records in user historical behavior data based on the policy gradient method, and the low-level agent is used to assign feature weights to key historical records based on the Q-learning algorithm to eliminate the interference of noisy history.

[0013] Optionally, the specific implementation process for correcting browsing history data and static features based on hierarchical reinforcement learning algorithms is as follows: The process of correcting browsing history data and static features is modeled as a hierarchical Markov decision process, where the state space of the hierarchical Markov decision process includes video resources and user history behavior data. The high-level agent adopts the Actor-Critic policy gradient method. The Actor network outputs the action to be selected based on browsing history data, and the Critic network evaluates the value of the state and provides an advantage function for policy updates. The formula for the Actor-Critic policy gradient method is as follows (1): (1); in, For network parameters, For the policy probability, The dominant function; The low-level agent uses the DoubleDQN algorithm to assign feature weights to the key historical records. The main Q network and the target Q network are maintained and updated periodically. The corrected user profile vector is calculated through weighted aggregation and feature fusion, specifically expressed by the following formula (2): (2); in, The number of historical records to be retained. Normalized weights assigned to lower-level agents. For the embedding vector of key historical records, It is a static feature. This is the feature fusion matrix.

[0014] Optionally, based on user profile vectors and knowledge graph embedding vectors, personalized recommendation results are generated for users through knowledge grouping and aggregation mechanisms and attention mechanisms, including: The corrected user profile vectors and knowledge graph embedding vectors of video resources are input into the KGAN recommendation network; By leveraging the knowledge grouping and aggregation mechanism in the KGAN recommendation network, the probability of a user clicking on candidate videos is predicted, thus outputting Top-K personalized recommendation results.

[0015] Optionally, the knowledge grouping and aggregation mechanism of the KGAN recommender network is as follows: The knowledge points associated with video resources are grouped according to their relationship type to obtain knowledge groups; Based on the self-attention enhancement intra-group attention mechanism, the attention weights of the knowledge points in each knowledge group are calculated using a multilayer perceptron, as shown in the following formula (3): (3); in, For user profile vectors, Embed the knowledge points into vectors after self-attention enhancement. It is a multilayer perceptron. This represents vector concatenation. For relation type The corresponding knowledge group; Simultaneously, an intra-group aggregation vector is generated, which is obtained through the following formula (4): (4); A cross-relationship fusion inter-group attention mechanism is applied to all knowledge groups. By integrating user profile vectors, video embedding vectors, and knowledge point embedding vectors, the inter-group attention weights are calculated to obtain the final knowledge aggregation vector. Through multi-layer interactive network computation, the user profile vector, knowledge aggregation vector, and video embedding vector are concatenated and then passed through a fully connected layer and... The activation function outputs personalized recommendation results.

[0016] Optionally, when a user selects a learning objective, an adaptive learning path is generated based on the user profile vector and the target knowledge graph, including: When a user selects a learning goal, retrieve the knowledge points corresponding to the selected learning goal; Extract all predecessor knowledge point sets and target reachable knowledge point sets from the target knowledge graph, and calculate the user's comprehensive mastery of the knowledge points, integrating viewing time, repetition count and test scores, specifically expressed by the following formula (5): (5); in, , , These are the weighting coefficients. To accumulate viewing time, For the number of times to watch repeatedly, Scores for knowledge point tests; Obtain the knowledge point difficulty function; where the knowledge point difficulty function includes the number of prerequisites, knowledge complexity, and average learner performance; The Dijkstra algorithm is used to calculate the optimal path from the user's current mastery level to the learning goal, and corresponding video resources are matched for each knowledge point in the optimal path to form an adaptive learning path. The state value function in Dijkstra's algorithm is expressed by the following formula (6): (6); in, To determine the degree threshold, As a discount factor, The difficulty function of the knowledge points. Penalty for path deviation Indicates the direct successor knowledge point of a knowledge point. The state value function.

[0017] Optionally, the method provided in this application embodiment further includes: collecting user feedback data on personalized recommendation results and adaptive learning paths, so as to use the feedback data as a reward signal for reinforcement learning; storing the interaction experience corresponding to the feedback data using a priority experience replay mechanism; and updating the loss function parameters of the agent policy network and recommendation network of the hierarchical reinforcement learning algorithm based on the reward signal and interaction experience.

[0018] Optionally, the method provided in this application embodiment further includes: processing each video resource and the knowledge points associated with each video resource using a large language model, and matching the video resource with the highest recommendation priority for each knowledge point; detecting whether the user is a new user; if the user is a new user, providing the new user with the video resource with the highest recommendation priority for each knowledge point. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0020] Figure 1 This is a flowchart of a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs, provided in one embodiment of this application; Figure 2 This is a schematic diagram of a hierarchical reinforcement learning framework provided in one embodiment of this application; Figure 3 This is a partial knowledge graph structure diagram provided in one embodiment of this application; Figure 4 This is a schematic diagram of the player and personalized recommendation interface provided in one embodiment of this application; Figure 5 This is a schematic diagram of the learning path recommendation interactive interface provided in one embodiment of this application; Figure 6 This is a schematic diagram of the overall system architecture provided in one embodiment of this application; Figure 7 This is a schematic diagram of the cold start module process provided in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of a personalized video learning recommendation device that integrates hierarchical reinforcement learning and knowledge graph provided in another embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0022] With the rapid development of internet technology and mobile devices, online education has become an important part of the education system. Platforms such as Massive Open Online Courses (MOOCs) provide users with a wealth of diverse learning resources, breaking through the time and space limitations of traditional education. However, with the rapid growth in the number of courses, these platforms face new challenges.

[0023] Existing platforms generally employ recommendation methods based on collaborative filtering, content feature matching, or sequence modeling. While these methods achieve resource filtering to some extent, they still have significant limitations. Collaborative filtering methods have limited effectiveness in handling sparse data and cold-start scenarios; content-based recommendations often struggle to capture deep semantic relationships within knowledge; and sequence recommendation methods are sensitive to noisy data and lack effective differentiation of learning behavior weights.

[0024] More importantly, the current platform's recommendation mechanism has the following shortcomings: The knowledge correlation mining is insufficient, and a logical relationship network between knowledge points has not been systematically constructed; the recommendation granularity is too coarse, mainly staying at the course level, and lacking fine recommendations at the knowledge unit level; the degree of personalization is limited, and it is difficult to dynamically adjust the recommendation strategy according to the user's real-time learning status; for users with clear learning goals, the path planning function is missing, and it is unable to provide users with a systematic learning plan from the current level to the achievement of the goal.

[0025] While existing knowledge graph augmentation methods have improved semantic understanding to some extent, there is still room for improvement in terms of relationship mining depth and personalized adaptation. Furthermore, reinforcement learning-based recommendation methods often oversimplify the decision-making process and struggle to handle complex, multi-layered learning scenarios.

[0026] In view of this, in order to solve the above-mentioned technical problems, the embodiments of this application propose a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph, which aims to solve the technical problems of historical noise interference, weak knowledge correlation and inaccurate path planning in traditional recommendation systems. For details, please refer to the following text.

[0027] One embodiment of this application proposes a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs, applied to an electronic device. The electronic device can be a terminal or a server; this embodiment and subsequent embodiments will use a server as an example. The implementation details of the personalized video learning recommendation method integrating hierarchical reinforcement learning and knowledge graphs proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0028] The specific process of the personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Obtain video data and user profile vectors.

[0029] The video data includes video resources and related knowledge points.

[0030] For example, a video resource can be a video unit with complete teaching content in an online education platform, and the knowledge points associated with the video resource can be knowledge points related to that teaching content. For example, if the video resource is "Basic Calculus: Detailed Explanation of Derivatives and Differentials", then the knowledge points associated with the video resource could be the definition of mappings, the geometric meaning of derivatives, etc.

[0031] For example, a user profile vector can be modified user historical behavior data, such as the user's knowledge level, learning behavior preferences, cognitive ability characteristics, etc., which can be represented by vectors.

[0032] In one possible embodiment, step 101 includes: acquiring user historical behavior data and a target database; acquiring video resources and related knowledge point data based on the mapping relationship between video resources and knowledge point data; and correcting browsing history data and static features based on a hierarchical reinforcement learning algorithm to obtain a corrected user profile vector.

[0033] The user's historical behavior data includes the user's browsing history data and the user's static features. The user's static features include the user's personal information. The target database is used to represent the mapping relationship between video resources and knowledge point data. The high-level agent in the hierarchical reinforcement algorithm is used to filter key historical records in the user's historical behavior data based on the policy gradient method, and the low-level agent is used to assign feature weights to key historical records based on the Q-learning algorithm to eliminate the interference of noisy history.

[0034] For example, a user's browsing history data may include time-series viewing records, such as records in the format of [user ID, video ID, start time, end time, viewing duration, completion rate, rating], fine-grained interaction behavior data, etc.

[0035] For example, a user's static characteristics may include the user's personal information. Static characteristics can represent relatively stable personal attributes, such as age and educational background.

[0036] For example, when acquiring video data, the text data after a series of processing steps from the source video can be input into a large model in batches. The large model can then process the text data multiple times, correct text errors, and divide it into knowledge points. Finally, a JSON file can be output for each video to store the timestamp information and description information of all knowledge points contained in the video.

[0037] For example, source video files to be processed are obtained from a video database, and audio tracks are extracted from the videos using FFmpeg for subsequent speech recognition. The extracted audio files are input into a large language model to convert the speech content into text and record timestamp information. The extracted raw text is cleaned, including removing filler words, correcting recognition errors, standardizing punctuation, and deleting duplicate sentences. Long texts are segmented into several paragraphs based on timestamps and semantic coherence, with each paragraph's length controlled between 100 and 500 characters to ensure semantic integrity. Each text paragraph is labeled with its start and end times in seconds. Due to the large model's limitation on input text length, the text data needs to be input in batches. Natural segmentation is performed, with each batch containing 3 to 5 consecutive paragraphs to ensure contextual coherence. The number of tokens in each batch is calculated, ensuring that the number of tokens in a single batch does not exceed 80% of the model's limit, leaving space for model output. Overlapping areas are preserved between batches to prevent knowledge points from being truncated at batch boundaries. Contextual information is constructed for each batch, including the video title, the current batch's time range, and a summary of knowledge points from the previous batch. A suitable large language model is selected for text processing. Three rounds of processing are performed: The first round involves text correction, identifying and correcting speech recognition errors, grammatical errors, and punctuation errors, converting colloquial expressions into written language. The second round involves knowledge point identification, recognizing knowledge points from the text that have clear concepts, independent teaching value, and clear start and end boundaries. The third round involves knowledge point refinement, supplementing each knowledge point with information such as name, detailed description, difficulty level, keywords, and prerequisite knowledge. If the output quality is unsatisfactory, multiple rounds of interactive optimization are performed. A JSON file is generated for each video, named "video_name.json". The JSON file contains basic information such as the name, description, and timestamp of all knowledge points contained in the video. An example is shown using the file "mapping.json". "knowledge_points": [ { "time": "02:36-03:01", "name": "Mapping definition", "description": "Let X and Y be non-empty sets. If there exists a rule f such that for every x in X, there is a unique y in Y, then f is called a mapping from X to Y." }, { "time": "05:15-07:20", "name": "Geometric meaning of the derivative", "description": "The derivative of a function at a point is equal to the slope of the tangent line to its graph at that point." } ] } Next, the data from the JSON file is imported into the database to create a video table and a knowledge point table, which are linked by video IDs. A full-text index is created for the name, description, and keyword fields of the knowledge points, supporting fast text retrieval. A pre-trained text encoding model is used to convert the knowledge point descriptions into vector representations, which are stored in the database, supporting semantic similarity retrieval.

[0038] Video metadata, knowledge point data, and user behavior data are collected from databases (e.g., video databases and knowledge point databases). Video metadata includes video ID, title, description, duration, author, etc. Knowledge point data includes knowledge point ID, name, detailed description, associated video, time range (start and end seconds), difficulty level, etc. User behavior data includes browsing history, viewing duration, dragging behavior, etc.

[0039] In one possible embodiment, the specific implementation process of modifying browsing history data and static features based on the Hierarchical Reinforcement Learning (HRL) algorithm is as follows: The process of correcting browsing history data and static features is modeled as a hierarchical Markov decision process, where the state space of the hierarchical Markov decision process includes video resources and user history behavior data. The high-level agent adopts the Actor-Critic policy gradient method. The Actor network outputs the action to be selected based on browsing history data, and the Critic network evaluates the value of the state and provides an advantage function for policy updates. The formula for the Actor-Critic policy gradient method is as follows (1): (1) ； in, For network parameters, For the policy probability, The dominant function; The low-level agent uses the DoubleDQN algorithm to assign feature weights to the key historical records. The main Q network and the target Q network are maintained and updated periodically. The corrected user profile vector is calculated through weighted aggregation and feature fusion, specifically expressed by the following formula (2): (2); in, The number of historical records to be retained. Normalized weights assigned to lower-level agents. For the embedding vector of key historical records, It is a static feature. This is the feature fusion matrix.

[0040] For example, in the user profile revision process based on a hierarchical reinforcement learning algorithm, the user's browsing history sequence and static features are acquired, and the user profile revision is modeled as a hierarchical Markov decision process. The high-level agent adopts an Actor-Critic architecture, outputting filtering decisions on historical records through a policy network, determining which records should be retained and which should be discarded. The Critic network evaluates the state value and provides an advantage function for policy updates. The low-level agent uses a Double DQN algorithm to assign feature weights to the historical records retained by the high-level agent, learning the importance of each historical record through a Q-network. The two layers of agents work collaboratively to effectively eliminate the interference of noisy historical records and generate a more accurate user profile vector.

[0041] For example, Figure 2 This is a schematic diagram of the hierarchical reinforcement learning framework provided in the embodiments of this application; from a high-level strategy In the process, the original interaction sequence is received. It outputs control signals for lower-level strategies (marked with 0 / 1 in the diagram, possibly representing "whether to switch subtasks" or "which subtask to start"); and makes decisions at the lower-level task level. In this process, based on instructions from higher levels, several lower-level actions are executed sequentially to generate a "corrected user interaction sequence," that is... : , , Simultaneously, the higher-level policy updates the lower-level policy through "reward R + internal reward G," and the lower-level policy updates its selection of lower-level actions through a reward mechanism. In the KGAN module, it can also be combined with generating corrected sequences and predicting the probability P of the target knowledge point P = (y = 1| , ).

[0042] Specifically, the user profile construction module adopts a hierarchical reinforcement learning framework, comprising two layers: a high-level agent and a low-level agent. These two layers work collaboratively, continuously optimizing the strategy through environmental interaction. The high-level agent is responsible for filtering key historical records, employing an Actor-Critic architecture. Its task is to identify and retain the most valuable records for the current recommendation decision from the user's entire browsing history, filtering out noisy and irrelevant history. The Actor network outputs a selection action for each historical record, deciding whether to retain it. The Critic network evaluates the value of the current state, providing an advantage function for policy updates. The low-level agent is responsible for assigning feature weights, employing a Double DQN architecture. After the high-level agent filters out key historical records, the low-level agent assigns importance weights to each retained historical record, determining its contribution to the user profile vector. Through Q-learning, the low-level agent learns the optimal weight allocation strategy, enabling the user profile vector to more accurately reflect the user's true preferences. The user profile vector correction process is modeled as a hierarchical Markov decision process, with the state space including the user's current browsing history sequence, candidate videos, and user static features. Specific formulas, such as the policy gradient formula and the weighted aggregation formula, can be found in the above embodiments and will not be repeated here.

[0043] It is understood that the embodiments of this application obtain user profile vectors through a hierarchical reinforcement learning algorithm. A high-level agent performs coarse-grained historical record filtering, while a low-level agent performs fine-grained feature weight optimization, achieving multi-layered noise filtering. Compared to traditional attention mechanisms, the embodiments of this application can effectively eliminate noise interference when the noise ratio is high.

[0044] Step 102: Based on video resources and the knowledge points associated with video resources, construct the target knowledge graph, and at the same time generate knowledge graph embedding vectors based on graph neural networks.

[0045] The target knowledge graph is used to represent the heterogeneous graph structure of video resource nodes, knowledge point nodes, and relation edges. The knowledge graph embedding vector includes video embedding vector and knowledge point embedding vector.

[0046] For example, the embedding vector can be the attribute features, structural features, and relational features of a node.

[0047] For example, video resource data may include video ID, title, duration, author, and source; knowledge point data may include knowledge point ID, name, description, associated video ID, start time, and end time. By establishing a target knowledge graph G, ,in Includes a set of video nodes and knowledge point node set , Let be the set of edges. For a set of relation types, For example, Figure 3 This is a partial knowledge graph structure diagram provided in the embodiments of this application; wherein, each knowledge point node is connected by relation edges to realize the correlation between knowledge point nodes.

[0048] Specifically, the target knowledge graph can include a heterogeneous graph structure of video resource nodes, knowledge point nodes, and various relation edges. Relationship types include "advanced relations" (a deeper level of explanation of a knowledge point), "predecessor relations" (prerequisite relations for a knowledge point), "successor relations" (subsequent relations for a knowledge point), and "related relations" (semantic relevance of knowledge points).

[0049] For example, a relation-aware graph convolutional network (R-GCN) is applied to perform multi-layer embedding learning on heterogeneous graphs. By aggregating neighborhood information under different relation types, video embedding vectors and knowledge point embedding vectors that fuse structural and semantic information are obtained. The graph convolutional network module maps each knowledge point node to a low-dimensional dense vector space, making semantically similar nodes closer together in the vector space.

[0050] Step 103: Based on the user profile vector and the knowledge graph embedding vector, personalized recommendation results for the user are generated through a knowledge grouping and aggregation mechanism and an attention mechanism. When the user selects a learning target, an adaptive learning path for the user is generated based on the user profile vector and the target knowledge graph.

[0051] In one possible embodiment, based on user profile vectors and knowledge graph embedding vectors, personalized recommendation results for users are generated through a knowledge grouping and aggregation mechanism and an attention mechanism. This includes: inputting the corrected user profile vectors and the knowledge graph embedding vectors of video resources into a Knowledge Grouping and Aggregation Network (KGAN); and using the knowledge grouping and aggregation mechanism in the KGAN recommendation network to predict the user's click probability on candidate videos, so as to output Top-K personalized recommendation results.

[0052] In one possible embodiment, the knowledge grouping and aggregation mechanism of the KGAN recommender network is as follows: The knowledge points associated with video resources are grouped according to their relationship type to obtain knowledge groups; Based on the self-attention enhancement intra-group attention mechanism, the attention weights of the knowledge points in each knowledge group are calculated using a multilayer perceptron, as shown in the following formula (3): (3); in, For user profile vectors, Embed the knowledge points into vectors after self-attention enhancement. It is a multilayer perceptron. This represents vector concatenation. For relation type The corresponding knowledge group; Simultaneously, an intra-group aggregation vector is generated, which is obtained through the following formula (4): (4); A cross-relationship fusion inter-group attention mechanism is applied to all knowledge groups. By integrating user profile vectors, video embedding vectors, and knowledge point embedding vectors, the inter-group attention weights are calculated to obtain the final knowledge aggregation vector. Through multi-layer interactive network computation, the user profile vector, knowledge aggregation vector, and video embedding vector are concatenated and then passed through a fully connected layer and... The activation function outputs personalized recommendation results.

[0053] Specifically, user profiles are input into the KGAN recommendation network module. This module fully utilizes various relationship types in the knowledge graph and calculates accurate recommendation probabilities through a hierarchical attention mechanism. The KGAN recommendation module receives user profiles and candidate videos as input and calculates recommendation probabilities through a knowledge grouping and aggregation mechanism. The module contains four main components: a knowledge grouper, responsible for extracting all knowledge points associated with candidate videos from the knowledge graph and grouping them according to relationship type. Relationship types include predecessor, successor, related, and advanced relationships. Each relationship type corresponds to a knowledge group, and knowledge points within the same group share the same relational semantics. The intra-group attention layer applies an attention mechanism to the knowledge points within each knowledge group. This layer first uses a self-attention mechanism to enhance the knowledge point representation, then combines the user profile to calculate the importance weight of each knowledge point, generating an intra-group aggregation vector. Intra-group attention focuses on the relative importance of knowledge points under the same type of relationship. The inter-group attention layer applies a cross-relationship fusion attention mechanism to all knowledge groups. This layer comprehensively considers the user profile, knowledge group vector, and candidate video embeddings, calculates the importance weights of different relationship types, and generates the final knowledge aggregation vector. Inter-group attention balance considers the contribution of different relationship types to recommendation decisions. The prediction layer fuses user profiles, knowledge aggregation vectors, and candidate video embeddings, and outputs the probability of a user clicking on a candidate video through a multi-layer fully connected network and a non-linear activation function.

[0054] The core calculation of knowledge grouping and aggregation includes two steps: intra-group aggregation and inter-group aggregation. The specific formula for intra-group aggregation can be found in the above embodiment, and will not be repeated here.

[0055] In knowledge component aggregation, an inter-group attention mechanism is applied to all knowledge groups, and the final knowledge aggregation vector is a weighted sum of all group vectors. The recommendation probability is calculated through a multi-layer interaction network. The user profile, knowledge aggregation vector, and video embedding are concatenated and input into a fully connected layer, then processed by ReLU activation and a sigmoid function before being output.

[0056] Specifically, the workflow of the KGAN recommendation module is as follows: Given a candidate video resource, query the knowledge graph to find all knowledge points contained in the video, as well as related knowledge points connected by various relationships. Divide the knowledge points into multiple knowledge groups according to relationship type. For example, a predecessor knowledge group contains all predecessor knowledge points, and a successor knowledge group contains all successor knowledge points. Within each knowledge group, apply a multi-head self-attention mechanism to capture the inherent connections between knowledge points and generate enhanced knowledge point representations. Combine user profiles to calculate attention weights for knowledge points within each knowledge group. Attention weights reflect the relevance of the knowledge point to the user. Calculate the intra-group aggregation vector. Combine user profiles, knowledge group vectors, and candidate video embeddings to calculate the importance weights for different relationship types. Different users pay different amounts of attention to different relationship types; inter-group attention enables personalized relationship type weighting. Sum all knowledge group vectors according to the inter-group attention weights to obtain the final knowledge aggregation vector. This vector integrates knowledge information from multiple relationship types. Concatenate the user profile, knowledge aggregation vector, and candidate video embeddings, and input them into a multi-layer fully connected network. The network contains two hidden layers and uses the ReLU activation function. Finally, the sigmoid function is used to output the recommendation probability, with a value between 0 and 1, representing the likelihood of a user clicking on the video. The recommendation probability is calculated for all candidate videos, and they are sorted from highest to lowest probability. The K videos with the highest probabilities are then selected as the recommendation results and presented to the user.

[0057] Figure 4 This is a schematic diagram of the player and personalized recommendation interaction interface provided in the embodiments of this application. For example, the personalized recommendation interaction interface can record the knowledge point information of the currently playing video, and clicking on the tag can jump to the corresponding time point; at the same time, there is a playlist, which includes a variety of knowledge points; the browsing history of the user can be recorded in the browsing history; in addition, the resource recommendation can display the cold start recommendation results generated by the large model by default, and clicking to generate personalized recommendations will recommend the courses that the user needs to learn most in the current state through reinforcement learning and combined with the user's browsing history.

[0058] Understandably, knowledge grouping and aggregation networks fully leverage the diverse relationship types within knowledge graphs. By grouping knowledge points according to relationship types and applying a hierarchical attention mechanism, knowledge connections can be captured with greater granularity. Intra-group attention focuses on the relative importance of knowledge points within the same type of relationship, while inter-group attention balances the contributions of different relationship types, thereby enhancing the model's ability to understand the semantics of knowledge.

[0059] In one possible embodiment, when a user selects a learning objective, an adaptive learning path is generated based on the user profile vector and the target knowledge graph. This includes: when a user selects a learning objective, obtaining the knowledge points corresponding to the selected learning objective; extracting all sets of predecessor knowledge points and the set of target reachable knowledge points from the target knowledge graph, and calculating the user's comprehensive mastery of the knowledge points, integrating viewing time, number of repetitions, and test scores, specifically expressed by the following formula (5): (5); in, , , These are the weighting coefficients. To accumulate viewing time, For the number of times to watch repeatedly, Scores for knowledge point tests; Obtain the knowledge point difficulty function; where the knowledge point difficulty function includes the number of prerequisites, knowledge complexity, and average learner performance; The Dijkstra algorithm is used to calculate the optimal path from the user's current mastery level to the learning goal, and corresponding video resources are matched for each knowledge point in the optimal path to form an adaptive learning path. The state value function in Dijkstra's algorithm is expressed by the following formula (6): (6); in, To determine the degree threshold, As a discount factor, The difficulty function of the knowledge points. Penalty for path deviation Indicates the direct successor knowledge point of a knowledge point. The state value function.

[0060] For example, based on the learning objectives selected by the user, relevant subgraphs are extracted from the knowledge graph, the user's mastery level and knowledge difficulty are calculated, the optimal path is calculated using a dynamic programming algorithm, and the optimal video is matched for each knowledge point.

[0061] Retrieve all video records from the video database. Each video record contains the following fields: video_id: unique video identifier, title: video title, duration: video duration (seconds), author: video author, source: video source, created_at: creation time, description: video description. Next, retrieve all knowledge point records from the knowledge point database. Each knowledge point record contains: knowledge_id: unique knowledge point identifier, video_id: video ID, name: knowledge point name, description: detailed knowledge point description, start_sec: knowledge point start time (seconds), end_sec: knowledge point end time (seconds), difficulty: difficulty level (1-5). Then, retrieve browsing history from the user behavior table. Each history record contains: user_id: user ID, video_id: viewed video ID, start_time: start viewing time, end_time: end viewing time, duration: viewing duration, rating: user rating, completed: whether the video was watched completely. Finally, data preprocessing is performed, including missing value handling: filling missing description fields with default values ​​and deleting records with missing key fields; outlier handling: filtering invalid records with viewing time less than 5 seconds and handling outlier records exceeding the total video duration; data standardization: standardizing numerical features such as viewing time and rating to the [0,1] range; and text preprocessing: performing word segmentation, stop word removal, and stemming on video titles, descriptions, and knowledge point descriptions. After the user selects a learning target knowledge point, relevant subgraphs are extracted from the knowledge graph. Breadth-first search (BFS) is used to extract all reachable predecessor knowledge points. Starting from the target knowledge point, the process traverses backward along the predecessor relationships, with a maximum of 6 hops, to obtain the set of all predecessor knowledge points. Simultaneously, successor knowledge points are extracted for path expansion, with a maximum of 3 hops. The subgraph node set contains the predecessor set, the target knowledge point, and the successor set, and the edge set contains all relational edges between these nodes. For each knowledge point in the subgraph, the user's overall mastery is calculated, fusing two dimensions with weight coefficients of 0.6 and 0.4, respectively. Duration Mastery: The cumulative viewing time of the knowledge points divided by the total viewing time of the knowledge points, with the minimum value being 1.0. Repeatability Mastery: The number of times the user has viewed the knowledge points repeatedly divided by the maximum number of repetitions (set to 3 times), with the minimum value being 1.0.

[0062] A knowledge point difficulty function is defined, combining three factors with weighting coefficients of 0.3, 0.4, and 0.3 respectively. Predecessor difficulty: Normalized by dividing the number of predecessors of the knowledge point by the maximum number of predecessors in the system. Complexity: The complexity of the knowledge (levels 1 to 5) labeled by experts divided by 5. Average difficulty: 1 minus the average mastery of all learners on the knowledge point. The lower the average mastery, the more difficult the knowledge point. The knowledge point difficulty is a weighted average of the three factors. An improved Dijkstra algorithm is used to calculate the optimal learning path. A state value function is defined to represent the minimum learning cost from the knowledge point to the goal. If the user's mastery of the knowledge point has reached the threshold (0.8), the cost is 0; otherwise, the cost equals the knowledge point difficulty plus a discount factor (0.9) multiplied by the minimum value of the successor node, plus a path deviation penalty. The path deviation penalty equals the penalty coefficient (0.5) multiplied by the shortest distance from the knowledge point to the main path.

[0063] In Dijkstra's algorithm, all node values ​​are initialized to infinity, and the target node's value is 0. Dynamic programming is then performed backward from the target node, calculating the value of each node in reverse topological order. The starting point is determined as the knowledge point with the minimum value that the user has not yet mastered. Starting from the starting point, the node that minimizes the value of its successor is selected at each step, and the optimal path is obtained by backtracking. The optimal video resource is matched for each knowledge point in the path. A matching score function is defined, combining four factors with weights of 0.3, 0.25, 0.25, and 0.2. The first term is the recommendation probability predicted by the KGAN recommendation network. The second term is 1 minus the user's mastery level, encouraging recommendations of videos for knowledge points the user has not yet mastered. The third term is the video quality score, calculated based on the user's average score (weight 0.6) and average completion rate (weight 0.4). The fourth term is the relevance between the video and the knowledge point, based on the proportion of the knowledge point's duration in the video. The video with the highest score is selected as the recommended video for that knowledge point. The final generated learning path includes a sequence of knowledge points and corresponding video resources. It also calculates metadata such as estimated learning time (the sum of all video durations), average difficulty (the average difficulty of all knowledge points in the path), and estimated completion time (total duration divided by the user's daily learning time multiplied by 1 and then subtracting the dropout rate).

[0064] Figure 5 This is a schematic diagram of the learning path recommendation interactive interface provided in the embodiments of this application; for example, in the learning path recommendation, it can support users to select learning goals, and then provide filtered learning paths that match the current user's selected learning goals, and clicking on them can jump to the corresponding knowledge points.

[0065] It is understood that the method provided in this application embodiment is based on personalized learning path planning using knowledge graphs, which differs from traditional fixed-path methods. This method dynamically calculates the optimal learning path based on the user's individual level of mastery and learning goals. By comprehensively considering knowledge difficulty, user mastery, and knowledge dependencies, the generated learning path is more in line with the user's cognitive patterns.

[0066] like Figure 6 As shown, Figure 6 This diagram illustrates the overall system architecture provided in this application embodiment. It shows the workflow of a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs, and distinguishes between new and old users.

[0067] The left side describes the "Knowledge Point Related Recommendation" process for new users. When a user is watching the lesson "k-order infinitesimals," the system recommends "Comparison of Infinitesimals," citing "a detailed explanation of the definition and solution steps of k-order infinitesimals, directly related to the current lesson." Therefore, the recommended content is directly related to the user's current learning of "k-order infinitesimals," thus helping to understand and expand their knowledge. Next, the system queries the database for related knowledge points. Based on the concept of "k-order infinitesimals," the system searches the database for other related knowledge points (such as the "Comparison of Infinitesimals" recommended later, and other potentially related content). Finally, the system finds the relevant knowledge points and converts the corresponding video resources into links that can be directly accessed, pushing them to the user.

[0068] The middle section consists of a "layered reinforcement learning + knowledge aggregation network" for existing users. User profiles are built using browsing history, then optimized through layered reinforcement learning, and finally, the knowledge aggregation network generates recommended video links. Specifically, in layered reinforcement learning, user viewing records include "monotonicity of functions, periodicity of functions, L'Hôpital's rule, and parity of functions." The corrected user viewing records include "monotonicity of functions, periodicity of functions, and parity of functions," because "L'Hôpital's rule is not highly relevant to other knowledge points and may be browsing history generated by accidental clicks." This corrected user profile is then input into the knowledge aggregation network, ultimately generating links to recommended videos.

[0069] The right side of the diagram shows the "goal-oriented learning path planning" process for existing users. When an existing user actively sets a learning goal (e.g., sequences and series), the system plans a suitable learning path around that goal. The user sets the learning goal to "sequences and series," and then, by querying the knowledge graph, a series of learning paths for prerequisite knowledge points leading to the target knowledge point are generated using graph algorithms. Each learning path is evaluated, and the highest-scoring paths are output. Then, based on user profiles, a filtering process is performed: if the user has already learned the definition of a sequence and has a mastery rate of over 80%, that knowledge point is removed from the learning path. Finally, the learning path is output to help the user efficiently learn "sequences and series."

[0070] This application proposes a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs. First, video data and user profile vectors are acquired. The video data includes video resources and related knowledge points. Then, a target knowledge graph is constructed based on the video resources and related knowledge points, and a knowledge graph embedding vector is generated using a graph neural network. Next, based on the user profile vector and the knowledge graph embedding vector, a personalized recommendation result is generated for the user through a knowledge grouping and aggregation mechanism and an attention mechanism. Finally, when the user selects a learning objective, an adaptive learning path is generated for the user based on the user profile vector and the target knowledge graph. Compared to traditional fixed-path methods, this application can accurately generate personalized recommendation results and adaptive learning paths for users based on their own circumstances, video resources, and related knowledge points, thereby reducing historical noise interference in traditional recommendation systems and improving the accuracy of knowledge relevance and path planning.

[0071] In one possible embodiment, the method provided in this application further includes: collecting user feedback data on personalized recommendation results and adaptive learning paths, using the feedback data as a reward signal for reinforcement learning; storing the interaction experience corresponding to the feedback data using a priority experience replay mechanism; and updating the loss function parameters of the agent policy network and recommendation network of the hierarchical reinforcement learning algorithm based on the reward signal and interaction experience.

[0072] For example, feedback data can include multi-dimensional user interactions with recommended videos, including clicks, viewing duration, dragging behavior, and completion rate, converting these interactions into reward signals. A comprehensive reward function integrates multi-dimensional feedback and incorporates a time delay penalty. A priority experience replay mechanism stores interaction data, calculating the priority of each experience based on temporal difference error, with higher-priority experiences having a higher probability of being sampled. Importance sampling weights are used to correct biases. The high-level agent policy network is updated using a policy gradient method and incorporating an entropy regularization term to encourage exploration. The low-level agent Q-network is updated using a Double DQN loss function and importance sampling weights. For the KGAN recommendation network, a weighted cross-entropy loss is used, assigning higher weights to difficult samples.

[0073] In one possible embodiment, the method provided in this application further includes: processing each video resource and the knowledge points associated with each video resource using a large language model, and matching the video resource with the highest recommendation priority for each knowledge point; detecting whether the user is a new user; if the user is a new user, providing the new user with the video resource with the highest recommendation priority for each knowledge point.

[0074] For example, for new users, we pre-process each video and its contained knowledge points using a large model, generating several highly relevant recommendation results for each knowledge point. Even if a user is using the system for the first time, they can still get relatively accurate recommendation results.

[0075] It is understood that the embodiments of this application propose auxiliary mechanisms such as cold start strategy, knowledge relationship mining, multi-objective optimization, and real-time recommendation adjustment to enhance the practicality and robustness of the system. The cold start strategy solves the recommendation problem for new users, knowledge relationship mining can automatically discover implicit knowledge associations, and multi-objective optimization balances multiple performance indicators. These mechanisms enable the system to cope with various real-world scenarios and provide better services.

[0076] like Figure 7 Show, Figure 7This application provides a schematic diagram of the cold start module process. The original video is an example of a math textbook video shown on the left. Text is extracted from the original video, i.e., text information is extracted from the video. The video may contain subtitles, audio-visual explanations converted to text, or the video's own text descriptions. This transforms the unstructured (or semi-structured) audiovisual information in the video into structured text data, facilitating subsequent processing. For example, a JSON-formatted text fragment represents the text content extracted from the video (which may include timestamps, knowledge point names, descriptions, etc.). Then, the extracted text is input into a Large Language Model (LLM). LLM's natural language understanding and knowledge reasoning capabilities are used to perform deep analysis and structuring of the text. From the LLM-processed results, representative sentences / fragments are selected or generated as input examples for subsequent text embedding. The vectors generated by text embedding are used to calculate the similarity between different knowledge points (or their corresponding text fragments). Similarity calculation, a common operation in vector spaces (such as cosine similarity), is used to measure the correlation between two vectors. Based on the similarity level, other related knowledge points are matched for each knowledge point. For example, in a scenario with video examples related to a math textbook, "definition of the derivative" and "geometric meaning of the derivative" might be highly similar knowledge points and would be matched together. This helps users establish knowledge connections, realize knowledge graph construction in the cold start phase, or achieve personalized recommendations and other applications.

[0077] This application proposes a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs. It dynamically corrects user profiles through hierarchical reinforcement learning to eliminate historical noise interference, and combines this with a knowledge graph aggregation network for accurate recommendations, fully utilizing knowledge associations. Furthermore, based on the predecessor-successor relationships of knowledge, the user's own learning progress, and the difficulty of the knowledge points, a graph search algorithm is used to generate adaptive learning paths, providing a systematic learning solution. To verify the effectiveness of the method proposed in this application, extensive experiments were conducted on the real-world MOOC dataset, as detailed below.

[0078] The experiment selected several classic recommendation models as benchmarks, including Bayesian Personalized Ranking (BPR), Multilayer Perceptron (MLP), Factorization Machine (FM), Factorization Item Similarity Model (FISM), Gated Recurrent Unit (GRU), Neural Attention Item Similarity (NAIS), Neural Attention Sequence Recommendation (NASR), and Knowledge Graph Aggregation Network (KGAN). The evaluation metrics used were hit rate (HR) and Normalized Diminished Cumulative Gain (NDCG), calculating the Top-5 and Top-10 results, respectively. Experimental results show that the model incorporating Hierarchical Reinforcement Learning (HRL) significantly outperformed the benchmark models on all metrics. Specifically, HRL+KGAN performed best in hit rate, achieving 66.32% for HR@5 and 80.18% for HR@10, both the highest values ​​among all models. HRL+NASR performed best in ranking quality, achieving 47.51% for NDCG@5 and 52.73% for NDCG@10, significantly outperforming other models. Compared to the baseline model, HRL+NAIS achieves improvements of 14.5% and 15.4% on HR@5 and HR@10, respectively, and 4.6% and 6.0% on NDCG@5 and NDCG@10, respectively. HRL+NASR achieves improvements of 8.1% and 7.2% on HR@5 and HR@10, respectively, and 12.1% and 11.4% on NDCG@5 and NDCG@10, respectively, compared to the original NASR model. The most significant improvement comes from HRL+KGAN, which achieves improvements of 32.0% and 27.0% on HR@5 and HR@10, respectively, and 41.0% and 32.8% on NDCG@5 and NDCG@10, respectively, compared to the KGAN model alone, validating the strong synergistic effect of hierarchical reinforcement learning and knowledge graph aggregation.

[0079] In-depth analysis reveals that the performance of KGAN alone is relatively low among benchmark models, with HR@10 at only 63.12% and NDCG@10 at 38.89%. This is because knowledge graph aggregation struggles to fully realize its potential without accurate user profiling. However, when KGAN is combined with hierarchical reinforcement learning (HRL), performance significantly improves, surpassing other model combinations across all hit rate metrics, reaching 80.18% for HR@10, making it the best performing model overall. While HRL+NASR is slightly lower than HRL+KGAN in hit rate, it performs best in the ranking quality metric NDCG, reaching 52.73% for NDCG@10, a 10.3% improvement over the best benchmark model, NAIS, indicating a significant advantage in optimizing recommendation result ranking. Overall, the three models integrating hierarchical reinforcement learning—HRL+NAIS, HRL+NASR, and HRL+KGAN—achieved HR@10 accuracy of 79.68%, 74.50%, and 80.18%, respectively, representing an average improvement of 12.3% compared to the best baseline model of 69.48%. On the NDCG@10 metric, they reached 50.69%, 52.73%, and 51.64%, respectively, representing an average improvement of 7.8% compared to the best baseline model of 47.82%. The experimental results fully demonstrate that the hierarchical reinforcement learning user profile correction method proposed in this application can effectively eliminate historical noise interference, significantly improve recommendation accuracy, and achieve performance improvements when combined with different recommendation networks. The greatest improvement is observed when combined with the knowledge graph aggregation network KGAN, validating the effectiveness and universality of the method.

[0080] This application constructs a complete personalized video learning recommendation system by integrating hierarchical reinforcement learning, knowledge graph embedding, and adaptive path planning. It has achieved significant advantages in multiple evaluation metrics and provides an effective technical solution for online education.

[0081] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0082] Another embodiment of this application proposes a personalized video learning recommendation device that integrates hierarchical reinforcement learning and knowledge graphs. The details of this personalized video learning recommendation device are described below. The following content is for ease of understanding and is not essential for implementing this example. Figure 8This is a schematic diagram of the structure of a personalized video learning recommendation device that integrates hierarchical reinforcement learning and knowledge graph proposed in this embodiment, including: The acquisition module 810 is used to acquire video data and user profile vectors; the video data includes video resources and knowledge points associated with the video resources. The construction module 820 is used to construct a target knowledge graph based on video resources and knowledge points associated with video resources, and at the same time generate knowledge graph embedding vectors based on graph neural networks; wherein, the target knowledge graph is used to represent the heterogeneous graph structure of video resource nodes, knowledge point nodes and relation edges, and the knowledge graph embedding vectors include video embedding vectors and knowledge point embedding vectors; The generation module 830 is used to generate personalized recommendation results for users based on user profile vectors and knowledge graph embedding vectors, through knowledge grouping and aggregation mechanisms and attention mechanisms. When a user selects a learning target, it generates an adaptive learning path for the user based on user profile vectors and target knowledge graphs.

[0083] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0084] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0085] Another embodiment of this application provides an electronic device, such as Figure 9 As shown, it includes a processor 91 and a memory 92. The memory 92 stores instructions that the processor 91 can execute. When the processor 91 is configured to execute the instructions, the electronic device can implement a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph as described in the above method embodiment.

[0086] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0087] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0088] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graphs as described in the above method embodiments.

[0089] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0090] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A personalized video learning recommendation method integrating hierarchical reinforcement learning and knowledge graphs, characterized in that, The method includes: Acquire video data and user profile vectors; the video data includes video resources and related knowledge points. Based on video resources and the knowledge points associated with them, a target knowledge graph is constructed, and knowledge graph embedding vectors are generated based on graph neural networks. The target knowledge graph is used to represent the heterogeneous graph structure of video resource nodes, knowledge point nodes, and relation edges, and the knowledge graph embedding vectors include video embedding vectors and knowledge point embedding vectors. Based on user profile vectors and knowledge graph embedding vectors, personalized recommendation results are generated for users through knowledge grouping and aggregation mechanisms and attention mechanisms. Furthermore, when a user selects a learning target, an adaptive learning path is generated for the user based on the user profile vector and the target knowledge graph.

2. The method according to claim 1, characterized in that, The acquisition of video data and user profile vectors includes: Acquire user historical behavior data and target database; where user historical behavior data includes user browsing history data and user static characteristics, user static characteristics include user personal information, and the target database is used to represent the mapping relationship between video resources and knowledge point data; Based on the mapping relationship between video resources and knowledge point data, obtain video resources and knowledge point data associated with video resources; Based on the hierarchical reinforcement learning algorithm, browsing history data and static features are corrected to obtain the corrected user profile vector; In the hierarchical reinforcement algorithm, the high-level agent is used to filter key historical records in user historical behavior data based on the policy gradient method, while the low-level agent is used to assign feature weights to key historical records based on the Q-learning algorithm to eliminate the interference of noisy history.

3. The method according to claim 2, characterized in that, The specific implementation process of correcting browsing history data and static features based on the hierarchical reinforcement learning algorithm is as follows: The process of correcting browsing history data and static features is modeled as a hierarchical Markov decision process, where the state space of the hierarchical Markov decision process includes video resources and user history behavior data. The high-level agent adopts the Actor-Critic policy gradient method. The Actor network outputs the action to be selected based on browsing history data, and the Critic network evaluates the value of the state and provides an advantage function for policy updates. The formula for the Actor-Critic policy gradient method is as follows (1): （1）； in, For network parameters, For the policy probability, The dominant function; The low-level agent uses the DoubleDQN algorithm to assign feature weights to the key historical records. The main Q network and the target Q network are maintained and updated periodically. The corrected user profile vector is calculated through weighted aggregation and feature fusion, specifically expressed by the following formula (2): （2）； in, The number of historical records to be retained. Normalized weights assigned to lower-level agents. For the embedding vector of key historical records, It is a static feature. This is the feature fusion matrix.

4. The method according to claim 3, characterized in that, The process of generating personalized recommendation results for users based on user profile vectors and knowledge graph embedding vectors, through knowledge grouping and aggregation mechanisms and attention mechanisms, includes: The corrected user profile vectors and knowledge graph embeddings of video resources are input into the knowledge grouping aggregation recommendation network. By utilizing the knowledge grouping and aggregation mechanism in the knowledge grouping and aggregation recommendation network, the probability of a user clicking on candidate videos is predicted, so as to output Top-K personalized recommendation results.

5. The method according to claim 4, characterized in that, The knowledge grouping and aggregation mechanism of the knowledge grouping and aggregation recommendation network is as follows: The knowledge points associated with video resources are grouped according to their relationship type to obtain knowledge groups; Based on the self-attention enhancement intra-group attention mechanism, the attention weights of the knowledge points in each knowledge group are calculated using a multilayer perceptron, as shown in the following formula (3): （3）； in, For user profile vectors, Embed the knowledge points into vectors after self-attention enhancement. It is a multilayer perceptron. This represents vector concatenation. For relation type The corresponding knowledge group; Simultaneously, an intra-group aggregation vector is generated, which is obtained through the following formula (4): （4）； A cross-relationship fusion inter-group attention mechanism is applied to all knowledge groups. By integrating user profile vectors, video embedding vectors, and knowledge point embedding vectors, the inter-group attention weights are calculated to obtain the final knowledge aggregation vector. Through multi-layer interactive network computation, the user profile vector, knowledge aggregation vector, and video embedding vector are concatenated and then passed through a fully connected layer and... The activation function outputs personalized recommendation results.

6. The method according to claim 5, characterized in that, When a user selects a learning objective, an adaptive learning path is generated based on the user profile vector and the target knowledge graph, including: When a user selects a learning goal, retrieve the knowledge points corresponding to the selected learning goal; Extract all predecessor knowledge point sets and target reachable knowledge point sets from the target knowledge graph, and calculate the user's comprehensive mastery of the knowledge points, integrating viewing time, repetition count and test scores, specifically expressed by the following formula (5): （5）； in, , , These are the weighting coefficients. To accumulate viewing time, For the number of times to watch repeatedly, Scores for knowledge point tests; Obtain the knowledge point difficulty function; where the knowledge point difficulty function includes the number of prerequisites, knowledge complexity, and average learner performance; The Dijkstra algorithm is used to calculate the optimal path from the user's current mastery level to the learning goal, and corresponding video resources are matched for each knowledge point in the optimal path to form an adaptive learning path. The state value function in Dijkstra's algorithm is expressed by the following formula (6): （6）； in, To determine the degree threshold, As a discount factor, The difficulty function of the knowledge points. Penalty for path deviation Indicates the direct successor knowledge point of a knowledge point. The state value function.

7. The method according to claim 6, characterized in that, The method further includes: Collect user feedback data on personalized recommendation results and adaptive learning paths, and use the feedback data as a reward signal for reinforcement learning; A priority-based experience replay mechanism is used to store the interaction experience corresponding to the feedback data. The loss function parameters of the agent policy network and recommendation network of the hierarchical reinforcement learning algorithm are updated based on reward signals and interaction experience.

8. The method according to claim 7, characterized in that, The method further includes: A large language model is used to process each video resource and the knowledge points associated with each video resource, and to match the video resource with the highest priority for each knowledge point. Detect whether the user is a new user; If the user is a new user, provide them with the video resources that are recommended with the highest priority for each knowledge point.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a personalized video learning recommendation method that integrates hierarchical reinforcement learning and knowledge graph as described in any one of claims 1 to 8.

Images