Concept recommendation method and system based on large model in the absence of knowledge graph modality

Abstract

The present invention discloses a method and system for recommending concepts in the absence of modality in a knowledge graph based on a large model, and belongs to the field of knowledge graph recommendation. The present invention first makes full use of the large model to optimize the construction of the knowledge graph, and then constructs a richer knowledge graph by using the feature completion technology of channel confidence to ensure the integrity of concept features, and finally combines the online learner model to capture the learner's knowledge status in real time, thereby realizing personalized recommendation of knowledge concepts in the online learning process. The present invention not only improves the integrity and dynamics of the knowledge graph, but also provides an effective solution to the problem of knowledge recommendation in online education.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge graph recommendation, and specifically relates to a concept recommendation method and system based on a large model in the absence of knowledge graph modality. Background Art

[0002] The continuous development of Internet technology has greatly promoted the popularity of online education, making it an important way for individuals to acquire professional knowledge and skills. However, online education platforms, especially massive online open courses (MOOCs), still face huge challenges in providing knowledge recommendations that meet learners' personalized needs and cognitive levels. With the rapid growth of educational content, traditional content classification and recommendation methods based on manual rules have gradually exposed their limitations and are difficult to dynamically respond to learners' diverse needs and changing learning situations.

[0003] In traditional online education systems, knowledge graphs (KGs) are widely used to build and manage educational content. Knowledge graphs provide support for the retrieval and recommendation of educational content by structured representation of knowledge points and their relationships. However, the construction of knowledge graphs usually relies on a lot of manual intervention, consumes a lot of resources but lags in updates, and is particularly difficult to effectively integrate multimodal information (such as text, images, videos, etc.). At the same time, when dealing with ever-changing learning needs and complex knowledge environments, the separation of entity recognition and entity relationship mining in this static knowledge graph based on artificial rules is more likely to lead to increased unreliability.

[0004] In recent years, the rapid development of large language models (LLMs) has provided new possibilities for the construction and optimization of knowledge graphs. With its powerful natural language processing and generation capabilities, LLMs can automatically extract and represent various knowledge entities and their relationships, thus making up for the shortcomings of traditional knowledge graphs in terms of dynamic and multimodal integration. The introduction of LLMs not only improves the efficiency of knowledge extraction and entity linking, but also reduces the burden of manually constructing knowledge graphs through large-scale automated processing, making knowledge graphs more intelligent and dynamic. This combination enables online education platforms to better realize intelligent content recommendation and personalized guidance, thereby improving learning efficiency and learning experience.

[0005] In order to improve the learner's knowledge mastery level, the strategy of planning concept learning paths needs to be based on two types of key information: 1) The learner's knowledge state, that is, the learner's mastery of the concept. The learner's knowledge state is predicted and evaluated through knowledge tracking models [such as deep knowledge tracking (DKT), attention knowledge tracking (AKT), etc.], which use neural networks to dynamically track the learner's knowledge state, capture the relationship between knowledge points, and make predictions based on historical learning data. However, these methods rely on manually designed meta-paths, and have problems such as labor-intensiveness, cold start, and lack of interpretability. 2) Semantic and structural relationships between concepts. Existing methods introduce the relationship between concepts through knowledge graphs, but often ignore the intrinsic semantics and multimodal knowledge structure of concepts, such as graph-based knowledge tracking (GKT), which fails to fully integrate specific factual knowledge and external knowledge and cannot reveal the deep semantic associations between concepts.

[0006] The introduction of multimodal information provides new possibilities for the construction and optimization of knowledge graphs, but it also brings about the problem of missing multimodal attributes, including the lack of effective associations between multimodal information such as text, images, videos, and audio. This lack of attributes not only affects the comprehensiveness and richness of learning resources, but also leads to the sparseness and incompleteness of knowledge graphs, making it difficult to fully reflect the complexity of the real world. Therefore, there are the following technical problems that need to be solved in the existing technology: 1) How to represent concepts in a way that integrates human knowledge systems and multimodal information to avoid ambiguity of concepts in different learning contexts; 2) How to reduce the impact of missing knowledge graph attributes on concept recommendation results; 3) How to combine offline multimodal learning resources and online learner information for effective knowledge concept recommendation and interpretation. Summary of the invention

[0007] The purpose of the present invention is to solve the above-mentioned problems in the prior art and to provide a concept recommendation method and system based on a large model in the absence of knowledge graph modality.

[0008] The specific technical solutions adopted by the present invention are as follows:

[0009] In a first aspect, the present invention provides a concept recommendation method based on a large model in the absence of a knowledge graph modality, which comprises:

[0010] S1. Based on the historical learning records of each student user on the online learning platform, all the multimodal learning materials that each student user has learned are obtained and multimodally encoded. The GraphGPS encoding mechanism is used to fuse the graph with multimodal learning materials as nodes, and the node features in the graph are updated from the original multimodal features to the fusion features that fuse local and global information.

[0011] S2. Based on the historical learning records of each student user on the online learning platform, online student nodes, multimodal learning material nodes, and main concept nodes contained in the multimodal learning material nodes are constructed in the knowledge graph, and edge connections are established; the large language model is used to generate the explanation text of each main concept, and the encoding features of the main concept explanation text are used as the node features of the main concept node, and the node features of the multimodal learning material node adopt the fusion features, and the student user's understanding status mark value of the main concept is recorded on the main concept node; then, based on the semantic knowledge base, the category set, the pre-concept set, and the post-concept set of each main concept are searched, and based on the similarity of the encoding features, a part of the categories, pre-concepts, and post-concepts most relevant to the main concept are selected from each of the three sets, and expanded into the neighborhood nodes of the corresponding main concept node, thereby obtaining a multimodal knowledge graph;

[0012] S3. For multimodal learning data nodes with missing node features in the multimodal knowledge graph, an offline knowledge graph feature completion method based on channel confidence is used. According to the channel confidence obtained by the channel quality assessment, the node features are first diffused and restored within the channel, and then the node features are propagated and refined between channels. The node features of the missing nodes are completed modally one by one, and the obtained missing node features are integrated and completed into the multimodal knowledge graph.

[0013] S4. During the online learning process of the target student users, the learner model representations of different historical time steps are generated through the online learner model. The learner model representation of each historical time step is obtained by weighting the node feature representations of all the main concepts that are not understood in the multimodal learning materials interacted by the target student users in this historical time step; then the preceding concept node features, the following concept node features, the category node features, the main concept node features, and the learner model representation in each historical time step are sequentially spliced ​​to form the final representation of each historical time step; the time series of the final representation is encoded through the self-attention Transformer model, and then the recommendation scores of all concepts are output through the prediction head to generate the recommendation results.

[0014] As a preferred embodiment of the above-mentioned first aspect, the multimodal learning material includes four modalities: text, image, video and audio, each of which is encoded using a corresponding encoder to constitute a multimodal feature corresponding to the multimodal learning material.

[0015] As a preferred embodiment of the above-mentioned first aspect, in S1, the method of fusing the graph using the GraphGPS encoding mechanism is: taking all the multimodal learning materials on the online learning platform as graph nodes, establishing edge connections for the nodes in the graph that have interacted with the same student user according to the interaction information of the student users for the multimodal learning materials, thereby constructing a learning material graph, and the node features in the learning material graph are the multimodal features of the corresponding multimodal learning materials; using the GraphGPS encoding mechanism to fuse the learning material graph, the node features in the graph are updated from the original multimodal features to fused features that fuse local and global information.

[0016] As a preferred embodiment of the above-mentioned first aspect, in S2, when establishing edge connections between nodes in the multimodal knowledge graph, it is necessary to edge-connect each online student node with each multimodal learning material node that has interacted with it according to the historical learning records of each student user, and to edge-connect each multimodal learning material node with each main concept node contained in it. At the same time, each main concept needs to be edge-connected with each screened category node, each preceding concept node, and each succeeding concept node respectively, and the edge connection relationship between the nodes in the graph is recorded in the adjacency matrix of the multimodal knowledge graph.

[0017] As a preferred embodiment of the above-mentioned first aspect, in the multimodal knowledge graph, each main concept node needs to record the student user's understanding status mark value of this main concept at different time steps according to the correctness of the answers in the historical learning records, where the understanding status mark values ​​are divided into three types: not understood, understood, and not interactively learned yet.

[0018] As a preferred embodiment of the first aspect, in S3, for any modality, using the offline knowledge graph feature completion method based on channel confidence, the specific steps of completing the missing node features of the modality in the multimodal knowledge graph are:

[0019] S31. For the current modality to be completed, all feature channels of the modality to be completed are extracted from the multimodal knowledge graph, and then the shortest path distance between each multimodal learning material node and the nearest source node is calculated for each feature channel, where the nearest source node is a multimodal learning material node that has a feature value in the calculated feature channel and is closest to the calculated multimodal learning material node; for each feature channel respectively, the shortest path distance is converted into a confidence by a power operation with a base range of (0,1);

[0020] S32. For each feature channel, calculate the confidence ratio of any two multimodal learning material nodes in this feature channel, and record them in the weighted adjacency matrix of this feature channel as the relative confidence between the nodes. Perform a row random normalization operation on the weighted adjacency matrix to obtain a normalized weighted adjacency matrix. Use the normalized weighted adjacency matrix to diffuse the current latest multimodal knowledge graph channel by channel nodes, and diffuse the characteristic values ​​of other nodes in each feature channel to the missing multimodal learning material nodes. Iterate the diffusion multiple times to obtain the initial recovery features of all multimodal learning material nodes in the multimodal knowledge graph.

[0021] S34. Based on the initial recovery features, calculate the feature correlation coefficient between any two feature channels, and then for each multimodal learning material node, convert the corresponding feature correlation coefficient to obtain the directed edge weight from one feature channel to another feature channel, and record the directed edge weight between the feature channels in the weight adjacency matrix; use the weight adjacency matrix to perform channel-to-channel propagation on each node in the current latest multimodal knowledge graph, and diffuse the average node features of each channel to the multimodal learning material nodes missing in each feature channel; iterate the diffusion multiple times to obtain the final recovery features of all multimodal learning material nodes in the multimodal knowledge graph;

[0022] S35. For multimodal learning material nodes in the multimodal knowledge graph that have missing node features in the current modality, the corresponding final restored features are used to complete the node features.

[0023] As a preferred embodiment of the above-mentioned first aspect, in S4, when the node feature representations of all the main concepts that are not understood are weighted to obtain the learner model representation, the weight used for each main concept that is not understood is the cosine distance between the node feature representation of this main concept that is not understood and the node feature representation of the multimodal learning material.

[0024] In a second aspect, the present invention provides a concept recommendation system based on a large model in the absence of knowledge graph modality, which includes:

[0025] The learning record collection module is used to collect the historical learning records of each student user on the online learning platform;

[0026] The recommendation object designation module is used to allow the management user to designate the student users who need concept recommendations through the GUI interface;

[0027] The concept recommendation module is used to obtain the historical learning records corresponding to the designated student users from the learning record collection module according to the student users designated in the recommendation object designation module, and then recommend the concepts for the designated student users to learn next according to the concept recommendation method based on the large model in the absence of knowledge graph modality as described in any one of the first aspects above.

[0028] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for recommending concepts in the absence of a knowledge graph modality based on a large model is implemented as described in any one of the schemes of the first aspect above.

[0029] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0030] The memory is used to store computer programs;

[0031] The processor is used to implement the concept recommendation method based on a large model in the absence of knowledge graph modality as described in any one of the solutions of the first aspect above when executing the computer program.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a concept recommendation method for a multimodal knowledge graph under attribute missing based on a large language model (LLMs) and an online learner model, making full use of the natural language processing and generation capabilities of LLMs to optimize the construction and recommendation process of the knowledge graph. The present invention constructs a richer knowledge graph by utilizing the feature completion technology of channel confidence to ensure the integrity of concept features, utilizes LLMs to enhance the ability of knowledge extraction and semantic association, and combines the online learner model to capture the learner's knowledge state in real time, thereby optimizing the recommendation effect. At the same time, the present invention also adopts a recommendation model of a self-attention mechanism to dynamically capture the learner's understanding state and make personalized recommendations based on cosine similarity scores. In addition, the present invention can further introduce a weighted path selection mechanism in practical applications, provide a clear recommendation explanation by evaluating multiple paths between learners and recommended concepts, and enhance the learner's trust in the recommendation system. In summary, the present invention not only improves the integrity and dynamics of the knowledge graph, but also provides an effective solution to the problem of knowledge recommendation in online education. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the steps of the concept recommendation method based on the knowledge graph modality missing based on the large model;

[0035] Figure 2This is a schematic diagram of the module composition of the concept recommendation system based on the large model with missing knowledge graph modality;

[0036] Figure 3 A schematic diagram of the components of computer electronic equipment;

[0037] Figure 4 A flowchart of a specific implementation of the concept recommendation method in an embodiment of the present invention;

[0038] Figure 5 An exemplary multimodal knowledge graph. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0040] In online learning platforms, the core of concept recommendation lies in accurately understanding and utilizing the semantics of concepts and their relationships, which constitute the basic framework of the human knowledge system. Traditional concept recommendation systems usually rely on static definitions of concepts or simple text descriptions, but this approach is prone to ambiguity, especially when the same concept has multiple meanings in different contexts. Therefore, how to reduce ambiguity and ensure the accuracy of concept semantics has become a key issue facing concept recommendation systems.

[0041] This paper proposes a concept recommendation method for multimodal knowledge graphs with missing attributes based on large models (LLMs) and online learner models, which can make full use of the natural language processing and generation capabilities of LLMs to optimize the construction and recommendation process of knowledge graphs. This paper constructs a richer knowledge graph by using the feature completion technology of channel confidence to ensure the integrity of concept features, uses LLMs to enhance the ability of knowledge extraction and semantic association, and combines the online learner model to capture the learner's knowledge status in real time, thereby optimizing the recommendation effect.

[0042] like Figure 1As shown, in a preferred embodiment of the present invention, a method for recommending concepts in the absence of modality in a knowledge graph based on a large model is provided, and its specific steps are shown in S1 to S4, wherein step S1 is used to realize the fusion of multimodal learning material features combined with the GraphGPS encoding mechanism, step S2 is used to realize the construction of a multimodal knowledge graph combined with a large language model, step S3 is used to realize the offline knowledge graph feature completion based on channel confidence, and step S4 is used to realize the recommendation of concepts combined with the learner model representation and historical information. The specific implementation of steps S1 to S4 is described in detail below.

[0043] S1. Based on the historical learning records of each student user on the online learning platform, all the multimodal learning materials that each student user has learned are obtained and multimodally encoded. The GraphGPS encoding mechanism is used to fuse the graph with multimodal learning materials as nodes, and the node features in the graph are updated from the original multimodal features to fused features that fuse local and global information.

[0044] In an embodiment of the present invention, the above-mentioned multimodal learning materials include four modalities: text, image, video and audio, each of which is encoded using a corresponding encoder to form a multimodal feature corresponding to the multimodal learning materials.

[0045] In an embodiment of the present invention, the method of fusing graphs using the GraphGPS encoding mechanism can be implemented according to the following process: using all multimodal learning materials on the online learning platform as graph nodes, establishing edge connections for the nodes in the graph that have interacted with the same student user according to the interaction information of the student users for the multimodal learning materials, thereby constructing a learning material graph, and the node features in the learning material graph are the multimodal features of the corresponding multimodal learning materials; using the GraphGPS encoding mechanism to fuse the learning material graph, the node features in the graph are updated from the original multimodal features to fused features that fuse local and global information.

[0046] S2. Based on the historical learning records of each student user on the online learning platform, online student nodes, multimodal learning material nodes, and main concept nodes contained in the multimodal learning material nodes are constructed in the knowledge graph, and edge connections are established; the large language model is used to generate an explanation text for each main concept, and the encoding features of the main concept explanation text are used as the node features of the main concept nodes, while the node features of the multimodal learning material nodes adopt the fusion features, and the student user's understanding status mark value of the main concept is recorded on the main concept node; then, based on the semantic knowledge base, the category set, the preceding concept set, and the following concept set of each main concept are searched, and based on the similarity of the encoding features, a part of the categories, preceding concepts, and following concepts that are most relevant to the main concept are selected from each of the three sets, and expanded into the neighborhood nodes of the corresponding main concept nodes, thereby obtaining a multimodal knowledge graph.

[0047] It should be noted that the main function of introducing the large language model in the present invention is to provide an explanatory text for each main concept. The encoding features of this explanatory text will be used to find the category to which the concept belongs, the related pre-concepts and the post-concepts from the external semantic knowledge base. In order to ensure the accuracy of the large language model for the concept interpretation, it is recommended to use the education large model fine-tuned on the education data set to ensure that the concept definition matches the specific education background. The semantic knowledge base can be implemented using databases such as DBpedia Spotlight, which itself contains explanatory texts of different concepts. For each main concept, the category to which the concept belongs, the related pre-concepts and post-concepts can be found from the semantic knowledge base, and then the encoding features of the explanatory text generated by the large language model for the main concept (for text, feature encoding can be achieved through language models such as BERT), as well as the encoding features of the explanatory text of each category, pre-concept and post-concept in the semantic knowledge base, calculate the similarity between the two, and then determine the similarity between the main concept and each element in the category set, pre-concept set, and post-concept set. Based on these similarities, it means to screen each set at a specified ratio. The specific screening ratio or similarity threshold can be adjusted according to actual conditions. Based on the similarity, a part of the categories, pre-concepts and post-concepts most relevant to the main concept can be screened from each of the three sets, which can be expanded into the neighborhood nodes of the corresponding main concept nodes and establish edge connections with the main concept nodes. After all the main concepts are expanded, a multimodal knowledge graph can be obtained.

[0048] In an embodiment of the present invention, when establishing edge connections between nodes in a multimodal knowledge graph, it is necessary to establish edge connections between each online student node and each multimodal learning material node that has interacted with it based on the historical learning records of each student user, and to establish edge connections between each multimodal learning material node and each main concept node contained in it. At the same time, each main concept needs to be edge-connected with each filtered category node, each preceding concept node, and each succeeding concept node. The edge connection relationship between the nodes in the graph is recorded in the adjacency matrix of the multimodal knowledge graph.

[0049] In the present invention, the basis of concept recommendation is to build an exclusive offline knowledge graph for multimodal learning materials, integrating the semantics of each concept and their mutual relationships. In the above step S2, a method for automatically creating a multimodal knowledge graph is provided, thereby ensuring that the system can generate more accurate concept interpretations based on a specific educational background. The present invention first constructs a preliminary knowledge graph for the modal information (such as text, image, video, audio) of each learning material, and gradually forms a complete multimodal offline knowledge graph. Next, the main concept is expanded, other related concepts and categories are added, and relationships between concepts are established in the knowledge graph, such as predecessor concepts, successor concepts, etc. Therefore, the constructed multimodal knowledge graph contains multiple nodes, involving multiple entities: learning materials, modal information, main concepts, predecessor concepts, successor concepts, categories, and relationship edges.

[0050] In addition, in an embodiment of the present invention, each main concept node of the multimodal knowledge graph needs to record the student user's understanding status mark value of this main concept at different time steps according to the correctness of the answers in the historical learning records, where the understanding status mark value is divided into three types: not understood, understood, and not interactively learned. Specifically, each concept node is marked as "understood", "not understood", or "not interactively learned" according to the learner's interaction, where "not interactively learned" means that the learner has not yet been exposed to the concept and it is a new concept for the current learner. As learners interact with different modalities, the understanding status mark of the main concepts in the multimodal knowledge graph will also be dynamically updated, providing real-time feedback to optimize the recommendation system.

[0051] Through step S2, the present invention can build a comprehensive and dynamically updated knowledge graph, enhance the accuracy and personalization ability of the recommendation system, and subsequently accurately recommend relevant concepts based on the learner's learning progress and cognitive state, and lay the foundation for providing learners with a more appropriate learning path.

[0052] S3. For multimodal learning material nodes with missing node features in the multimodal knowledge graph, an offline knowledge graph feature completion method based on channel confidence is used. According to the channel confidence obtained by channel quality assessment, the node features are first restored by diffusion within the channel and then refined by propagation between channels. The node features of the missing nodes are completed modally one by one, and the missing node features are integrated and completed into the multimodal knowledge graph.

[0053] On online learning platforms such as MOOCs, the construction and updating of multimodal knowledge graphs often face the problems of incomplete multimodal data and missing information, especially when learners use multiple modalities for learning, the lack of data may significantly affect the effect of knowledge recommendation. In order to solve this challenge, the present invention proposes the above-mentioned offline knowledge graph feature completion method based on channel confidence, which aims to effectively fill the missing features in the knowledge graph and improve its completeness and expression ability. The core of this method is to evaluate the reliability and relevance of each node and edge in the knowledge graph through channel confidence. Data of different modalities (such as text, images, videos, and audio) may differ in quality, and the data quality of some modalities is low, which may affect the accuracy of the overall knowledge representation. By evaluating the contribution of each channel in knowledge dissemination, it is possible to give priority to using higher quality information for feature completion, thereby reducing the negative impact of missing data on the graph.

[0054] Therefore, the above-mentioned offline knowledge graph feature completion method based on channel confidence first evaluates each modal data through the channel confidence model to extract the most representative features. The key process includes the following steps: 1) Channel quality evaluation: Evaluate the confidence of each modality on a specific knowledge point and analyze its contribution and reliability to the knowledge graph. 2) Feature completion strategy: Use channel confidence to determine the completion strategy for missing features: first perform feature diffusion recovery between nodes in the channel, and then refine the features propagated within nodes between channels. 3) Offline feature integration: Integrate the completed features back into the knowledge graph through the graph neural network to enhance the integrity of the nodes and their relationships.

[0055] In an embodiment of the present invention, for any modality, the specific steps of completing the missing node features of the modality in the multimodal knowledge graph using the offline knowledge graph feature completion method based on channel confidence are as follows:

[0056] S31. For the current modality to be completed, all feature channels of the modality to be completed are extracted from the multimodal knowledge graph, and then the shortest path distance between each multimodal learning material node and the nearest source node is calculated for each feature channel, where the nearest source node is a multimodal learning material node that has a feature value in the calculated feature channel and is closest to the calculated multimodal learning material node; for each feature channel respectively, the shortest path distance is converted into a confidence by a power operation with a base range of (0,1);

[0057] S32. For each feature channel, calculate the confidence ratio of any two multimodal learning material nodes in this feature channel, and record them in the weighted adjacency matrix of this feature channel as the relative confidence between the nodes. Perform a row random normalization operation on the weighted adjacency matrix to obtain a normalized weighted adjacency matrix. Use the normalized weighted adjacency matrix to diffuse the current latest multimodal knowledge graph channel by channel nodes, and diffuse the characteristic values ​​of other nodes in each feature channel to the missing multimodal learning material nodes. Iterate the diffusion multiple times to obtain the initial recovery features of all multimodal learning material nodes in the multimodal knowledge graph.

[0058] S34. Based on the initial recovery features, calculate the feature correlation coefficient between any two feature channels, and then for each multimodal learning material node, convert the corresponding feature correlation coefficient to obtain the directed edge weight from one feature channel to another feature channel, and record the directed edge weight between the feature channels in the weight adjacency matrix; use the weight adjacency matrix to perform channel-to-channel propagation on each node in the current latest multimodal knowledge graph, and diffuse the average node features of each channel to the multimodal learning material nodes missing in each feature channel; iterate the diffusion multiple times to obtain the final recovery features of all multimodal learning material nodes in the multimodal knowledge graph;

[0059] S35. For multimodal learning material nodes in the multimodal knowledge graph that have missing node features in the current modality, the corresponding final restored features are used to complete the node features.

[0060] Through the above-mentioned feature completion method based on channel confidence, the present invention can significantly enhance the structural integrity of the knowledge graph, improve the accuracy of knowledge recommendation, and enhance the adaptability of the recommendation system to multimodal information. At the same time, this completion method ensures that learners can obtain more comprehensive information in a multimodal learning environment, providing reliable data support for personalized learning recommendations.

[0061] S4. During the online learning process of the target student users, the learner model representations of different historical time steps are generated through the online learner model. The learner model representation of each historical time step is obtained by weighting the node feature representations of all the main concepts that are not understood in the multimodal learning materials interacted by the target student users in this historical time step; then the preceding concept node features, the following concept node features, the category node features, the main concept node features, and the learner model representation in each historical time step are sequentially spliced ​​to form the final representation of each historical time step; the time series of the final representation is encoded through the self-attention Transformer model, and then the recommendation scores of all concepts are output through the prediction head to generate the recommendation results.

[0062] Since the effectiveness of knowledge concept recommendation depends on a deep understanding of the learner's state and the comprehensiveness of the knowledge graph, in the above step S3 of the present invention, a concept recommendation framework combining a multimodal knowledge graph with offline feature completion and an online learner model is proposed to achieve personalized knowledge concept recommendation and interpretation. Since in the above steps S1 and S2, the present invention has constructed a multimodal knowledge graph with rich feature completion, which contains complete concept features obtained through channel confidence evaluation, and can effectively reflect the relationship between information and concepts in the learning materials. Therefore, in the S3 step, it is also necessary to construct an online learner model, which captures the learner's understanding state when interacting with multimodal learning materials and represents the learner's cognitive level in real time. The online learner model updates the learner's mastery of different concepts by recording the learner's learning behavior (such as clicking, browsing, answering questions, etc.).

[0063] In the concept recommendation process, the present invention determines the weight of the concept by calculating the embedding score between each knowledge concept and the current learning material. Then, the self-attention mechanism is used to dynamically capture the learner's current learning state and identify the most suitable knowledge concept for recommendation. The self-attention mechanism can effectively focus on the correlation between different nodes in the knowledge graph, thereby generating more accurate personalized recommendation results for learners. This process makes full use of the structural and semantic information in the multimodal knowledge graph, making the recommendation more in line with the actual needs of learners.

[0064] In an embodiment of the present invention, the learner model representation is based on the student user's understanding status mark value of this main concept recorded on the multimodal knowledge graph at different time steps to extract all the main concepts that the current student user does not understand, that is, the main concepts whose mark value is "not understood" and have a connection relationship with the student user. When the node feature representations of all the main concepts that are not understood are weighted to obtain the learner model representation, the weight used for each main concept that is not understood can be the cosine distance between the node feature representation of this main concept that is not understood and the node feature representation of the multimodal learning material.

[0065] In addition, in an embodiment of the present invention, in order to enhance the transparency of the recommendation and help learners understand the logic behind the recommendation, a weighted path selection method can also be introduced to explain the recommendation process. By evaluating multiple possible paths between the learner and the recommended concept, the system calculates the weight of each path and selects the path with the highest weight as the recommendation explanation. This means that learners can intuitively see the reasons for recommending a certain concept, such as its relevance to existing knowledge or the learner's current learning status. This transparent explanation not only helps learners understand the recommendation logic, but also enhances their trust in the system.

[0066] It can be seen that the knowledge concept recommendation method of the present invention realizes a personalized and intelligent recommendation process by combining offline and online information. The present invention can not only dynamically adapt to the learner's cognitive state, but also provide clear recommendation explanations through rich multimodal knowledge graphs and online learning data, thereby optimizing the learner's learning path and learning experience.

[0067] Similarly, based on the same inventive concept, Figure 2 As shown, another preferred embodiment of the present invention also provides a concept recommendation system based on a large model in the absence of knowledge graph modality, which includes:

[0068] The learning record collection module is used to collect the historical learning records of each student user on the online learning platform;

[0069] The recommendation object designation module is used to allow the management user to designate the student users who need concept recommendations through the GUI interface;

[0070] The concept recommendation module is used to obtain the historical learning records corresponding to the designated student users from the learning record collection module according to the designated student users in the recommendation object designation module, and then recommend the concepts for the designated student users to learn next according to the concept recommendation method based on the large model in the absence of knowledge graph modality as described in S1 to S4 in the above embodiments.

[0071] It should be noted that the above-mentioned learning record collection module can be built in the background server of the online learning platform to record the historical learning records of student users in real time. The recommendation object designation module can be located on the front-end interface of the online learning platform and provide designated functions through buttons and the like. Of course, the recommended object can also be set by default in the recommended object designation module. If the management user does not modify the default setting, the recommendation is made according to the default setting. Every student on the platform can be a recommended object. The recommendation method in the concept recommendation module can be implemented through a fixed module or display logic in the user interface, or through system messages, pop-up windows, or emails, text messages, and the like.

[0072] It should also be noted that the concept recommendation method based on the large model in the absence of knowledge graph modality described in S1 to S4 of the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, such as Figure 3 As shown, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the concept recommendation method based on the large model in the absence of knowledge graph modality provided in the above embodiment, which includes a memory and a processor;

[0073] The memory is used to store computer programs;

[0074] The processor is used to implement the concept recommendation method based on the large model in the absence of knowledge graph modality in the above embodiment when executing the computer program.

[0075] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0076] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the concept recommendation method in the absence of a knowledge graph modality based on a large model provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the concept recommendation method in the absence of a knowledge graph modality based on a large model in the above embodiment.

[0077] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0078] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0079] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0080] The following will use a specific example to demonstrate the specific implementation and technical effect of the concept recommendation method based on the large model in the absence of knowledge graph modality described in S1 to S4 above.

[0081] Example

[0082] In this embodiment, the implementation method of the concept recommendation method based on the large model knowledge graph modality loss described in S1 to S4 is specifically demonstrated. The process can be found in Figure 4 As shown in the figure, it is divided into three parts: multimodal knowledge graph construction based on educational large language model, offline knowledge graph feature completion based on channel confidence, and knowledge concept recommendation and interpretation based on offline and online information. The following describes the specific implementation of the three parts in this process in detail.

[0083] 1. Construction of multimodal knowledge graph based on educational large language model

[0084] The characteristic of the concept recommendation problem is that the learner's choice depends largely on the human knowledge system, which is composed of the semantics of concepts and their mutual relations. In a structured teaching program, the connotation and extension of each concept must be fully understood. Therefore, integrating the semantics of each concept with contextual information is crucial to developing an effective concept recommendation system. However, directly using the concept name or relying on the educational large language model to generate an explanation for each concept often leads to ambiguity problems. This is because the same concept may have different meanings and interpretations in different subject areas or specific contexts. For example, "graphics" may refer to geometric figures in mathematics, and may involve image processing or graph theory in computer science. In order to address this challenge, this embodiment proposes a method based on a knowledge graph to improve the prompts for the educational large language model by combining the leading and trailing nodes in the knowledge graph. This method can not only help the educational large language model provide accurate explanations for specific educational backgrounds and significantly reduce the risk of ambiguity, but also effectively construct a multimodal offline knowledge graph, thereby improving the accuracy of knowledge concept recommendations.

[0085] 1.1 Feature Extraction of Multimodal Learning Materials

[0086] Text Features Using Sentence-Bert encoding; image features Using Visual Transformer (ViT), Visual Transformer (ViT); video features It uses a 3D convolutional network (C3D), which is specially designed for processing video data and can capture temporal and spatial information at the same time; audio features CPC is used to extract and learn audio features through self-supervision, which has achieved remarkable results. The feature encoding of the vth multimodal learning data is the multimodal feature :

[0087]

[0088] All multimodal learning materials are used as graph nodes. According to the interaction information of student users on multimodal learning materials, edge connections are established for the nodes in the graph that have interacted with the same student user (that is, if a student has interacted with w multimodal learning materials at the same time, then edge connections need to be established between these w multimodal learning materials), thereby constructing a learning material graph, and the node features in the learning material graph are the multimodal features of the corresponding multimodal learning materials. .

[0089] Based on this learning material graph, the GraphGPS encoding mechanism can be used to fuse the learning material graph. As the core encoding mechanism for feature fusion, GraphGPS can solve the limitations of processing long-distance dependencies and enhance the concept node representation capability through global information. GraphGPS itself belongs to the existing technology, which includes two modules: an embedding module and a processing module. The embedding module is used to effectively fuse the features of position encoding (PE), structure encoding (SE), nodes, edges, and graphs; while the processing module is used to combine local message passing with the global attention mechanism. GraphGPS uses the graph convolutional network GCN to capture local graph information and introduces the Transformer architecture to capture global information. By combining local graph information and global information, a richer and more comprehensive node representation can be generated, namely:

[0090]

[0091] In the formula is the vth multimodal learning material in Node representation of the layer; is the node representation of the previous layer; is a nonlinear transformation function; The Transformer architecture is introduced to use the softmax function to capture global information; Represents the vth multimodal learning material in the The query matrix, key matrix, and value matrix of the layer; Representative in the The query matrix, key matrix, and value matrix of the layer; is the aggregation function of the neighbor embeddings of the vth multimodal learning material; Representation concept The previous layer represents the combination function with the aggregation result.

[0092] In this embodiment, based on the learning material graph, the GraphGPS encoding mechanism is used to fuse the learning material graph, so that the node features in the graph can be updated from the original multimodal features to the fused features that fuse local and global information.

[0093] 1.2. Construction of Multimodal Knowledge Graph

[0094] In order to enrich the multimodal knowledge graph (KG), the main concepts are expanded by obtaining related concepts and categories using DBpedia Spotlight. These expanded concepts will enhance the structure of the teaching materials, promote learners' concept discovery, and help learners further master knowledge. Expanding the main concepts can also improve the effectiveness of the recommendation process by recommending relevant, novel, and diverse related knowledge concepts to learners. Concept expansion is performed by querying DBpedia using SPARQL. This query retrieves all related concepts and categories associated with each main concept. However, in the DBpedia knowledge base, each concept is associated with a large number of related concepts and categories. In addition, if all these related concepts and categories are merged into the KG without a filtering process, the KG will expand to an unmanageable size. Node embeddings are used to calculate the cosine similarity between each main concept and its preceding and succeeding concepts / categories to quantify the correlation between them. For each main concept , it is necessary to calculate it and Category , Pre-concept And the concept of postposition Cosine similarity of:

[0095]

[0096]

[0097]

[0098] In addition, this embodiment calculates the cosine similarity between each preceding and succeeding concept / category and the multimodal learning material. Category , Pre-concept and post-concept and multimodal learning materials The similarity is calculated as follows:

[0099]

[0100]

[0101]

[0102] These two similarity scores are combined by applying a summation to obtain an overall score for each preceding and succeeding concept / category.

[0103]

[0104]

[0105]

[0106] In order to decide which relevant concepts and categories should be retained in the KG, the educational large language model is combined to provide accurate explanations for specific educational contexts to significantly reduce the risk of ambiguity, and this overall similarity score is used to rank the most relevant preceding and following concepts / categories. In this way, this embodiment adds the top 20 preceding and following concepts and the top 3 categories of each main concept to the KG.

[0107] like Figure 5 As shown in the figure, a multimodal knowledge graph based on multimodal learning materials is created. In this multimodal knowledge graph KG, taking the v-th multimodal learning material of online student i as an example, a KG node is first created for this multimodal learning material ( ), which corresponds to 2 main concepts ( ), the main concept selects several pre-concepts according to the above process ( ) and post-concepts ( ), each concept has its own category ( ). Thus, these nodes and edges can form a multimodal knowledge graph:

[0108]

[0109] in, is a set of multimodal learning material nodes, is the main concept node set, is the set of preceding concept nodes, is the set of post-concept nodes, is the set of category nodes to which the main concept belongs, is the edge set in the multimodal knowledge graph.

[0110] It should be noted that for each main concept in the multimodal knowledge graph KG , it is necessary to use the educational language model to generate the main concepts The main concept explanation text is then encoded using a pre-trained language model (such as BERT), and the corresponding encoding features are used as the node features of this main concept node in KG. The remaining categories and concept nodes do not need to be interpreted and encoded by a large language model. In addition, the node features of the multimodal learning material nodes use the fusion features.

[0111] In addition, the main concept node in the KG also needs to record the student user's understanding status mark value of this main concept. ) nodes have three states relative to learners: "not understood", "understood" and "not interactively learned yet", where "not interactively learned yet" represents the initial state of MC, that is, the learner has not interacted with the main concept and it is a new concept. When learners interact with multimodal learning materials, learners can mark the main concepts extracted from the multimodal learning materials as "understood", "not understood" or "not interactively learned yet". The three states can be set with different tag values ​​and recorded on the nodes, for example, they can be recorded as 0, 1 and 2 respectively. In this way, concepts marked as "not understood" can be easily read from KG to build subsequent learner models.

[0112] To store the KG, this example uses a graph database (Neo4j), which is well suited for this purpose because it uses a graph structure with nodes and edges to represent and store data.

[0113] 2. Offline knowledge graph feature completion based on channel confidence

[0114] When constructing a multimodal knowledge graph in the field of education, channel data of different modalities have their own characteristics and limitations, and often lead to incomplete information or even missing key features due to factors such as the collection environment, transmission loss, or data noise. To address this challenge, this embodiment proposes an offline knowledge graph feature completion method based on channel confidence, which effectively completes missing features by evaluating the confidence of multimodal channel information, thereby improving the accuracy and practicality of the knowledge graph.

[0115] 2.1 Channel Quality Assessment

[0116] Based on the existing modal information, a channel quality assessment mechanism is constructed to calculate the confidence of each modality at a specific knowledge point. The present invention calculates the confidence using the shortest path distance between nodes.

[0117] The confidence of the restored channel features is generally determined by the completed features With the real original characteristics The similarity between is defined as a value between 0 and 1. However, when we get Before, The confidence level of is not available, and without confidence, the proposed scheme cannot obtain To solve this problem, this embodiment uses the shortest path distance (SPD-S) between a node of a specific channel and its nearest source node to design the confidence. For example, the SPD-S of the ith node of the dth channel feature is express

[0118]

[0119] in Represents the adjacency matrix Upper set The shortest path distance between the ith node and the nearest source node in . It is worth noting that if the ith node is a source node, its nearest source node is itself, which means becomes zero. The construction element is SPD-S Matrix , where N is the number of nodes in the multimodal learning data in the entire KG, and F is the number of node feature channels (i.e., feature dimension) corresponding to one modality in a single multimodal learning data.

[0120] Considering feature homogeneity, the feature similarity between any two nodes tends to increase as the shortest path distance between the two nodes decreases. Based on feature homogeneity, this embodiment assumes that as SPD-S decreases, the node Recovery characteristics More confidently similar to a given feature of its nearest source node. Based on the assumption, the SPD-S definition confidence used in this embodiment is:

[0121]

[0122] in is a hyperparameter, which can be set to 0.9 in this embodiment. , Confidence becomes 1. In addition, as The increase in the confidence of the missing node features Declining exponentially.

[0123] 2.2. Restoring node features through intra-channel diffusion

[0124] In order to restore the missing node features in a channel-wise manner through graph diffusion, the source node independently propagates its features to the adjacent nodes on each channel. The present invention does not simply aggregate all neighborhood features with the same value, but aggregates features of different importance according to confidence, thereby performing low-confidence aggregation on the restored features of the missing nodes and high-confidence aggregation on the given features of the source nodes. Based on this design goal, a confidence-based diffusion matrix is ​​designed in this embodiment.

[0125] First, we define "relative confidence", which represents the confidence in a specific node feature relative to another node feature:

[0126]

[0127] Then, assuming a missing node Features Gathered If the features in and are neighbors, then SPD-S and The difference between the SPD-S of a node and its neighbors cannot exceed 1. Therefore, the relative confidence of a node and its neighbors can be determined.

[0128] Before defining the transition matrix, this embodiment temporarily reorders the nodes according to whether the eigenvalue of the dth channel is known, that is, x(d) and a(d) are reordered for each channel. After the feature diffusion stage is completed, the nodes are sorted according to the original number. This embodiment constructs a weighted adjacency matrix for the dth channel , the self-connection is added with a weight of 1 so that each node can retain some of its own characteristics. The message from Deliver to The corresponding edge weight, A node is allowed to aggregate more high-confidence channel features from its neighbors than low-confidence channel features. Further, consider two connected nodes and Message transmission between

[0129]

[0130]

[0131]

[0132] This means that there is a lot of message passing from high confidence nodes to low confidence nodes, and very little message passing in the opposite direction. Adjust the strength of message transmission according to the confidence level. In this embodiment, .

[0133] In order to ensure the convergence of the diffusion process, this embodiment uses random normalization (np.linalg.norm function can be used) to Normalize:

[0134]

[0135]

[0136] The features of the missing nodes can be formed at the same scale as the known features. Keeping the original scale can restore the features to be close to the actual features. In this embodiment, the channel-by-channel node diffusion of the dth channel is defined as

[0137]

[0138] In the formula, for go through The feature vector recovered after step propagation.

[0139] If the total number of diffusion steps is preset to T', the final output after diffusion is completed is , all channels Composition recovery characteristics .

[0140] 2.3. Inter-channel propagation refines node features

[0141] In the previous stage, the restored features are obtained by performing channel-to-channel diffusion on each channel separately. However, the proposed feature diffusion is based on graph structure and pseudo confidence, but it does not consider the dependency between channels. Since the dependency between channels may be another important factor in inputting missing node features, this embodiment improves , the performance of downstream tasks is improved by considering channel correlation and pseudo confidence. In this stage, within a node, a low-confidence channel feature is refined by reflecting a high-confidence channel feature according to the correlation between the two channels.

[0142] This embodiment first prepares the correlation coefficient matrix , gives the correlation coefficient between each pair of channels a and b of the proposed scheme. In the example, the characteristics of channel a are With the characteristics of channel b Correlation coefficient The calculation formula is

[0143]

[0144]

[0145]

[0146] This embodiment designs a weight matrix The inter-channel propagation for each node is used to achieve three goals: highly correlated channels should exchange more information with each other than uncorrelated channels; low-confidence channel features should receive more information from other channels for refinement than high-confidence channel features; high-confidence channel features should propagate more information to other node channels than low-confidence channel features. Based on these design goals, From channel b to channel a The weight of the directed edge of

[0147]

[0148] in These are all hyperparameters. In this embodiment, , .

[0149] Node-by-node channel propagation output recovery features Node characteristics of node i The final input feature of this embodiment is defined as the node intelligent channel propagation

[0150]

[0151] 2.4 Offline feature integration and iterative optimization

[0152] After feature completion, the completed feature data is integrated back into the knowledge graph to ensure that the feature relationship of each node and its edges is enhanced, thereby improving the completeness and accuracy of the graph. The integrated node features are updated to capture deeper connections between knowledge points.

[0153] 3. Knowledge concept recommendation and explanation based on offline and online information

[0154] 3.1. Online Learner Model Representation Construction

[0155] The effectiveness of knowledge concept recommendation depends on a deep understanding of the learner's state and the comprehensiveness of the knowledge graph. To address this challenge, this embodiment proposes a concept recommendation method based on a multimodal knowledge graph with offline feature completion and combined with an online learner model to achieve personalized knowledge concept recommendation and explanation.

[0156] In the above steps, a rich feature-completed multimodal knowledge graph has been constructed, which contains complete concept features obtained through channel confidence evaluation and can effectively reflect the relationship between information and concepts in the learning materials. This embodiment needs to further construct an online learner model, which represents the learner's cognitive level based on the learner's understanding state when interacting with multimodal learning materials.

[0157] The learner model L is based on the concepts that the learner understands and does not understand when interacting with the multimodal learning materials. For example, the learner does not understand main concept 1 and main concept 3, but understands main concept 2. Assuming that the concept value understood by the learner is 0, the learner model can only be represented by the concepts that the learner does not understand. Therefore, this embodiment represents the learner model as a weighted average of the concepts that the learner does not understand, where the weight of the concept is calculated as the cosine similarity score between its embedding and the embedding of the learning material (based on the content of the learning material). As a result, concepts related to the learning material are given higher weights than irrelevant concepts. These concepts indicate that the learner needs to master these concepts first, making them more important to the learner. This embodiment can use the rich embeddings of the learner, as well as their respective weights, to obtain the embedding of the learner model:

[0158]

[0159]

[0160]

[0161] in It is the main concept corresponding to the current learner Embedded, is its weight in the multimodal learning matrix LM.

[0162] 3.2 Knowledge Concept Recommendation Based on Self-Attention

[0163] In the main concept and learner model After encoding the graph adapted to the text representation, two embedding layers are further added to map the preceding and following concepts and categories. Specifically, there is a preceding and following concept embedding matrix (K is the number of preceding and following concepts) and a category embedding matrix , indicating whether the answer is right or wrong. The final representation of each time step t is Encoded by four embedded parts:

[0164]

[0165] in For the final expression; It is the embedding of the preceding and following concepts; for category embedding; It is a conceptual representation; is the learner model representation.

[0166] Given a sequence of codes of a learner interacting with a multimodal learning resource , this paper further uses the widely used self-attention Transformer structure to encode the sequence:

[0167]

[0168] In the formula It is a feed-forward network; is the time step Output encoding of It is a multi-head attention algorithm.

[0169] From this, the scores of all concepts (including main concepts and preceding and succeeding concepts) can be calculated:

[0170]

[0171] in is a learnable parameter matrix.

[0172] When making personalized recommendations based on the learner model, the learner embedding model is calculated Based on the scores of candidate knowledge concepts, the top five related concepts with the highest scores and related preceding and succeeding knowledge concepts are found, and these knowledge concepts and related multimodal learning materials are recommended to learners to promote their learning process.

[0173] 3.3. Suggested explanations of knowledge concepts

[0174] To help learners understand the logic behind the recommended knowledge concepts, the system also combines the structural information and semantic associations in the knowledge graph (KG) to explain the recommendations. This explanation allows learners to clearly understand why the system recommends certain specific knowledge concepts, thereby enhancing the purposefulness and motivation of learning. To achieve this, this embodiment proposes a weighted path method for selecting the most explanatory path connecting the learner model with the recommended concept in the KG. Specifically, this embodiment performs weighted calculations on the paths in the KG that connect the learner's current knowledge state with the recommended concept, and preferentially selects the path with the highest weight as the basis for the recommended explanation. Suppose the learner has not yet mastered main concept 1 and main concept 3, and the recommended concept is related concept 1. At this time, the recommended explanation may be because related concept 1 is a pre- or post-concept of main concept 1 and main concept 3 in the KG, or because it shares the same pre- or post-concept 2 with main concept 1. This graph-structure-based explanation not only makes the recommendation more transparent, but also helps learners establish logical connections between different knowledge points.

[0175] In this method, the system uses four main paths in KG to make recommendation explanations and distinguishes these paths according to the type of recommended concepts. These four paths include:

[0176] 1) Direct Pre-path: If the recommended concept is a pre-knowledge point of a concept that the learner has not yet mastered in the KG, it is used as the explanation of the recommendation. This indicates that learning the recommended concept helps the learner better master the current core concept.

[0177] 2) Direct subsequent path: If the recommended concept is a direct subsequent knowledge point of a concept that the learner has not yet mastered, the system will recommend it as a knowledge point for further learning to help learners gain a deeper understanding of the current concept.

[0178] 3) Shared Predecessor Path: When the recommended concept shares a common predecessor concept with a concept that the learner has not mastered, the system will explain it based on this commonality. For example, if the recommended concept shares a predecessor concept 2 with both main concepts 1 and 3, then the connection established by concept 2 will provide an explanation for the recommendation.

[0179] 4) Shared post-path: Similarly, when the recommended concept has a common post-path concept with a concept that the learner has not mastered, it can also be interpreted as a recommendation. For example, if the recommended concept and main concepts 1 and 3 all point to a post-path concept 4, then it can be explained that learning the recommended concept helps the learner eventually master concept 4.

[0180] This embodiment quantifies the explanatory power of each path using path weight, which is determined by the association strength and semantic relevance between knowledge points in the KG. The calculation formula of path weight is as follows:

[0181]

[0182] in, represents the weight of the path, Indicates the path The weight of the node, Indicates the association between the ith node and its predecessor node on the path. In addition, the association strength and semantic relationship can be weighted in the path weight calculation to ensure that the selected information is the most explanatory and instructive for learners.

[0183] Finally, the online learning platform selects the path with the highest weight as the recommended explanation, so that learners can understand the recommended content and see the direct connection between the recommended knowledge points and existing knowledge points through the path relationship in the graph structure. This process not only provides a clear logical explanation for the recommendation, but also helps learners gradually expand their cognitive structure based on their existing knowledge, thereby effectively improving their learning experience and efficiency.

[0184] In order to verify the effectiveness of the concept recommendation method of the multimodal knowledge graph based on the large language model proposed in the present invention in the absence of attributes, this embodiment selects an actual online education data set for experimentation. The data set is collected from a real large-scale online learning platform and contains students' learning records, concept mastery and multimodal learning materials. The initial learning state of each student is initialized by historical learning data and concept mastery, and the performance in the subsequent learning process will continuously update its representation. At the same time, the semantic associations between concepts are dynamically captured and modeled by combining the knowledge graph proposed in the present invention with the large language model. Through the above process, this embodiment uses the concept recommendation method proposed in the present invention to recommend the next learning concept. The evaluation index of the experiment is AUC (i.e., the area under the ROC curve), and the AUC value ranges from 0 to 1. The larger the value, the better the effect of the model.

[0185] In this embodiment, multiple comparison methods are also provided, including:

[0186] Attentional graph convolutional networks for knowledge concept recommendation in moocs in a heterogeneous view[C] / / Proceedings ofthe 43rd international ACM SIGIR conference on research and development ininformation retrieval. 2020: 79-88.

[0187] ConceptGCN, a concept graph neural network, is derived from the existing technical literature: Alatrash R, Chatti MA, Ain QU, et al. ConceptGCN: Knowledge concept recommendation in MOOCs based on knowledge graph convolutional networks and SBERT[J]. Computers and Education: Artificial Intelligence, 2024, 6: 100193.

[0188] Heuristic concept construction approach to collaborative recommendation HCSC, which is derived from the existing technical literature: Liu ZH, Zhao Q, Zou L, et al. A heuristic concept construction approach to collaborative recommendation[J]. International Journal of Approximate Reasoning, 2022, 146:119-132.

[0189] The comparison of recommendation results of the method of the present invention and three comparative methods on educational datasets with different attribute missing rates is shown in Table 1, Table 2 and Table 3.

[0190] Table 1 Concept recommendation results of the proposed method on an education dataset with 30% missing attributes

[0191] Recommended Method AUC(%) ACKRec 68.46 ConceptGCN 75.33 HCSC 71.28 The present invention 91.26

[0192] Table 2 Concept recommendation results of the proposed method on an education dataset with 50% missing attributes

[0193] Recommended Method AUC(%) ACKRec 61.33 ConceptGCN 70.05 HCSC 64.33 The present invention 87.26

[0194] Table 3 Concept recommendation results of the proposed method on an education dataset with 70% missing attributes

[0195] Recommended Method AUC(%) ACKRec 58.81 ConceptGCN 67.41 HCSC 61.23 The present invention 74.33

[0196] The experimental results show that the proposed method has achieved significant improvement in the AUC evaluation index under different attribute missing rates. Compared with the existing benchmark methods, the concept recommendation model based on structure and knowledge perception can more effectively capture the deep semantic relationship between concepts and make accurate recommendations based on the students' knowledge status.

[0197] In addition, an ablation study is also performed in this embodiment to better examine the contribution of different links in the method of the present invention. The experimental results are shown in Table 4. The specific details of each module are as follows:

[0198] W / LLM: Remove LLM's hints for the preceding and succeeding concepts in the knowledge graph, directly use the concept names to construct the knowledge graph, and examine the role of LLM in eliminating concept ambiguity.

[0199] W / FP: Remove the offline knowledge graph feature completion module based on channel confidence and only use the uncompleted knowledge graph for concept embedding learning. This will explore whether feature completion plays a key role in improving the stability and performance of concept embedding.

[0200] W / OS: Do not use online learner model representation, only offline learner model representation. This helps to seriously determine whether the module captures the learner's understanding state when interacting with multimodal learning materials, and represents the learner's cognitive level in real time.

[0201] Table 4 Ablation test results of the proposed method with 30% attribute missing

[0202] Recommended method AUC(%) W / LLM 75.43 W / FP 70.63 W / OS 82.44

[0203] Table 5 Ablation test results of the proposed method with 50% attribute missing

[0204] Recommended method AUC(%) W / LLM 71.86 W / FP 68.83 W / OS 78.84

[0205] Table 6 Ablation test results of the proposed method with 70% attribute missing

[0206] Recommended method AUC(%) W / LLM 64.83 W / FP 61.63 W / OS 71.34

[0207] The above ablation experiment results show that the two improved modules proposed in the present invention have significant effects. The main reason is that the use of the LLM module to prompt the preceding and succeeding concepts in the knowledge graph is crucial in eliminating concept ambiguity. If the LLM module is not used to generate prompts but the concept name is directly used to generate explanations for each concept, it often leads to ambiguity problems, thereby reducing the recommendation accuracy; in addition, the offline knowledge graph feature completion module based on channel confidence plays an important role in enhancing embedding stability. After removing this module, although the model is still usable, the generalization ability of the model is significantly reduced, and the performance is poor in concept recommendation; replacing the online learner model representation with the offline learner model representation, the recommendation accuracy decreases, reflecting the effectiveness of the real-time representation of the learner's cognitive level. In summary, the offline knowledge graph feature completion module based on channel confidence and the LLM concept prompts in the knowledge graph are crucial to improving the performance of the model, and the online learner model representation further optimizes the recommendation accuracy of the model.

[0208] In summary, the concept recommendation method proposed in the present invention can make full use of LLM to optimize the knowledge graph, enhance the knowledge graph through the feature completion module based on channel confidence, and combine with the online learner model to provide an efficient concept recommendation solution for the online education platform.

[0209] The above-described embodiments are only some preferred implementations of the present invention, but are not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A concept recommendation method based on a large model in the absence of knowledge graph modality, characterized in that: include: S1. Based on the historical learning records of each student user on the online learning platform, all the multimodal learning materials that each student user has learned are obtained and multimodally encoded. The GraphGPS encoding mechanism is used to fuse the graph with multimodal learning materials as nodes, and the node features in the graph are updated from the original multimodal features to the fusion features that fuse local and global information. S2. Based on the historical learning records of each student user on the online learning platform, online student nodes, multimodal learning material nodes, and main concept nodes contained in the multimodal learning material nodes are constructed in the knowledge graph, and edge connections are established; the large language model is used to generate the explanation text of each main concept, and the encoding features of the main concept explanation text are used as the node features of the main concept node, and the node features of the multimodal learning material node adopt the fusion features, and the student user's understanding status mark value of the main concept is recorded on the main concept node; then, based on the semantic knowledge base, the category set, the pre-concept set, and the post-concept set of each main concept are searched, and based on the similarity of the encoding features, a part of the categories, pre-concepts, and post-concepts most relevant to the main concept are selected from each of the three sets, and expanded into the neighborhood nodes of the corresponding main concept node, thereby obtaining a multimodal knowledge graph; S3. For multimodal learning data nodes with missing node features in the multimodal knowledge graph, an offline knowledge graph feature completion method based on channel confidence is used. According to the channel confidence obtained by the channel quality assessment, the node features are first diffused and restored within the channel, and then the node features are propagated and refined between channels. The node features of the missing nodes are completed modally one by one, and the obtained missing node features are integrated and completed into the multimodal knowledge graph. S4. During the online learning process of the target student users, the learner model representations of different historical time steps are generated through the online learner model. The learner model representation of each historical time step is obtained by weighting the node feature representations of all the main concepts that are not understood in the multimodal learning materials interacted by the target student users in this historical time step; then the preceding concept node features, the following concept node features, the category node features, the main concept node features, and the learner model representation in each historical time step are sequentially spliced ​​to form the final representation of each historical time step; the time series of the final representation is encoded through the self-attention Transformer model, and then the recommendation scores of all concepts are output through the prediction head to generate the recommendation results.

2. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: The multimodal learning materials include four modalities: text, image, video and audio. After being encoded by corresponding encoders, each modality constitutes a multimodal feature corresponding to the multimodal learning materials.

3. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: In S1, the method of fusing the graph using the GraphGPS encoding mechanism is as follows: using all the multimodal learning materials on the online learning platform as graph nodes, establishing edge connections between the nodes in the graph that have interacted with the same student user according to the interaction information of the student users for the multimodal learning materials, thereby constructing a learning material graph, and the node features in the learning material graph are the multimodal features of the corresponding multimodal learning materials; using the GraphGPS encoding mechanism to fuse the learning material graph, the node features in the graph are updated from the original multimodal features to fused features that fuse local and global information.

4. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: In S2, when establishing edge connections between nodes in the multimodal knowledge graph, it is necessary to establish edge connections between each online student node and each multimodal learning material node that has interacted with it based on the historical learning records of each student user, and to establish edge connections between each multimodal learning material node and each main concept node contained in it. At the same time, each main concept needs to be edge-connected with each filtered category node, each preceding concept node, and each succeeding concept node. The edge connection relationship between the nodes in the graph is recorded in the adjacency matrix of the multimodal knowledge graph.

5. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: In the multimodal knowledge graph, each main concept node needs to record the student user's understanding status mark value of this main concept at different time steps based on the correctness of the answers in the historical learning records, where the understanding status mark values ​​are divided into three types: not understood, understood, and not interactively learned yet.

6. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: In S3, for any modality, the specific steps of using the offline knowledge graph feature completion method based on channel confidence to complete the missing node features of the modality in the multimodal knowledge graph are as follows: S31. For the current modality to be completed, all feature channels of the modality to be completed are extracted from the multimodal knowledge graph, and then the shortest path distance between each multimodal learning material node and the nearest source node is calculated for each feature channel, where the nearest source node is a multimodal learning material node that has a feature value in the calculated feature channel and is closest to the calculated multimodal learning material node; for each feature channel respectively, the shortest path distance is converted into a confidence by a power operation with a base range of (0,1); S32. For each feature channel, calculate the confidence ratio of any two multimodal learning material nodes in this feature channel, and record it in the weighted adjacency matrix of this feature channel as the relative confidence between the nodes, apply row random normalization operation to the weighted adjacency matrix to obtain a normalized weighted adjacency matrix, use the normalized weighted adjacency matrix to diffuse the current latest multimodal knowledge graph channel by channel node, and diffuse the feature values ​​of other nodes in each feature channel to the missing multimodal learning material nodes; Iterate and diffuse multiple times to obtain the initial recovery features of all multimodal learning material nodes in the multimodal knowledge graph; S34, based on the initial recovery features, calculate the feature correlation coefficient between any two feature channels, and then for each multimodal learning material node, convert the corresponding feature correlation coefficient to obtain the directed edge weight from one feature channel to another feature channel, and record the directed edge weights between the feature channels in the weight adjacency matrix; use the weight adjacency matrix to perform channel-to-channel propagation on each node in the current latest multimodal knowledge graph, and diffuse the average node features of each channel to the multimodal learning material nodes missing in each feature channel; Iterate and diffuse multiple times to obtain the final restored features of all multimodal learning material nodes in the multimodal knowledge graph; S35. For multimodal learning material nodes in the multimodal knowledge graph that have missing node features in the current modality, the corresponding final restored features are used to complete the node features.

7. The concept recommendation method based on a large model in the absence of knowledge graph modality as claimed in claim 1, characterized in that: In S4, when all node feature representations of the main concepts that are not understood are weighted to obtain the learner model representation, the weight used for each main concept that is not understood is the cosine distance between the node feature representation of the main concept that is not understood and the node feature representation of the multimodal learning material.

8. A concept recommendation system based on a large model with missing knowledge graph modality, characterized in that: include: The learning record collection module is used to collect the historical learning records of each student user on the online learning platform; The recommendation object designation module is used to allow the management user to designate the student users who need concept recommendations through the GUI interface; The concept recommendation module is used to obtain the historical learning records corresponding to the designated student users from the learning record collection module according to the designated student users in the recommendation object designation module, and then recommend the concepts for the designated student users to learn next according to the concept recommendation method based on the large model in the absence of knowledge graph modality as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the concept recommendation method based on the large model in the absence of knowledge graph modality is implemented as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the concept recommendation method based on a large model in the absence of a knowledge graph modality as described in any one of claims 1 to 7 when executing the computer program.

Images