Educational Resource Recommendation Method and System under Modal Missing Based on Dual Collaborative Hypergraph

Abstract

The present invention discloses an educational resource recommendation method and system under modality missing based on dual collaborative hypergraphs. The method includes: S1, constructing a user-resource interaction graph and a resource co-occurrence graph, recording the connection relationship through an adjacency matrix, and extracting resource features by using multi-modal coding; S2, aiming at the modality missing problem, calculating the confidence between resource nodes, and adopting a two-stage feature completion strategy of in-channel diffusion and inter-channel propagation; S3, constructing a user hypergraph and a modality hypergraph, realizing cross-modal collaborative signal propagation through a dual collaborative hypergraph neural network, and respectively capturing user preferences and in-modality feature fusion; S4, performing multi-modal embedding fusion on the user to be recommended, and calculating the recommendation preference score to achieve personalized recommendation. This method effectively solves the personalized recommendation problem in the modality missing scenario and can adapt to the complex requirements of the educational scenario.

Description

Technical Field

[0001] The present invention belongs to the field of personalized recommendation, and particularly relates to an educational resource recommendation method and system under modality missing based on a dual collaborative hypergraph. Background Art

[0002] With the rapid development of online education platforms, educational resources have gradually shown multi-modal characteristics (such as course videos, text lectures, interactive exercises, etc.). However, due to differences in the habits of resource creators or platform limitations, there are often modality missing problems in educational resources (for example, some courses have only text without videos, or there are no supporting exercises). Traditional recommendation systems rely on complete modal data and are difficult to effectively handle such missingness, resulting in a decline in recommendation accuracy and affecting students' learning experience.

[0003] In the prior art, multi-modal recommendation methods usually assume modal integrity or adopt simple feature imputation strategies, which cannot fully explore the global associations between modalities and significantly degrade in performance under high missing rates. In addition, existing methods usually ignore the high-order relationships between modalities (such as the association between videos and texts), resulting in a lack of semantic consistency in the recommendation results. For example, a certain student prefers video courses, but due to the lack of videos in some courses, traditional methods may recommend irrelevant text courses, affecting the learning effect. Therefore, there is an urgent need for a recommendation technology that can accurately complete the missing modalities and model the high-order relationships of multi-modalities to adapt to the complex requirements of the educational scenario. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that it is difficult to achieve personalized recommendation of multi-modal educational resources in the educational field under the scenario of modality missing, and to provide an educational resource recommendation method and system under modality missing based on a dual collaborative hypergraph.

[0005] The specific technical solutions adopted by the present invention are as follows:

[0006] In the first aspect, the present invention provides an educational resource recommendation method under modality missing based on a dual collaborative hypergraph, which includes:

[0007] S1. Obtain the behavior data of all users and all multi-modal educational resources on the online learning platform, construct a user-resource interaction graph with all users and all multi-modal educational resources as two types of nodes, establish a first adjacency matrix according to the interaction information in the behavior data to record the edge connection relationship between user nodes and resource nodes, and each multi-modal educational resource is respectively encoded through multi-modalities to obtain multi-modal features;

[0008] S2. Construct a resource co-occurrence graph with all multimodal educational resources as nodes, and record the co-occurrence relationships between multimodal educational resources through the second adjacency matrix. For each resource modality, divide the resource nodes in the resource co-occurrence graph into missing nodes and complete nodes according to whether there is modality missing, and calculate the modality confidence between resource nodes. Based on the modality confidence and the multimodal features of the complete nodes, complement the modality features of the missing nodes in the current resource modality in a way that first diffuses within the channel to restore node features and then propagates between channels to refine node features.

[0009] S3. For each resource modality, construct a user hypergraph and a modality hypergraph within the resource modality respectively, and based on the dual collaborative hypergraph convolution in the dual collaborative hypergraph neural network, on the one hand, the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) propagates collaborative signals between users and resources to capture the correlation between user preferences and resource features, and on the other hand, the modality collaborative signal hypergraph convolutional neural network (MSHNN) propagates collaborative signals within each resource modality to achieve intra-modal fusion and complementation of features, and outputs user embeddings and resource embeddings in different modalities.

[0010] S4. For the user to be recommended, fuse the user embeddings and resource embeddings in different modalities, calculate the recommendation preference score of each multimodal educational resource relative to the user to be recommended, and perform personalized recommendation.

[0011] As a preference of the above first aspect, the multimodal educational resources include various learning resource modalities such as text, video, image, and audio. When performing the multimodal encoding, ResNet is used to encode the image modality and the video modality, BERT is used to encode the text modality, and Mel cepstral coefficients are used to encode the audio modality.

[0012] As a preference of the above first aspect, in S2, for each resource modality, the method for complementing the modality features of the missing nodes in the current resource modality is:

[0013] S21. For each resource node in the resource co-occurrence graph, determine the shortest path distance from the resource node to the nearest complete node, and convert the shortest path distance into a modality confidence through a power operation with a base range of (0, 1).

[0014] S22. For any two resource nodes in the resource co-occurrence graph, take the ratio of their modal confidence levels as the relative confidence level of these two resource nodes, and convert the second adjacency matrix into a weighted adjacency matrix according to the relative confidence level. The values on the diagonal of the second adjacency matrix are all set to 1, and the elements with the remaining values of 0 remain unchanged, while the elements with the remaining values of 1 are set to the relative confidence level of the two resource nodes corresponding to this element; perform a random normalization operation on the weighted adjacency matrix. After obtaining the normalized weighted adjacency matrix, then combine the graph diffusion algorithm to perform node-to-node diffusion on the resource co-occurrence graph channel by channel. After iteratively diffusing multiple times, obtain the initial recovery features of all resource nodes in the current resource modality;

[0015] S23. Based on the initial recovery features, calculate the feature correlation coefficient between any two feature channels. Then, for each resource node in the resource co-occurrence graph, obtain the directed edge weight from one feature channel to another feature channel through conversion calculation of the corresponding feature correlation coefficient, and record the directed edge weights between feature channels in the weight adjacency matrix; use the weight adjacency matrix to perform inter-channel propagation on each resource node in the resource co-occurrence graph to obtain the modal features of the missing nodes in the current resource modality.

[0016] As a preference of the above first aspect, in S3, for each resource modality, the specific steps for calculating the user embedding and the resource embedding are as follows:

[0017] S31. For the current resource modality m in the student-resource interaction graph, generate a membership matrix based on fuzzy c-means (FCM) and perform 0-1 binarization on the membership matrix using the membership threshold to obtain the hypergraph incidence matrix of the resources , and then multiply the first adjacency matrix by the hypergraph incidence matrix of the resources to obtain the hypergraph incidence matrix of the users ; finally, construct the user hypergraph and the modality hypergraph of the current resource modality m based on the two hypergraph incidence matrices respectively;

[0018] S32. Take the student-resource interaction graph as a hypergraph and input it into the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) stacked by multiple first hypergraph convolutional layers, and layer by layer propagate the collaborative signal between users and resources; during the propagation process of the first hypergraph convolutional layer, adopt the method of normalized graph convolution to propagate the node features of the resource nodes to the user nodes, and at the same time propagate the node features of the user nodes to the resource nodes; the last hypergraph convolutional layer outputs the first user embedding of each user node and the first resource embedding of each resource node;

[0019] S33. Combine the user hypergraph with the modality hypergraph Input the Modal Synergy Signal Hypergraph Convolutional Neural Network (MSHNN) stacked by multiple second hypergraph convolutional layers, and perform collaborative signal propagation layer by layer within each resource modality; during the collaborative signal propagation of each resource modality in the second hypergraph convolutional layer, in the way of normalized graph convolution, for each user node, it is necessary to aggregate all user node features under the hyperedges to which it belongs in the user hypergraph to itself, for each resource node, it is necessary to aggregate all resource node features under the hyperedges to which it belongs in the resource hypergraph to itself, and for each hyperedge, it is necessary to first aggregate all user node features under the hyperedges to which it belongs in the user hypergraph to the current user node itself to obtain the user hyperedge aggregation feature, then aggregate all resource features under the hyperedges to which it belongs in the resource hypergraph to the current resource node itself to obtain the resource hyperedge aggregation feature, and then superimpose the user hyperedge aggregation feature and the resource hyperedge aggregation feature and update the node features of the hyperedge; the last second hypergraph convolutional layer outputs the second user embedding of each user node under each resource modality m and the second resource embedding of each resource node .

[0020] As a preference of the above first aspect, in S4, the calculation method of the recommendation preference score between the to-be-recommended user and each to-be-recommended multi-modal educational resource is: multiply the first user embedding of the to-be-recommended user by the transposed result of the first resource embedding of the to-be-recommended multi-modal educational resource to obtain the first score value; then for each resource modality, multiply the second user embedding of the to-be-recommended user by the transposed result of the second resource embedding of the to-be-recommended multi-modal educational resource to obtain the second score value of each resource modality; finally, sum the first score value and the second score values of all resource modalities as the recommendation preference score of the to-be-recommended multi-modal educational resource relative to the to-be-recommended user.

[0021] As a preference of the above first aspect, the dual collaborative hypergraph neural network is pre-optimized and trained on the training dataset based on the total loss function;

[0022] The total loss function is the weighted sum of the BPR loss and the hypergraph reconstruction loss;

[0023] The dual collaborative hypergraph BPR loss is obtained by weighted summing the BPR loss of the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) and the BPR losses of the modal collaborative signal hypergraph convolutional neural networks (MSHNNs) of all resource modalities;

[0024] The hypergraph reconstruction loss is obtained by summing the hypergraph correlation matrix reconstruction losses corresponding to the user hypergraph and the resource hypergraph in all resource modalities respectively.

[0025] Preferably, in the first aspect above, when the dual collaborative hypergraph neural network is used in the actual recommendation process, it is necessary to collect the feedback data of student users and construct an incremental learning data set, and regularly use the incremental learning data set to perform incremental learning optimization on the network parameters.

[0026] In a second aspect, the present invention provides an educational resource recommendation system under modality loss based on a dual collaborative hypergraph, which includes:

[0027] A historical information recording module, configured to record the behavior data of all users on the online learning platform and all multimodal educational resources uploaded;

[0028] A recommended object selection module, configured to allow a user to select a user to be recommended;

[0029] An educational resource recommendation module, configured to, according to the information recorded in the historical information recording module, perform educational resource recommendation for the user-selected user to be recommended according to the educational resource recommendation method under modality loss based on a dual collaborative hypergraph according to any one of the above-mentioned first aspect solutions.

[0030] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the educational resource recommendation method under modality loss based on a dual collaborative hypergraph according to any one of the above-mentioned first aspect solutions is implemented.

[0031] In a fourth aspect, the present invention provides a computer electronic device, which includes a memory and a processor;

[0032] The memory is used to store a computer program;

[0033] The processor is configured to, when executing the computer program, implement the educational resource recommendation method under modality loss based on a dual collaborative hypergraph according to any one of the above-mentioned first aspect solutions.

[0034] The present invention has the following beneficial effects compared with the prior art:

[0035] Through feature completion driven by modal confidence, and dual-channel modeling of user-resource collaborative hypergraph and modal collaborative hypergraph, the present invention realizes personalized recommendation in the scenario of modal missing of multimodal educational resources (such as text, video, audio, etc.), which can significantly improve the learning experience of students and the resource utilization rate to adapt to the complex requirements of educational scenarios. Experimental results prove that the method proposed by the present invention has achieved significant improvement in the recall evaluation index under different attribute missing rates. Moreover, ablation experiment results show that both modal completion based on modal confidence and dual collaborative hypergraph neural network improve the accuracy of personalized recommendation, among which the user-resource collaborative hypergraph convolutional neural network has the greatest impact on the recommendation results. Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the steps of the educational resource recommendation method under modal missing based on dual collaborative hypergraph;

[0037] Figure 2 It is a schematic diagram of multimodal encoding of multimodal educational resources;

[0038] Figure 3 It is a schematic diagram of the module composition of the educational resource recommendation system under modal missing based on dual collaborative hypergraph;

[0039] Figure 4 It is a schematic diagram of the composition of a computer electronic device;

[0040] Figure 5 It is a specific implementation flowchart of the educational resource recommendation method in the embodiment of the present invention. Detailed Embodiments

[0041] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following describes the detailed embodiments of the present invention with reference to the accompanying drawings. Many specific details are set forth in the following description to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined correspondingly without conflict.

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of these features.

[0043] In a preferred embodiment of the present invention, a method for recommending educational resources in the absence of modalities based on a dual collaborative hypergraph is provided, which includes the following steps:

[0044] S1. Obtain the behavior data of all users and all multimodal educational resources on the online learning platform. Construct a user-resource interaction graph with all users and all multimodal educational resources as two types of nodes, and establish a first adjacency matrix according to the interaction information in the behavior data to record the edge connection relationship between user nodes and resource nodes. Each multimodal educational resource obtains multimodal features through multimodal encoding.

[0045] It should be noted that multimodal educational resources refer to learning resources on the online learning platform, such as course videos, course audios, text lectures, interactive exercises, etc. In the embodiments of the present invention, a single above-mentioned multimodal educational resource may include multiple learning resource modalities such as text, video, image, and audio. As Figure 2 shown, when performing the multimodal encoding, ResNet is used to encode the image modality and video modality, BERT is used to encode the text modality, and Mel Frequency Cepstral Coefficients (MFCC) are used to encode the audio modality. After the four modalities of text, image, video, and audio are encoded by their respective encoders, the multimodal features corresponding to the multimodal educational resources are formed.

[0046] It should be noted that the edge connection relationship between user nodes and resource nodes in the user-resource interaction graph is constructed according to the interaction information in the behavior data of users. Such interaction information is the learning record of student users for multimodal educational resources and the interactive scoring record of student users for multimodal educational resources. When there is an interaction between a student user and a certain multimodal educational resource, an edge connection relationship needs to be established between the corresponding two nodes, and the element value in the first adjacency matrix R corresponding to these two nodes needs to be set to 1. Otherwise, no edge connection is established between these two nodes, and the element value in the first adjacency matrix R corresponding to these two nodes needs to be set to 0.

[0047] S2. Construct a resource co-occurrence graph with all multimodal educational resources as nodes, and record the co-occurrence relationship between multimodal educational resources through a second adjacency matrix; for each resource modality, divide the resource nodes in the resource co-occurrence graph into missing nodes and complete nodes according to whether there is a modality missing, and calculate the modality confidence between resource nodes. Based on the modality confidence and the multimodal features of the complete nodes, the modality features of the missing nodes in the current resource modality are complemented in a way of first diffusing and restoring node features within the channel and then propagating and refining node features between channels.

[0048] On online learning platforms such as MOOCs, multi-modal educational resources often face the problem of incomplete modal data and missing information. The lack of modal data may significantly affect the effectiveness of resource recommendation. Therefore, the present invention needs to construct a resource co-occurrence graph to complete the modal features for each resource modality separately. The resource co-occurrence graph is constructed with all multi-modal educational resources as nodes, and the feature of each node is the multi-modal feature of the multi-modal educational resource corresponding to this node. The distinction between missing nodes and complete nodes and the feature completion are carried out for a single resource modality, that is: for a certain resource modality m, if the modal feature of a resource node v in the resource co-occurrence graph corresponding to the resource modality m is missing, then this resource node v is regarded as a missing node under the resource modality m, and it is necessary to use the modal features of other complete nodes under the resource modality m for completion.

[0049] It should be noted that the co-occurrence relationship between multi-modal educational resources refers to that multi-modal educational resources are interacted with by the same student users simultaneously. In the embodiments of the present invention, the second adjacency matrix A for recording the co-occurrence relationship between multi-modal educational resources can be calculated through the first adjacency matrix R, and the calculation formula is .

[0050] In the embodiments of the present invention, for each resource modality m, the method for completing the modal features of the missing nodes under the current resource modality m is as follows:

[0051] S21. For each resource node in the resource co-occurrence graph, determine the shortest path distance from this resource node to the nearest complete node, and convert the shortest path distance into a modal confidence (modal confidence, MC) through a power operation with a base range of (0, 1).

[0052] It should be noted that the above-mentioned modal confidence MC needs to be calculated separately for each missing node and complete node. For any resource node, the calculation formula of its MC can be expressed as:

[0053]

[0054] Among them, is the shortest path distance from the resource node to the nearest complete node d in the resource co-occurrence graph. If the resource node itself is a complete node, then the corresponding , is a hyperparameter used as the base of the power operation to control the feature propagation intensity.

[0055] S22. For any two resource nodes in the resource co-occurrence graph, the ratio of their modal confidence levels is used as the relative confidence level of these two resource nodes, and the second adjacency matrix is converted into a weighted adjacency matrix according to the relative confidence level. The values on the diagonal of the second adjacency matrix are all set to 1, and the elements with the remaining values of 0 remain unchanged, while the elements with the remaining values of 1 are set to the relative confidence level of the two resource nodes corresponding to this element. After performing a random normalization operation on the weighted adjacency matrix to obtain a normalized weighted adjacency matrix, the node-to-node diffusion of the resource co-occurrence graph is performed channel by channel in combination with the graph diffusion algorithm, and after multiple iterations of diffusion, the initial recovery features of all resource nodes in the current resource modality are obtained.

[0056] It should be noted that if the modal confidence levels MC of resource node and resource node are respectively denoted as and , then the relative confidence level of resource node relative to resource node . Correspondingly, the conversion formula for converting the second adjacency matrix A into a weighted adjacency matrix W according to the relative confidence level can be expressed as:

[0057]

[0058] In the formula: represents the element value at the coordinate position in the second adjacency matrix A, and represents the element value at the coordinate position in the weighted adjacency matrix W.

[0059] In addition, it should be noted that performing a random normalization operation on the weighted adjacency matrix W belongs to the prior art and can be implemented using the degree matrix. The formula for obtaining the normalized weighted adjacency matrix through the random normalization operation can be expressed as:

[0060]

[0061] In the formula: is the degree matrix of the weighted adjacency matrix .

[0062] S23. Calculate the feature correlation coefficient between any two feature channels based on the initial recovery features. Then, for each resource node in the resource co-occurrence graph, calculate the directed edge weight from one feature channel to another by transforming the corresponding feature correlation coefficient, and record the directed edge weights between feature channels in the weight adjacency matrix. Use the weight adjacency matrix to perform inter-channel propagation for each resource node in the resource co-occurrence graph to obtain the modal features of the missing nodes in the current resource modality.

[0063] Through the above feature completion method based on modal confidence MC, the present invention can complete the modal features in multi-modal educational resources, enhance the integrity of the feature space of the knowledge graph, and lay a foundation for personalized learning recommendations of educational resources.

[0064] S3. For each resource modality, construct the user hypergraph and the modal hypergraph within the resource modality respectively. Based on the double co-hypergraph convolution in the dual co-hypergraph neural network, on the one hand, the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) propagates the collaborative signal between users and resources to capture the correlation between user preferences and resource features. On the other hand, the modal collaborative signal hypergraph convolutional neural network (MSHNN) performs collaborative signal propagation within each resource modality to achieve intra-modal fusion and completion of features, and outputs user embeddings and resource embeddings in different modalities.

[0065] In the embodiment of the present invention, the specific steps for calculating the user embeddings and resource embeddings in each resource modality m are as follows:

[0066] S31. For the current resource modality m in the student-resource interaction graph, generate the membership matrix based on fuzzy c-means (FCM) and perform 0-1 binarization on the membership matrix using the membership threshold to obtain the hypergraph incidence matrix of the resources. , and then multiply the first adjacency matrix by the hypergraph incidence matrix of the resources to obtain the hypergraph incidence matrix of the users. ; Finally, construct the user hypergraph and the modal hypergraph of the current resource modality m based on the two hypergraph incidence matrices respectively.

[0067] It should be noted that fuzzy clustering (FCM), that is, fuzzy C-means (FCM) clustering, is a clustering method with flexible partitioning. By calculating the membership matrix of samples, the similarity between objects assigned to the same cluster is maximized, while the similarity between different clusters is minimized. The specific method of generating the membership matrix based on fuzzy clustering (FCM) belongs to the prior art and will not be elaborated here. The membership matrix can be binarized to 0-1 using a membership threshold. The element values in the membership matrix that exceed the threshold are set to 1, and the element values that do not exceed the threshold are set to 0, thereby obtaining the hypergraph incidence matrix of the resources. And the hypergraph incidence matrix of the user can be calculated in combination with the first adjacency matrix R, that is . Since the hypergraph incidence matrix records the association relationship between each node and each hyperedge, the user hypergraph of the current resource modality m can be constructed according to the hypergraph incidence matrix of the resources , and the modality hypergraph of the current resource modality m can be constructed according to the hypergraph incidence matrix of the user . .

[0068] S32. Take the student-resource interaction graph as a hypergraph and input it into the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) stacked by multiple first hypergraph convolutional layers, and layer by layer propagate the collaborative signal between users and resources; during the propagation process of the first hypergraph convolutional layer, in the way of normalized graph convolution, propagate the node features of the resource nodes to the user nodes, and at the same time propagate the node features of the user nodes to the resource nodes; output the first user embedding of each user node and the first resource embedding of each resource node .

[0069] It should be noted that although the above-mentioned student-resource interaction graph is not a hypergraph similar to the user hypergraph and the modality hypergraph , it can still be regarded as a special hypergraph (Hypergraph), in which the number of nodes connected by the hyperedge is 2.

[0070] It should be noted that the normalized graph convolution used in the propagation process of the first hypergraph convolutional layer means that it is necessary to use the degree matrix of nodes and hyperedges in the input hypergraph (that is, the student-resource interaction graph) for normalization to solve the problem that stacking multiple hypergraph convolutional layers will increase the vanishing gradient and numerical instability. The propagation method of the hypergraph convolutional layer belongs to the prior art of graph convolution, and its core is to aggregate the neighbor node features with edge connections to its own node, and can be implemented with reference to the prior art.

[0071] S33. Take the user hypergraph With the modal hypergraph Input the Modal Synergistic Signal Hypergraph Convolutional Neural Network (MSHNN) stacked by multiple second hypergraph convolutional layers, and perform collaborative signal propagation layer by layer within each resource modality; during the collaborative signal propagation of each resource modality in the second hypergraph convolutional layer, in the way of normalized graph convolution, for each user node, it is necessary to aggregate all user node features under the hyperedges to which it belongs in the user hypergraph to its own node, for each resource node, it is necessary to aggregate all resource node features under the hyperedges to which it belongs in the resource hypergraph to its own node, and for each hyperedge, it is necessary to first aggregate all user node features under the hyperedges to which it belongs in the user hypergraph to the current user node itself to obtain the user hyperedge aggregation feature, then aggregate all resource features under the hyperedges to which it belongs in the resource hypergraph to the current resource node itself to obtain the resource hyperedge aggregation feature, and then superimpose the user hyperedge aggregation feature and the resource hyperedge aggregation feature and update the node features of the hyperedge; the last second hypergraph convolutional layer outputs the second user embedding of each user node under each resource modality m and the second resource embedding of each resource node .

[0072] Similarly, it should be noted that the normalized graph convolution is also used in the propagation process of the second hypergraph convolutional layer, which means that it is necessary to use the degree matrices of nodes and hyperedges in the input hypergraph for normalization to solve the problem that stacking multiple hypergraph convolutional layers will increase the problem of gradient disappearance and numerical instability. Since the input hypergraphs corresponding to user nodes and resource nodes are different, the degree matrices of nodes and hyperedges in the user hypergraph are used for normalization when calculating for each user node and the degree matrices of nodes and hyperedges in the modal hypergraph are used for normalization when calculating for each resource node . The propagation method of the hypergraph convolutional layer belongs to the existing technology of graph convolution, and its core is to aggregate the features of neighbor nodes connected by edges to its own node, which can be implemented with reference to the existing technology

[0073] S4. For the user to be recommended, fuse the user embeddings and resource embeddings in different modalities, calculate the recommendation preference scores of each multimodal educational resource relative to the user to be recommended, and perform personalized recommendation

[0074] It should be noted that in the embodiments of the present invention, for the user to be recommended, the specific calculation method of fusing the user embeddings and resource embeddings in different modalities and calculating the recommendation preference scores of each multimodal educational resource relative to the user to be recommended is as follows: Multiply the first user embedding of the user to be recommended by the transposed result of the first resource embedding of the multimodal educational resource to be recommended to obtain the first score value ; Then, for each resource modality m, multiply the second user embedding of the user to be recommended by the transposed result of the second resource embedding of the multi-modal educational resource to be recommended to obtain the second score value for each resource modality ; Finally, sum the first score value and the second score values of all resource modalities as the recommendation preference score of the multi-modal educational resource to be recommended relative to the user to be recommended, which is expressed by the formula:

[0075]

[0076] In the formula: represents the set of all resource modalities. In this embodiment is the set of 4 resource modalities: text, video, image, and audio.

[0077] In addition, it should be noted that before the above dual collaborative hypergraph neural network is actually used to perform the educational resource recommendation task, it needs to be optimized and trained in advance on the training dataset based on the total loss function. The total loss function is the weighted sum of the BPR loss and the hypergraph reconstruction loss .

[0078] The above dual collaborative hypergraph BPR loss is obtained by weighted summing the BPR loss of the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) and the BPR losses

[0079] of the modality collaborative signal hypergraph convolutional neural networks (MSHNNs) of all resource modalities. The BPR loss function (Bayesian Personalized Ranking loss) is a loss function used to learn the user's personalized preferences in the recommendation system, and its specific calculation formula belongs to the prior art and will not be elaborated here.

[0080] In addition, when the dual collaborative hypergraph neural network is used in the actual recommendation process, it can also collect the feedback data of student users and construct an incremental learning dataset, and regularly use the incremental learning dataset to optimize the network parameters through incremental learning. For example, it can monitor the learning behavior of students in real time to dynamically update the matching scores and recommendation lists, and introduce a feedback mechanism that allows students to rate the recommendation results or mark "not interested", so as to further feedback and optimize the model performance.

[0081] In summary, the present invention provides an educational resource recommendation method under modality missing based on a dual collaborative hypergraph. This method realizes robust recommendation under a high missing rate through feature completion driven by modality confidence, dual-channel modeling of user-resource collaborative hypergraph and modality collaborative hypergraph, and joint optimization of multi-level loss functions. The present invention is particularly applicable to personalized recommendation in scenarios where there are modality missing in multi-modal educational resources (such as text, video, audio, etc.), and can significantly improve the learning experience and resource utilization rate of students.

[0082] Similarly, based on the same inventive concept, as Figure 3 shown, in another preferred embodiment of the present invention, there is also provided an educational resource recommendation system under modality missing based on a dual collaborative hypergraph, which includes:

[0083] A historical information recording module, which is used to record the behavior data of all users on the online learning platform and all multi-modal educational resources uploaded;

[0084] A recommended object selection module, which is used for users to select the user to be recommended;

[0085] An educational resource recommendation module, which is used to execute educational resource recommendation for the user-selected user to be recommended according to the information recorded in the historical information recording module and according to the educational resource recommendation method under modality missing based on the dual collaborative hypergraph described in S1~S4 in the above embodiment.

[0086] It should be noted that the above historical information recording module can be built in the background server of the online learning platform to record the historical learning records of student users in real time. The recommended object selection module can be located on the front-end interface of the online learning platform and provide the specified function through buttons and other means. Of course, the recommended object can also be default set in the recommended object specification module. If the management user does not modify the default setting, the recommendation will be made according to the default setting. Each student user on the platform can be used as the recommended object. The recommendation method in the educational resource recommendation module can be implemented through a fixed module or display logic in the user interface, or through system messages, pop-ups, or forms such as emails and text messages.

[0087] It should be further noted that the method for recommending educational resources under modality loss based on the dual collaborative hypergraph described in S1 to S4 in the above embodiments can essentially be executed by a computer program or module.

[0088] Therefore, similarly, based on the same inventive concept, in another preferred embodiment of the present invention, there is also provided a computer program product corresponding to the method for recommending educational resources under modality loss based on the dual collaborative hypergraph provided in the above embodiments, including a computer program / instructions, which when executed by a processor, can implement the method for recommending educational resources under modality loss based on the dual collaborative hypergraph in the above embodiments.

[0089] Similarly, based on the same inventive concept, as Figure 4 shown, in another preferred embodiment of the present invention, there is also provided a computer electronic device corresponding to the method for recommending educational resources under modality loss based on the dual collaborative hypergraph provided in the above embodiments, which includes a memory and a processor;

[0090] The memory is used to store a computer program;

[0091] The processor is used to implement the method for recommending educational resources under modality loss based on the dual collaborative hypergraph in the above embodiments when executing the computer program.

[0092] In addition, when the logical instructions in the above memory are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0093] Thus, based on the same inventive concept, in another preferred embodiment of the present invention, there is also provided a computer-readable storage medium corresponding to the method for recommending educational resources under modality loss based on a dual collaborative hypergraph provided in the above embodiments. The storage medium stores a computer program, which when executed by a processor, can implement the method for recommending educational resources under modality loss based on the dual collaborative hypergraph in the above embodiments.

[0094] It can be understood that the above storage medium may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Meanwhile, the storage medium may also be various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc.

[0095] It can be understood that the above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0096] In addition, it should be noted that those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be elaborated herein. In the various embodiments provided in the present application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0097] Next, a specific embodiment will be used to demonstrate the specific implementation manner and technical effects of the educational resource recommendation method based on the dual collaborative hypergraph in the case of missing modalities described in S1~S4 above.

[0098] Embodiment

[0099] In this embodiment, the implementation manner of the educational resource recommendation method based on the dual collaborative hypergraph in the case of missing modalities described in S1~S4 above is specifically demonstrated, and its process can be seen Figure 5 as shown, which is divided into four parts: data input and preprocessing, multi-modal learning resource completion, dual collaborative hypergraph neural network, and multi-modal educational resource recommendation. The specific implementation of the four parts in this process will be described in detail below.

[0100] 1. Data Input and Preprocessing

[0101] First, collect the behavioral data of all student users on the online learning platform (including two types of interaction information: learning records of redundant multimodal educational resources and interaction scores), and the original multimodal features of the multimodal educational resources (including four resource modalities: text, video, image, and audio). Then, construct a student-resource interaction graph with all users and all multimodal educational resources as two types of nodes, and establish an adjacency matrix based on the interaction information of student users. Record the user nodes and the edge connection relationships between the resource nodes, where is the number of users in the student user set and is the number of multimodal educational resources in the educational resource set. Finally, for each multimodal educational resource, use ResNet to capture the image and video modalities, BERT to capture the text modality, and MFCC to capture the audio modality to obtain the modality feature matrix of each resource modality of all multimodal educational resources, where is the modality feature corresponding to the resource modality

[0102] 2. Multimodal Learning Resource Completion

[0103] 2.1) Modal Confidence Calculation

[0104] First, construct a resource co-occurrence graph with all multimodal educational resources as nodes, and represent the co-occurrence relationship between multimodal educational resources (such as the course selection association between courses) through the second adjacency matrix of the resource co-occurrence graph. Then, for each resource modality, divide the resource nodes in the resource co-occurrence graph into missing nodes and complete nodes according to whether there is modality missing. Finally, calculate the modal confidence (MC) between all pairs of resource nodes in the resource co-occurrence graph:

[0105]

[0106] where is the shortest path distance from the resource node to the nearest complete node d in the resource co-occurrence graph. If the resource node itself is a complete node, then the corresponding is is a hyperparameter used as the base of the power operation to control the feature propagation intensity.

[0107] 2.2) Feature Completion​

[0108] First, based on the modal confidence and the multimodal features of each complete node, the features of the missing nodes are restored in a way of in-channel diffusion.

[0109] Specifically, for any two resource nodes in the resource co-occurrence graph, the ratio of their modal confidences is used as the relative confidence of these two resource nodes , and according to the relative confidence, the second adjacency matrix A is converted into a weighted adjacency matrix. The constructed weighted adjacency matrix is used to assign a weight to each edge in the resource co-occurrence graph, that is, to define the edge weights. The element value at the coordinate position in the weighted adjacency matrix can be determined according to the element value at the coordinate position in the second adjacency matrix A according to the following rules: Then, to ensure the convergence of the diffusion process, the weighted adjacency matrix needs to be randomly normalized

[0110]

[0111] to obtain the normalized weighted adjacency matrix : :

[0112]

[0113] where: is the degree matrix of the weighted adjacency matrix .

[0114] Finally, combined with the graph diffusion algorithm, node-to-node diffusion propagation is carried out for each channel of the resource co-occurrence graph. The formula for a single diffusion is as follows:

[0115]

[0116] where and are the modal features of the resource nodes in the resource co-occurrence graph restored after t-step and (t - 1)-step diffusion propagation respectively . It should be noted that the node-to-node diffusion propagation here is calculated by all the resource nodes in all the resource co-occurrence graphs participating in the propagation diffusion together, rather than calculating each single node separately.

[0117] The above node-to-node diffusion propagation needs to be iterated multiple times. After reaching the predetermined number of iterative diffusions T, the initial restored features of all resource nodes in the current resource modality are obtained .

[0118] 2.3) Channel Correlation Optimization

[0119] Since the dependence between channels may be another important factor for missing node features in the input, this embodiment designs an additional scheme to optimize , that is, feature refinement by considering channel correlation and MC. At this stage, for a node internally, it is necessary to refine a low-MC channel feature with a high-MC channel feature according to the correlation degree between two feature channels. That is to say, on the basis of the initial restored features, it is also necessary to propagate and refine the node features between channels to further optimize the node features. The specific method is as follows:

[0120] First, calculate the feature correlation coefficient between any two feature channels. Define the feature correlation coefficient matrix , where is the modal feature dimension, giving the correlation coefficient between each pair of channels. Among them, for any two channels corresponding to the initial restored feature and , the feature of channel and the feature of channel The feature correlation coefficient The calculation formula is:

[0121]

[0122] In the formula: is the eigenvalue of the resource node i in channel , is the eigenvalue of the resource node i in channel , and are the feature mean and feature variance in feature respectively, and are the feature mean and feature variance in feature respectively, and N is the total number of resource nodes.

[0123] Then, this embodiment designs a weighted adjacency matrix for refining the feature of the missing node i in the initial restored feature . For inter-channel propagation of each node, where represents the element value at the coordinate position in the weighted adjacency matrix , representing The weight of the directed edge from channel to channel in, and its calculation formula is as follows:

[0124]

[0125] Among them are all hyperparameters, which can be set in this embodiment , . is a scaling hyperparameter, which is used to adjust the intensity of message passing according to MC and is set to 1 in this embodiment

[0126] Finally, the features of the missing node i are propagated between channels, and the modal features of the missing node i in the current resource modality are output as:

[0127]

[0128] The modal features of each missing node i in all resource modalities can form the multi-modal features of this missing node i. Integrating them into the student-resource interaction graph can complete the missing modal features in the missing nodes, while the multi-modal features of the complete nodes in the student-resource interaction graph remain unchanged. Thus, a student-resource interaction graph with complete feature modalities can be formed and input into the dual collaborative hypergraph neural network for subsequent calculations. The node features mentioned in the subsequent dual collaborative hypergraph neural network are all the features completed here and will not be specifically described

[0129] 3. Dual Collaborative Hypergraph Neural Network

[0130] For each resource modality, a user hypergraph and a modality hypergraph within the resource modality are respectively constructed, and a dual collaborative hypergraph neural network for dual collaborative hypergraph convolution is constructed. The dual collaborative hypergraph neural network includes two main collaborative modules: the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) and the modality collaborative signal hypergraph convolutional neural network (MSHNN). The CSHNN module is responsible for propagating collaborative signals between users and resources to capture the correlation between user preferences and resource features; while the MSHNN module performs collaborative signal propagation within each modality to achieve intra-modal fusion and completion of features, and outputs user embeddings and resource embeddings in different modalities. Through the dual role of collaborative hypergraph convolution, these two modules enable the model to more precisely understand the interaction relationship and potential semantics of multi-modal information

[0131] 3.1) Modality Hypergraph Construction

[0132] First, for each resource modality m in the student-resource interaction graph, a membership matrix is generated based on fuzzy c-means (FCM), where represents the number of clusters, i.e., the number of hyperedges, Denotes the number of resources. Samples with high membership degrees have higher weights in the clustering clusters, thus playing a more significant role in expressing global dependencies in hypergraph construction.

[0133] Then, generate the hypergraph incidence matrix of the resources . The set of resources in each clustering cluster forms a hyperedge , and its resource nodes are samples whose membership degrees exceed the threshold . In this process, a sample can belong to multiple clustering clusters due to the fuzziness of its membership degree, thus connecting multiple hyperedges. According to the membership matrix and the membership threshold, construct the hypergraph incidence matrix of the resources through 0-1 binarization. The element corresponding to the hyperedge e and the node i in is:

[0134]

[0135] Finally, according to the hypergraph incidence matrix of the resources , calculate the hypergraph incidence matrix of the users For each resource modality , respectively and independently construct the user hypergraph of the current resource modality m and the modality hypergraph based on the hypergraph incidence matrix of the resources and the hypergraph incidence matrix of the users , which respectively represent the intra-modal associations of users and resources. Through this hypergraph construction method based on FCM, the model can more effectively maintain the diversity of sample features in the case of missing modalities, while capturing the global dependencies between modalities, thereby improving the robustness and accuracy of the recommendation results.

[0136] 3.2) User-Resource Collaborative Signal Hypergraph Convolutional Neural Network (CSHNN)

[0137] First, define the collaborative hypergraph convolution through the user-resource interaction graph, that is, use the student-resource interaction graph as the hypergraph input to the user-resource collaborative signal hypergraph convolutional neural network (CSHNN) stacked by multiple first hypergraph convolutional layers, and propagate the collaborative signal layer by layer between users and resources. Hypergraph convolution enables each user or resource to share and propagate information with multiple associated nodes (such as other users and resources) through hyperedges, thereby capturing the collaborative relationship between user preferences and resource features. This collaborative relationship can help the model better integrate the preference information from multiple modalities, thus improving the recommendation quality.

[0138] In the traditional graph convolution process, the features of user nodes are aggregated according to the following formula:

[0139]

[0140] Among them is the user node at layer node representation, is the node representation of resource node i at layer.

[0141] However, stacking multiple hypergraph convolutional layers increases the possibility of gradient vanishing and numerical instability. Therefore 's scale may change. To solve this problem, this embodiment normalizes it, that is, during the propagation of the first hypergraph convolutional layer, the normalized graph convolution method is adopted to propagate the node features of the resource nodes to the user nodes, specifically according to the following formula:

[0142]

[0143] Among them and are respectively the degree matrices of nodes and hyperedges in the user-resource interaction graph, is the node representation of user node at layer, is the node representation of resource node i at layer. Among them, when , is the aggregated feature of the resources interacted by user , , represents the multimodal features of all resource nodes in the user-resource interaction graph, is the node feature corresponding to resource node i .

[0144] Finally, similarly, it is also necessary to propagate the node features of the user nodes to the resource nodes, and the node representation of resource node at layer can be obtained:

[0145]

[0146] Assume that the first hypergraph convolutional layer has a total of L layers. Then the last layer of the hypergraph convolutional layer outputs the first user embedding of each user node and the first resource embedding of each resource node.

[0147] It can be seen that the CSHNN weighted-aggregates the preference signals between users and resources, effectively enhancing the expressive ability of the embeddings, enabling users and resources to maintain the integrity of collaborative information even in the case of modality absence, and enhancing the robustness of recommendations.

[0148] 3.3) Multimodal Collaborative Signal Hypergraph Convolutional Neural Network (MSHNN)

[0149] In the MSHNN module, the feature collaboration within the same modality is enhanced through multimodal collaborative signals. In the multimodal recommendation task, there may also be collaborative signals among the information of each modality within the same type of nodes. To effectively mine this signal, multiple user hypergraphs within different modalities and modality hypergraphs are used to achieve feature enhancement and information diffusion within each modality.

[0150] The features of users and resources within different modalities are propagated and complemented, enabling each modality to achieve self-update of information in an independent hypergraph structure. Using MSHNN ensures the efficient propagation of the collaborative signals within each modality, thus guaranteeing the information completeness of that modality even in the case of modality absence.

[0151] Specifically, the user hypergraph and the modality hypergraph are input into a Multimodal Collaborative Signal Hypergraph Convolutional Neural Network (MSHNN) stacked by multiple second hypergraph convolutional layers, and the collaborative signal propagation is carried out layer by layer within each resource modality. During the collaborative signal propagation of each resource modality in the second hypergraph convolutional layer, the normalized graph convolution method is adopted. The normalized graph convolution process of MSHNN is as follows:

[0152] For the user node u, the features of all user nodes under the hyperedge e to which the user node u belongs in the user hypergraph are aggregated to its own node, which is expressed by the formula:

[0153]

[0154] where and are the degree matrices of the node and the hyperedge in the user hypergraph respectively, and are the node representations of the user node u in the resource modality m at the layer and the layer respectively, is the node representation of the hyperedge e in the resource modality m at the layer and the layer.

[0155] For resource node i, aggregate the features of all resource nodes under the hyperedge e to which resource node i belongs in the resource hypergraph into its own node, which is expressed by the formula:

[0156]

[0157] Where and are the degree matrices of node and hyperedge in the resource hypergraph respectively, and and are the node representations of resource node i in the resource modality m at layer and layer respectively.

[0158] For hyperedge e, aggregate the features of all user nodes under the hyperedge e to which user node u belongs in the user hypergraph into its own node to obtain the user hyperedge aggregation feature, and then aggregate the features of all resource nodes under the hyperedge e to which resource node i belongs in the resource hypergraph into its own node to obtain the resource hyperedge aggregation feature, and then superimpose the user hyperedge aggregation feature and the resource hyperedge aggregation feature to obtain the node representation of hyperedge which is expressed by the formula:

[0159]

[0160] Where, is the node representation of hyperedge in the resource modality m at layer is the average feature of all resource nodes included in hyperedge in the resource modality m.

[0161] Similarly, assuming that the second hypergraph convolutional layer has a total of L layers, the last layer of the hypergraph convolutional layer outputs the second user embedding of each user node u and the second resource embedding of each resource node i in each resource modality m.

[0162] 4. Multi-modal Educational Resource Recommendation

[0163] 4.1) Multi-modal Feature Fusion

[0164] To achieve more accurate recommendations in a multi-modal scenario, a prediction method for multi-modal feature fusion is designed. By fusing the user embeddings and resource embeddings in different modalities under the collaborative signal, for any user node u and resource node i, the final recommended preference score is calculated​ is:

[0165]

[0166] wherein, and respectively represent the embeddings of the user and the resource in the CSHNN, reflecting the direct collaborative matching degree between the user and the resource under the global collaborative relationship. represents the collaborative score between the user and the resource within each modality, representing the degree of relevance between the user and the resource in that modality. Among them and are respectively the embeddings in the MSHNN under the modality . By calculating the matching degree between the user and the resource under each modality and summing them up, the present invention can capture the independent collaborative features in each modality, make up for the deficiency of single-modal information, and avoid the influence of redundant information between modalities.

[0167] 4.2) Loss function optimization

[0168] The total loss function in the present invention adopts a multi-level loss function, including the BPR loss and the hypergraph reconstruction loss . The dual collaborative hypergraph neural network is pre-optimized and trained on the training dataset based on the total loss function. The specific training method belongs to the prior art and can be iteratively optimized by an optimizer based on the purpose of minimizing the total loss function, which will not be elaborated here. The following focuses on the specific calculation method of the total loss function.

[0169] First of all, the BPR loss aims to maximize the gap between the predicted scores of positive samples and negative samples, so as to improve the ranking performance of recommendations,

[0170]

[0171] wherein, represents the loss of the CSHNN, represents the loss of the MSHNN of the modality .

[0172] In order to better characterize the relationship between the user and the resource under each modality, this embodiment designs a hypergraph reconstruction loss for reconstructing the relationship between the user and the resource in the hypergraph. The core idea of this loss is to minimize the binary cross-entropy loss (BCE) between the user, resource embeddings and the true hypergraph incidence matrix, so that the model can learn more discriminative embedding representations.

[0173] Then, given the user embedding matrix and the resource embedding matrix , the hyperedge embedding matrix , for the user and resource hypergraphs of each modality, calculate the predicted user-hyperedge association matrix and the resource-hyperedge association matrix :

[0174]

[0175]

[0176] Among them, represents the Sigmoid function, which is used to compress the association value into the interval [0,1] to adapt to the calculation of the BCE loss.

[0177] Secondly, the hypergraph reconstruction loss calculates the difference between the predicted user-hyperedge and resource-hyperedge association matrices and the true hypergraph association matrix. The decoding loss is calculated as follows:

[0178]

[0179]

[0180]

[0181] Finally, the total loss function of the model includes the above BPR loss and hypergraph reconstruction loss, where the weight of the hypergraph reconstruction loss is controlled by the hyperparameter , and the final total loss function is expressed as follows:

[0182]

[0183] 4.3) Recommendation result generation

[0184] First, for each student user , according to the matching score , sort all resources in descending order to generate a candidate recommendation list. Then, according to the platform requirements, filter the top resources as the final recommendation results. For example, represents recommending the top 10 educational resources.

[0185] Then, to enhance the interpretability of the recommendation results, prompt information can be generated by combining modal features. For example, if the recommended resource contains high-confidence modal features (text descriptions of high-scoring courses), prompt "Recommend the text lecture notes of high-scoring courses". The prompt information can help users understand the basis of the recommendation and make selections according to their personal needs, thereby enhancing the learning experience.

[0186] Finally, the matching scores and recommendation lists are updated in real time by monitoring the learning behaviors of students, and a feedback mechanism is introduced to allow students to rate the recommendation results or mark "not interested", further optimizing the model performance.

[0187] To verify the effectiveness of the present invention, an actual educational resource dataset is selected for experiments in this embodiment. This dataset is collected from a real large-scale online learning platform and contains students' learning records and multimodal learning resources. The initial learning state of each student is initialized through historical learning data, and their performance in the subsequent learning process will continuously update their representation. Through the above process, the evaluation metric for the experiment is Recall@20, and the larger the value, the better the recommendation effect of the model.

[0188] Table 1 Concept recommendation results of the method of the present invention on an educational dataset with 30% missing attributes

[0189]

[0190] Table 2 Concept recommendation results of the method of the present invention on an educational dataset with 50% missing attributes

[0191]

[0192] Table 3 Concept recommendation results of the method of the present invention on an educational dataset with 70% missing attributes

[0193]

[0194] As can be seen from the experimental results, the method proposed in the present invention has achieved significant improvements in the Recall@20 evaluation index under different attribute missing rates. The above-mentioned comparison methods are as follows: MMGCN is from the prior art literature: Wei Y, Wang X, Nie L, et al. MMGCN: Multi-modal graph convolution network for personalized recommendation of microvideo[C] / / Proceedings of the 27th ACM international conference on multimedia. 2019: 1437-1445. SLMRec is from the prior art literature: Tao Z, Liu X, Xia Y, et al. Self-supervised learning for multimedia recommendation[J]. IEEE Transactions on Multimedia, 2022, 25: 5107- 5116. CI2MG is from the prior art literature: Lin Z, Tan Y, Zhan Y, et al. Contrastive intra- and intermodality generation for enhancing incomplete multimedia recommendation[C] / / Proceedings of the 31st ACM International Conference on Multimedia. 2023: 6234-6242.

[0195] In addition, an ablation study was also conducted to better examine the contribution of the innovations proposed in the present invention. The experimental results are shown in the following table, and the specific details of each module are as follows:

[0196] w / o MC: Remove the modality completion module based on modality confidence to examine its role in the case of missing learning resource modalities.

[0197] w / o CSHNN: Remove the user-resource collaborative hypergraph convolutional neural network to examine whether the high-order information between users and resources plays a key role in improving embedding stability and recommendation performance.

[0198] w / o MSHNN: Remove the modality collaborative hypergraph convolutional neural network to examine whether the high-order intra-modality information of users and resources plays a key role in improving embedding stability and recommendation performance.

[0199] Table 4 Ablation experiment results of the method of the present invention with 30% attribute missing

[0200]

[0201] Table 5 Ablation experiment results of the method of the present invention with 30% attribute missing

[0202]

[0203] Table 6 Ablation experiment results of the method of the present invention with 30% attribute missing

[0204]

[0205] The ablation experiment results show that the innovative modules proposed in the present invention all have significant effects. The modality completion module based on modality confidence and the dual collaborative hypergraph neural network are crucial for improving the performance of the model. Among them, the user-resource collaborative hypergraph convolutional neural network has the greatest impact on the results.

[0206] The above-described embodiments are only some preferred implementation solutions of the present invention, but are not intended to limit the present invention. Those of ordinary skill in the relevant technical field can still make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by adopting the means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for recommending educational resources in the absence of modality based on dual collaborative hypergraph, characterized in that: include: S1. Obtain the behavior data of all users and all multimodal educational resources on the online learning platform, construct a user-resource interaction graph by treating all users and all multimodal educational resources as two types of nodes, and establish a first adjacency matrix based on the interaction information in the behavior data to record the edge connection relationship between user nodes and resource nodes, and obtain multimodal features of each multimodal educational resource through multimodal encoding; S2. Construct a resource co-occurrence graph by taking all multimodal educational resources as nodes, and record the co-occurrence relationship between multimodal educational resources through a second adjacency matrix; for each resource modality, divide the resource nodes in the resource co-occurrence graph into missing nodes and complete nodes according to whether there is modality missing, and calculate the modal confidence between the resource nodes. Based on the modal confidence and the multimodal features of the complete nodes, the modal features of the missing nodes under the current resource modality are completed by first diffusing the node features within the channel and then propagating the refined node features between channels; For each resource node in the resource co-occurrence graph, its modal confidence is obtained by converting the shortest path distance from the resource node to the nearest complete node through a power operation with a base range of (0,1); S3. For each resource modality, a user hypergraph and a modality hypergraph are constructed within the resource modality respectively. Based on the dual collaborative hypergraph convolution within the dual collaborative hypergraph neural network, on the one hand, the user-resource collaborative signal hypergraph convolution neural network propagates collaborative signals between users and resources to capture the correlation between user preferences and resource features. On the other hand, the modality collaborative signal hypergraph convolution neural network propagates collaborative signals within each resource modality to achieve intra-modal fusion and completion of features, and outputs user embedding and resource embedding under different modalities. S4. For the user to be recommended, the user embedding and resource embedding under different modalities are integrated, the recommendation preference score of each multimodal educational resource relative to the user to be recommended is calculated, and personalized recommendations are made.

2. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 1, characterized in that: The multimodal educational resources include multiple learning resource modalities such as text, video, image and audio. When performing the multimodal encoding, ResNet is used to encode the image modality and video modality, BERT is used to encode the text modality, and Mel cepstral coefficients are used to encode the audio modality.

3. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 1, characterized in that: In S2, for each resource modality, the method for completing the modality features of the missing nodes under the current resource modality is: S21. For each resource node in the resource co-occurrence graph, determine the shortest path distance from the resource node to the nearest complete node, and convert the shortest path distance into a modal confidence by a power operation with a base range of (0,1); S22. For any two resource nodes in the resource co-occurrence graph, the ratio of their modal confidences is used as the relative confidence of the two resource nodes, and the second adjacency matrix is ​​converted into a weighted adjacency matrix according to the relative confidence, wherein all the values ​​on the diagonal of the second adjacency matrix are set to 1, and the elements with other values ​​of 0 remain unchanged, and the elements with other values ​​of 1 are set to the relative confidence of the two resource nodes corresponding to the element; the weighted adjacency matrix is ​​subjected to random normalization operation to obtain the normalized weighted adjacency matrix, and then the resource co-occurrence graph is diffused between nodes channel by channel in combination with the graph diffusion algorithm, and the initial recovery features of all resource nodes under the current resource modality are obtained after multiple iterations of diffusion; S23. Based on the initial recovery features, calculate the feature correlation coefficient between any two feature channels, and then for each resource node in the resource co-occurrence graph, convert the corresponding feature correlation coefficient to obtain the directed edge weight from one feature channel to another, and record the directed edge weights between the feature channels in the weighted adjacency matrix; use the weighted adjacency matrix to perform inter-channel propagation on each resource node in the resource co-occurrence graph to obtain the modal features of the missing node under the current resource modality.

4. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 1, characterized in that: In S3, for each resource modality, the specific steps of calculating user embedding and resource embedding are as follows: S31. For the current resource modality in the student-resource interaction graph, a membership matrix is ​​generated based on fuzzy clustering and the membership matrix is ​​binarized from 0 to 1 using a membership threshold to obtain a hypergraph association matrix of resources. The first adjacency matrix is ​​then multiplied by the hypergraph association matrix of resources to obtain a hypergraph association matrix of users. Finally, a user hypergraph and a modality hypergraph of the current resource modality are constructed based on the two hypergraph association matrices. S32, using the student-resource interaction graph as a hypergraph input into a user-resource collaborative signal hypergraph convolutional neural network formed by stacking multiple first hypergraph convolutional layers, propagating collaborative signals between users and resources layer by layer; in the propagation process of the first hypergraph convolutional layer, using a normalized graph convolution method to propagate the node features of resource nodes to user nodes, and at the same time propagate the node features of user nodes to resource nodes; the last layer of hypergraph convolutional layer outputs the first user embedding of each user node and the first resource embedding of each resource node; S33. Input the user hypergraph and the modal hypergraph into the modal collaborative signal hypergraph convolutional neural network formed by stacking multiple second hypergraph convolutional layers, and perform collaborative signal propagation within each resource modality layer by layer; in the collaborative signal propagation process of each resource modality in the second hypergraph convolutional layer, the normalized graph convolution method is adopted. For each user node, it is necessary to aggregate all user node features under its hyperedge in the user hypergraph to its own node. For each resource node, it is necessary to aggregate all resource node features under its hyperedge in the resource hypergraph to its own node. For each hyperedge, it is necessary to first aggregate all user node features under the hyperedge of the user node in the user hypergraph to the current user node itself to obtain the user hyperedge aggregation feature, and then aggregate all resource features under the hyperedge of the resource node in the resource hypergraph to the current resource node itself to obtain the resource hyperedge aggregation feature, and then superimpose the user hyperedge aggregation feature and the resource hyperedge aggregation feature and update the node feature of the hyperedge; the last layer of the second hypergraph convolutional layer outputs the second user embedding of each user node and the second resource embedding of each resource node under each resource modality m.

5. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 1, characterized in that: In S4, the recommendation preference score of the to-be-recommended user and each to-be-recommended multimodal educational resource is calculated by multiplying the transposed result of the first user embedding of the to-be-recommended user and the first resource embedding of the to-be-recommended multimodal educational resource to obtain a first score value; Then, for each resource modality, the second user embedding of the user to be recommended is multiplied by the transposed result of the second resource embedding of the multimodal educational resource to be recommended to obtain the second score value of each resource modality; finally, the first score value is summed with the second score values ​​of all resource modalities as the recommendation preference score of the multimodal educational resource to be recommended relative to the user to be recommended.

6. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 1, characterized in that: The dual collaborative hypergraph neural network is optimized and trained in advance on a training data set based on a total loss function; The total loss function is the weighted sum of BPR loss and hypergraph reconstruction loss; The dual collaborative hypergraph BPR loss is obtained by weighted summing the BPR loss of the user-resource collaborative signal hypergraph convolutional neural network and the BPR loss of the modal collaborative signal hypergraph convolutional neural network of all resource modalities; The hypergraph reconstruction loss is obtained by summing the hypergraph association matrix reconstruction losses corresponding to the user hypergraph and the resource hypergraph under all resource modes.

7. The method for recommending educational resources in the absence of modality based on dual collaborative hypergraph as claimed in claim 6, characterized in that: When the dual collaborative hypergraph neural network is used in the actual recommendation process, it is necessary to collect feedback data from student users and construct an incremental learning data set, and regularly use the incremental learning data set to perform incremental learning optimization on network parameters.

8. A dual collaborative hypergraph-based educational resource recommendation system under modality loss, characterized in that: include: The historical information recording module is used to record the behavior data of all users on the online learning platform and all uploaded multimodal educational resources; A recommendation object selection module is used for allowing users to select users to be recommended; An educational resource recommendation module is used to perform educational resource recommendations for users selected by users according to the information recorded in the historical information recording module and in accordance with the educational resource recommendation method under modality loss based on dual collaborative hypergraph as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for recommending educational resources in the absence of modality based on a dual collaborative hypergraph as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the method for recommending educational resources in the absence of modality based on a dual collaborative hypergraph as described in any one of claims 1 to 7 when executing the computer program.

Images