Knowledge recommendation method and system based on cross-modal comparison and Riemannian hyperbolic geometry

Abstract

The invention relates to the technical field of knowledge recommendation, and discloses a knowledge recommendation method and system based on cross-modal comparison and Riemannian hyperbolic geometry, and the method comprises the steps: obtaining a user and project interaction bipartite graph; obtaining a basic knowledge graph; combining the user and project interaction bipartite graph with the basic knowledge graph to obtain a collaborative knowledge graph; multi-modal features are set for each item in the collaborative knowledge graph, and the multi-modal features comprise visual modal features and text modal features; inputting the collaborative knowledge graph and the multi-modal features of the items into the trained knowledge recommendation model, and recommending the items to the user through the trained knowledge recommendation model; according to the trained knowledge recommendation model, potential information among different modals is further mined by using cross-modal comparative learning, and an extra supervision signal is introduced for a recommendation task. Movie items or game items that the user is interested in can be recommended to the user.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge recommendation, and in particular to a knowledge recommendation method and system based on cross-modal comparison and Riemannian hyperbolic geometry. Background Art

[0002] With the advent of the era of big data and the Internet, the explosive growth of information has brought about a great abundance of data resources, but it has also made the process of obtaining, filtering and processing information increasingly complex. While the rapid development of the Internet has brought convenience to people, it has also caused problems such as information redundancy and overload. In this context, how to efficiently filter content that meets the needs of users from massive data has become an urgent problem to be solved in Internet services. As an important tool to alleviate the problem of information overload, the recommendation system aims to analyze users' interests and behaviors, push relevant information or products to them in a personalized manner, and improve user experience and platform service quality. However, due to problems such as data sparsity and cold start, traditional content-based or collaborative filtering recommendation methods have gradually exposed their inherent limitations when faced with complex and diverse user behaviors and needs.

[0003] In order to cope with the above difficulties and challenges, in recent years, related methods based on auxiliary information enhancement such as multimodality or knowledge graph (KG) have become a hot area of ​​recommendation system research. Multimodal recommendation provides a new perspective for user interest modeling and project feature characterization by introducing heterogeneous information such as images, texts, audio and videos, thereby improving the personalized recommendation and context perception capabilities of the recommendation system. Its typical process is to first extract multimodal features through a pre-trained deep neural network, and then fuse the multimodal features with the interactive features of the project as the multimodal representation of the project. As another key auxiliary information in the recommendation system, knowledge-aware recommendation dominated by knowledge graphs has also attracted widespread attention. Knowledge graphs are heterogeneous graphs, usually composed of a large number of triples, which provide various facts related to projects in the real world in the form of <head entity, relationship, tail entity>. By structuring and associating project and user information, the recommendation system can deeply understand the relationship between projects and user preferences, further improving the accuracy of recommendation results. However, most traditional knowledge-aware recommendation methods embed nodes in Euclidean space, which cannot capture important hierarchical relationships in knowledge graphs. Recently, models based on Riemann hyperbolic space have made up for the problem of insufficient embedding ability of Euclidean space when processing large-scale graphs. This type of method proposes to embed representations with hierarchical relationships into the Poincaré Ball model and Lorentz model in hyperbolic space, which shows superior performance compared to traditional Euclidean space methods.

[0004] Although the above-mentioned auxiliary information enhanced recommendation method has shown great potential, the inventors have found that due to the limitations of research field barriers and data construction, there is currently a lack of effective fusion between visual and textual multimodal information and knowledge graphs in the film or game fields. Recommendation algorithms based only on single auxiliary information often cannot fully explore the potential representations in the auxiliary information and cannot accurately recommend movies or game projects of interest to users. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a knowledge recommendation method and system based on cross-modal contrast and Riemann hyperbolic geometry; the method first builds multimodal information and knowledge graphs of project-side links based on interactions and project information in online communities, combined with technologies such as entity recognition, relationship extraction, and web crawlers. Then, a knowledge-aware aggregation recommendation model based on a multimodal perspective is proposed. The model combines cross-modal contrast learning with a graph convolutional neural network (GCN) in Riemann hyperbolic space, and uses a user co-occurrence graph of virtual relationship perception to enhance user representation, thereby achieving effective fusion of heterogeneous auxiliary information.

[0006] On the one hand, a knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry is provided, including: Obtain a bipartite graph of user-item interactions; obtain a basic knowledge graph; combine the bipartite graph of user-item interactions with the basic knowledge graph to obtain a collaborative knowledge graph; set multimodal features for each item in the collaborative knowledge graph, the multimodal features including: visual modal features and textual modal features; The multimodal features of the collaborative knowledge graph and the project are input into the trained knowledge recommendation model, and the project is recommended to the user through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal contrastive learning to further mine the potential information between different modalities, introducing additional supervisory signals for the recommendation task.

[0007] On the other hand, a knowledge recommendation system based on cross-modal comparison and Riemannian hyperbolic geometry is provided, including: An acquisition module is configured to: acquire a bipartite graph of user-item interactions; acquire a basic knowledge graph; combine the bipartite graph of user-item interactions with the basic knowledge graph to obtain a collaborative knowledge graph; set multimodal features for each item in the collaborative knowledge graph, the multimodal features including: visual modal features and textual modal features; The recommendation module is configured to: input the multimodal features of the collaborative knowledge graph and the project into the trained knowledge recommendation model, and recommend projects to users through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal contrastive learning to further explore the potential information between different modalities, introducing additional supervisory signals for the recommendation task.

[0008] The above technical solution has the following advantages or beneficial effects: (1) The present invention constructs a collaborative knowledge graph (CKG) based on the historical interaction information between users and projects in the online community. The CKG stores the complex interaction patterns between users and projects, and uses web crawler technology to construct visual and textual modal information of project-side links. The CKG can recommend movie projects or game projects that the users are interested in.

[0009] (2) This paper uses knowledge graphs and multimodal features as auxiliary information to learn the representation of nodes in hyperbolic space from the perspective of different modalities, and then further enhances user representation through user co-occurrence graphs; (3) This paper proposes to use cross-modal contrastive learning to further explore the potential information between different modalities, introducing additional supervisory signals for the recommendation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0011] Figure 1 This is a method flow chart of Embodiment 1 of the present disclosure. DETAILED DESCRIPTION

[0012] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0013] Embodiment 1, as Figure 1 As shown, this embodiment provides a knowledge recommendation method based on cross-modal comparison and Riemann hyperbolic geometry, including: S101: Obtain a bipartite graph of user-item interactions; obtain a basic knowledge graph; combine the bipartite graph of user-item interactions with the basic knowledge graph to obtain a collaborative knowledge graph; set multimodal features for each item in the collaborative knowledge graph, the multimodal features including: visual modal features and textual modal features; S102: Input the collaborative knowledge graph and the multimodal features of the project into the trained knowledge recommendation model, and recommend projects to the user through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal comparative learning to further mine the potential information between different modalities, thereby introducing additional supervisory signals for the recommendation task.

[0014] Furthermore, the trained knowledge recommendation model is designed through the following steps: (1) Initialize each node of the collaborative knowledge graph in Euclidean space to obtain the embedded representation of each node; project the embedded representation of each node from Euclidean space to the hyperbolic space of the Poincare sphere model; (2) Using a hyperbolic relationship-aware graph neural network, the embedded representation of the hyperbolic space node is mapped to the local Euclidean tangent space. After completing the aggregation operation in the tangent space, it is mapped back to the hyperbolic space to obtain the user's knowledge representation and the project's knowledge representation. (3) Based on the user’s knowledge representation and the number of common interactions between users, a user co-occurrence graph is constructed. Based on the user co-occurrence graph, a hyperbolic relationship-aware graph neural network is used again to capture the hierarchical and adjacency information, and the final co-occurrence representation of users and the final representation of items under different modalities are obtained. (4) Conduct cross-modal comparative learning on the final co-occurrence representations of users and the final representations of the items in different modalities to explore the potential information embedded in users and items in different modalities; (5) Based on the final co-occurrence representation of the user under different modalities and the final representation of the item, output the recommendation results.

[0015] Further, the S101: obtains a bipartite graph of user-project interaction, wherein the bipartite graph of user-project interaction includes: user nodes and project nodes, and the project nodes include: games or movies, etc.; if there is an interaction relationship between the user node and the project node, there is an edge connecting the two nodes, and if there is no interaction relationship between the user node and the project node, there is no edge connecting the two nodes.

[0016] For example, if user A has ever purchased game B, there is a mutually connected edge between the user A node and the game B node; otherwise, it means that there is no mutually connected edge between the two.

[0017] Furthermore, the basic knowledge graph is obtained, wherein the basic knowledge graph includes: head entity-relationship-tail entity; for example: game-includes-role, movie-includes-actor.

[0018] Furthermore, the user-project interaction bipartite graph is combined with the basic knowledge graph to obtain a collaborative knowledge graph, including: if a node in the bipartite graph and a node in the basic knowledge graph represent the same concept, the two nodes are merged into one node.

[0019] For example, if the node in the bipartite graph is "Honor of Kings Game" and the node in the basic knowledge graph is also "Honor of Kings Game", then these two nodes are merged into one as one node. In this way, the bipartite graph and the basic knowledge graph can be combined.

[0020] Furthermore, multimodal features are set for each item in the collaborative knowledge graph, including: using a pre-trained ViT model to extract visual modal features and textual modal features of each item.

[0021] For example, respectively and express Users and A collection of items.

[0022] The user-item interaction bipartite graph is defined as ,in Represents an interactive relationship.

[0023] The user-item interaction matrix is ​​defined as ,in Indicates user and Projects Interaction occurs, otherwise .

[0024] The knowledge graph is defined as ,in and Represents a collection of entities and relationships respectively.

[0025] Will and Combine and build a collaborative knowledge graph ,in , .

[0026] In addition to the collaborative knowledge graph, each project is also linked to multimodal information Related, among which and Represent the visual and textual modality features of the item respectively, and define Corresponding mode for the project The characteristics of , For modal feature dimension.

[0027] In this disclosure, only visual and text Two modes.

[0028] The hyperbolic embedding is learned using the Poincare sphere model with constant curvature-c in Riemann hyperbolic space, with The open Poincare sphere of radius d, is defined as: , (1) in, represents the Euclidean norm. When c=1, the present disclosure uses represents the Poincare sphere.

[0029] Due to the lack of feature transformation and mean aggregation operations in hyperbolic space, in order to perform neural network operations in hyperbolic space, the tangent space and hyperbolic space need to be continuously converted, which requires the use of logarithmic mapping and exponential mapping. represent superior Point dimensional local Euclidean space. The exponential map can transform the tangent space Mapping to hyperbolic space , on the contrary, the logarithmic map transforms the hyperbolic space Mapping to tangent space . Both in The definition of the point is as follows: , (2) , (3) in It stands for Möbius Addition, a mathematical operation mainly used in operations in hypercurvature geometric space, especially in Riemann hyperbolic geometry. The specific formula is as follows: , (4) In particular, when c=1, .

[0030] Furthermore, the (1) initializes each node of the collaborative knowledge graph in the Euclidean space to obtain an embedded representation of each node, including: Initialize node embeddings in Euclidean space and then perform Multimodal features Preprocessing is performed, and from the perspective of different modalities, a collaborative knowledge graph is constructed for each modality, which is defined as .

[0031] For different , keeping the same graph structure, but preserving the modality project features.

[0032] Since the pre-trained model With high dimensional characteristics, Modal features are unified into embedding dimensions: , (5) in, represents the trainable parameter matrix, represents the bias vector, Display items exist The initial embedding in express The initial embedding of the remaining nodes in Perform random initialization.

[0033] Furthermore, the embedding representation of each node in (1) is projected from the Euclidean space to the hyperbolic space of the Poincare sphere model, including: , (6) , (7) in and represents the embedding representation of the node in the hyperbolic space, Represents an exponential map in Riemannian hyperbolic geometry.

[0034] exist Modal Collaborative Knowledge Graph In, users and Projects appear together as an entity, and The relationship between the user and the item includes the interaction between the user and the item and the various semantic relationships between the items. The entity completes the aggregation and obtains the user and Projects The embedding representation of , while also considering the high-order connectivity between the two. Entities in ,use Represents the first-order neighbor entities and relationships of node h, by continuously integrating entities Neighborhood information, get entity The embedded representation of .

[0035] Furthermore, the (2) uses a hyperbolic relationship-aware graph neural network to map the embedding representation of the hyperbolic space node to the local Euclidean tangent space. After completing the aggregation operation in the tangent space, it is mapped back to the hyperbolic space to obtain the user's knowledge representation and the project's knowledge representation, including: The hyperbolic relation-aware graph neural network is defined as: , (8) in, and They are logarithmic mapping and exponential mapping in Riemann hyperbolic geometry, It is the Möbius addition. and For Entity and relationship The 0th-order embedding representation in hyperbolic space is, For Entity An embedding representation of rank 1 in hyperbolic space.

[0036] The hyperbolic relationship-aware graph neural network is recursively executed L times to further mine the L-order knowledge representation of entity h: , (9) After L layers, the representations of users and items at different layers are accumulated in the tangent space and reprojected back to the hyperbolic space to obtain the modal representations of users and items. The following knowledge representation: , (10) , (11) in, and Indicates that in modal Knowledge representation of users and items.

[0037] Furthermore, the (3) constructs a user co-occurrence graph based on the user's knowledge representation and the number of common interactions between users, including: Introducing Virtual Relationships in User Co-occurrence Graphs , given the user-item interaction matrix , the user co-occurrence graph with virtual relations is defined as: , (12) in, Represents the user, Represents a defined collection of users. Selection and user co-occurrence sorting set related, Indicates the sort number.

[0038] For each user, keep the top users and generate them according to the sorting number A virtual relationship.

[0039] For example, if user A has the largest number of common interactions with user B and the second largest number of common interactions with user C, then one relationship is generated between user A and user B, and another relationship is generated between user A and user C.

[0040] The addition of this virtual relationship makes the model more flexible in simulating user preferences. The user co-occurrence graph is regarded as a knowledge graph designed to enhance user representation and use the user’s knowledge representation to initialize its co-occurrence representation.

[0041] It should be understood that the user co-occurrence graph is based on the assumption that users who interact with similar items usually have similar preferences. By modeling the user's potential preferences from the perspective of co-occurring items, constructing a user co-occurrence graph can further enrich the user's representation, thereby improving the recommendation effect.

[0042] Furthermore, (3) above again uses a hyperbolic relationship-aware graph neural network to capture hierarchical and adjacency information based on the user co-occurrence graph, and obtains the final co-occurrence representation of users and the final representation of items under different modalities, including: Use hyperbolic relation-aware graph neural networks to capture hierarchical and adjacency information: , (13) in, Indicates user The first-order co-occurrence representation in hyperbolic space is, Indicates user Knowledge representation in hyperbolic space, Representing relationships Embedding representation in hyperbolic space.

[0043] Similarly, the hyperbolic relationship-aware graph neural network is recursively executed L times to obtain an L-order co-occurrence representation. The sum of all L layers can obtain the user's modality The final co-occurrence representation is: , (14) , (15) in, Indicates user The L-order co-occurrence representation in hyperbolic space is, Indicates user The L-1 order co-occurrence representation in hyperbolic space is, Representing relationships The embedding representation in hyperbolic space is, Indicates user exist Final co-occurrence representation by modality.

[0044] In addition, for the convenience of calculation and expression, users and items are The embedded representations under the modality are mapped back to the Euclidean space and given a unified symbolic representation: , (16) , (17) in, For users exist The final representation under the modal, For Project exist Final representation under modality.

[0045] Furthermore, the (4) conducts cross-modal comparative learning on the final co-occurrence representation of the user and the final representation of the project in different modalities to mine the potential information of the user and project embedded in different modalities, including: Collaborative Knowledge Graph and user co-occurrence graph After the hyperbolic relation-aware GCN in the above dataset is aggregated, the representation of users and items in the visual modality is obtained. and , and representation of users and items in text mode and , mapping them into the contrastive loss space: , (18) , (19) , (20) ,(twenty one) in, represents the representation of users and items after feature transformation in visual and textual modalities, represents the trainable parameter matrix, Represents the bias vector.

[0046] The representation of the same node learned in the corresponding other modality is defined as a positive pair, and the samples other than itself in the two modalities are defined as negative samples, that is, there are intra-modal negative pairs and inter-modal negative pairs. Therefore, this embodiment defines the following contrast loss: ,(twenty two); ,(twenty three); ,(twenty four); , (25); in and Respectively represent the user's visual mode and text modal The contrast loss calculated from the perspective, and Respectively represent the visual mode of the project and text modal The contrast loss calculated from the perspective, Represents the cosine similarity calculation function, Represents the temperature coefficient.

[0047] Finally, this embodiment uses the total contrast loss of cross-modal contrastive learning Defined as: , (26) in Indicates the number of users, Indicates the number of items.

[0048] Furthermore, the (5) outputs a recommendation result based on the final co-occurrence representation of the user in different modalities and the final representation of the item, including: Based on the final representations of users and items in different modalities, they are concatenated to obtain the final user representation and project characterization , and then predict the probability score through the inner product : , (27) , (28) , (29) in (·) represents the Sigmoid function, It will be used for click-through rate (CTR) prediction tasks and TopK recommendation tasks.

[0049] Next, the binary cross entropy function is used to calculate the recommendation loss: , (30) in is the true label, Represents the set of training samples.

[0050] The final joint optimization loss of this implementation model is as follows: , (31) in is the recommendation loss, is the contrast loss, is the regularization term, and are hyperparameters, which are used to adjust the weight of contrast loss and regularization term respectively.

[0051] This embodiment crawls user-project interaction information from the Steam game community and the MovieLens movie community to construct a multimodal knowledge collaboration dataset. The statistical information of the Steam and MovieLens datasets is shown in Table 1. At the same time, the dataset also contains multimodal information of the project, which provides data support for the smooth implementation of this embodiment.

[0052] For the Steam dataset, this embodiment collects data based on the Steam game community, including games purchased by users (considering purchases as an interaction) and attribute entities. After data cleaning and preprocessing, this embodiment retains 2613 users and 4086 projects. As for the knowledge graph, this embodiment retains 10077 entities in the Steam community, including attribute entities (such as tags, series). Based on these data, this embodiment first constructs a collaborative knowledge graph, which includes 8 types of entities and 14 types of relationships. In addition, this embodiment also captures visual images and text descriptions of relevant projects from the Steam community. Most of the current multimodal methods use two unaligned pre-trained models to encode visual and text information, which may cause conflicts between modal features. Therefore, this embodiment uses the Contrastive Language Image Pre-training (CLIP) model to encode visual and text modalities respectively. Specifically, this embodiment uses the ViT-B / 32 model pre-trained by CLIP to obtain 512-dimensional visual and text multimodal features.

[0053] Table 1 Statistics of Steam and MovieLens datasets

[0054] For the MovieLens dataset, this dataset is built on the basis of ML-1M, a public dataset widely used in the recommendation field. Although ML-1M cannot provide knowledge graphs and multimodal auxiliary information options, it can be linked to different auxiliary information through its project-side attributes.

[0055] Specifically, based on the ML-1M public dataset, this embodiment uses the open source repository KB4Rec to link to the Freebase knowledge graph related to project information. At the same time, this embodiment captures the multimodal information of movies from the MovieLens website, including movie posters and text introductions, and also uses the ViT-B / 32 model pre-trained by CLIP to obtain 512-dimensional multimodal features. Finally, the original user-project interaction records are filtered through the above two processes to match them with the auxiliary information.

[0056] This example will evaluate the performance of the proposed model in two tasks: CTR prediction and TopK recommendation. For the CTR prediction task, this example uses three indicators, Area Under Curve (AUC), Accuracy (ACC), and F1 score, to evaluate the recommendation performance of different models. For the TopK recommendation task, this example uniformly sets , the performance of different models is evaluated by calculating Precision@K, Recall@K and NDCG@K.

[0057] This example is implemented based on the PyTorch framework. Hyperparameters include learning rate , batch size , embedding dimension 、GCN hop count , user co-occurrence coefficient , Comparative Temperature Coefficient , contrastive learning loss coefficient and the regularization coefficient In this example, the learning rate is set to 0.001, the batch size is set to 1024, and the user co-occurrence coefficient is Set to 30, compare the temperature coefficient Set to 0.1.

[0058] In this example, all baseline models have the same learning rate and batch size, and the rest use their recommended settings. For other hyperparameters, grid search is used to determine the best combination, that is, , , and .

[0059] This embodiment divides the baseline model into collaborative filtering-based methods, knowledge graph-based methods, and multimodal-based methods according to the fields to which they belong. The performance of this implementation model and other baseline methods is compared in CTR prediction and TopK recommendation (taking K=20 as an example) tasks. The results are shown in Table 2 and Table 3: Table 2 Performance comparison of Steam dataset

[0060] As shown in Tables 2 and 3, the proposed model outperforms other methods in all indicators of the two datasets Steam and MovieLens. The proposed model has an average improvement of 4% in the CTR prediction task and an average improvement of 60% in the TopK recommendation task. This can be attributed to the fact that the proposed model combines knowledge-aware recommendation and multimodal recommendation, making full use of these two heterogeneous auxiliary information. It can also be noted that the multimodal method shows more competitive performance compared with the knowledge-aware recommendation method, which shows that in this scenario, multimodal information provides more auxiliary information for recommendation. This is consistent with the intuition in the real world. When purchasing items, users are more likely to be attracted by intuitive images and texts, and are more likely to ignore potential attribute information.

[0061] Table 3 Performance comparison of MovieLens dataset

[0062] In order to verify the effectiveness of each module of this implementation model, this embodiment designs the following four variants for ablation experiments: (1) No hyperbolic space: The relevant calculations in the hyperbolic space are removed and the aggregation is performed in the Euclidean space. (2) No multimodality: The multimodal information of the project is removed and the ID of the project is embedded, and the subsequent cross-modal comparison is removed. (3) No user aggregation: The aggregation of user co-occurrence graphs with virtual relationships is removed, and the knowledge representation of users is directly contrasted. (4) No contrastive learning: By changing the contrastive learning loss coefficient Set to 0 to remove the cross-modal comparison learning module. The comparison results of the ablation experiment in this embodiment are shown in Table 4 and Table 5: Table 4 Ablation experiment on Steam dataset

[0063] Table 5 Ablation experiments on MovieLens dataset

[0064] As can be seen from Tables 4 and 5, the ablation study results show that the performance of all variants is worse than the complete model, which shows that the key components in the implementation model have made positive contributions. Specifically, the performance of the model is the worst after removing the user co-occurrence aggregation, which shows that the user co-occurrence graph with virtual relationships used in this embodiment can significantly enhance the representation of users. The effect is the second worst after removing the multimodal information, which means that multimodal information provides key auxiliary information in the process of knowledge aggregation. In addition, the hyperbolic GCN module and contrastive learning module proposed in this embodiment also improve the recommendation performance to a certain extent. The above results fully illustrate the rationality and effectiveness of the design of each module of the implementation model.

[0065] Based on the rich information in the online community, the present invention combines entity recognition, relationship extraction, web crawler and other technologies to construct multimodal information and knowledge graphs of project side links, and uses them as input data of the model; the multimodal pre-trained features of the project are used to initialize the nodes in the Euclidean space, and the hyperbolic embedding layer is used to embed and project the nodes into the Riemann hyperbolic space; the hyperbolic relationship-aware GCN aggregation module is used to realize high-order modeling of users and projects under different modal perspectives through mutual mapping of hyperbolic space and tangent space; a user co-occurrence graph is constructed based on the number of common interactions between users, and a virtual relationship based on co-occurrence ranking is designed, and hyperbolic GCN aggregation is carried out on the basis of the co-occurrence graph to further enhance the multimodal representation of users; cross-modal comparative learning is carried out for the embedding of users and projects under different modalities, and the potential information of users and projects embedded in different modalities is mined; the user's knowledge representation is combined with the co-occurrence representation, and the inner product is made with the final representation of the project to output the prediction score, and the proposed model is optimized using a joint loss function based on binary cross entropy loss and contrast loss, and finally the recommendation process is completed.

[0066] Embodiment 2: This embodiment provides a knowledge recommendation system based on cross-modal comparison and Riemann hyperbolic geometry, including: an acquisition module, which is configured to: acquire a bipartite graph of user-item interaction; acquire a basic knowledge graph; combine the bipartite graph of user-item interaction with the basic knowledge graph to obtain a collaborative knowledge graph; set multimodal features for each item in the collaborative knowledge graph, the multimodal features including: visual modal features and textual modal features; The recommendation module is configured to: input the multimodal features of the collaborative knowledge graph and the project into the trained knowledge recommendation model, and recommend projects to users through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal contrastive learning to further explore the potential information between different modalities, introducing additional supervisory signals for the recommendation task.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry, characterized by: include: Get the bipartite graph of user-item interactions; Obtain basic knowledge graph; Combining the user-item interaction bipartite graph with the basic knowledge graph to obtain a collaborative knowledge graph; Setting a multimodal feature for each item in the collaborative knowledge graph, wherein the multimodal feature includes: a visual modality feature and a textual modality feature; The multimodal features of the collaborative knowledge graph and the project are input into the trained knowledge recommendation model, and the project is recommended to the user through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal contrastive learning to further mine the potential information between different modalities, introducing additional supervisory signals for the recommendation task.

2. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 1, characterized in that: The trained knowledge recommendation model is designed through the following steps: (1) Initialize each node of the collaborative knowledge graph in Euclidean space to obtain the embedded representation of each node; project the embedded representation of each node from Euclidean space to the hyperbolic space of the Poincare sphere model; (2) Using a hyperbolic relationship-aware graph neural network, the embedded representation of the hyperbolic space node is mapped to the local Euclidean tangent space. After completing the aggregation operation in the tangent space, it is mapped back to the hyperbolic space to obtain the user's knowledge representation and the project's knowledge representation. (3) Based on the user’s knowledge representation and the number of common interactions between users, a user co-occurrence graph is constructed. Based on the user co-occurrence graph, a hyperbolic relationship-aware graph neural network is used again to capture the hierarchical and adjacency information, and the final co-occurrence representation of users and the final representation of items under different modalities are obtained. (4) Conduct cross-modal comparative learning on the final co-occurrence representations of users and the final representations of the items in different modalities to explore the potential information embedded in users and items in different modalities; (5) Based on the final co-occurrence representation of the user under different modalities and the final representation of the item, output the recommendation results.

3. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (1) step initializes each node of the collaborative knowledge graph in the Euclidean space to obtain an embedded representation of each node, including: Initialize node embeddings in Euclidean space and then perform Multimodal features Preprocessing is performed, and from the perspective of different modalities, a collaborative knowledge graph is constructed for each modality, which is defined as ; ; For different , keeping the same graph structure, but preserving the modality Project characteristics; Since the pre-trained model With high dimensional characteristics, Modal features are unified into embedding dimensions: ，（5）； in, represents the trainable parameter matrix, represents the bias vector, Display items exist The initial embedding in express The initial embedding of the remaining nodes in Perform random initialization.

4. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (1) embedding representation of each node is projected from the Euclidean space to the hyperbolic space of the Poincare sphere model, including: ，（6）； ，（7）； in and represents the embedding representation of the node in the hyperbolic space, Represents exponential maps in Riemannian hyperbolic geometry; exist Modal Collaborative Knowledge Graph In, users and Projects appear together as an entity, and The relationship between the user and the item includes the interaction between the user and the item and the various semantic relationships between the items. The entity completes the aggregation and obtains the user and Projects The embedding representation of , while also considering the high-order connectivity between the two; Entities in ,use Represents the first-order neighbor entities and relationships of node h, by continuously integrating entities Neighborhood information, get entity The embedded representation of .

5. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (2) uses a hyperbolic relationship-aware graph neural network to map the embedding representation of the hyperbolic space node to the local Euclidean tangent space. After completing the aggregation operation in the tangent space, it is mapped back to the hyperbolic space to obtain the user's knowledge representation and the project's knowledge representation, including: The hyperbolic relation-aware graph neural network is defined as: ，（8）； in, and They are logarithmic mapping and exponential mapping in Riemann hyperbolic geometry, It is the Möbius addition. and For Entity and relationship The 0th-order embedding representation in hyperbolic space is, For Entity 1st-order embedding representation in hyperbolic space; The hyperbolic relationship-aware graph neural network is recursively executed L times to further mine the L-order knowledge representation of entity h: ，（9）； After L layers, the representations of users and items at different layers are accumulated in the tangent space and reprojected back to the hyperbolic space to obtain the modal representations of users and items. The following knowledge representation: ，（10）； ，（11）； in, and Indicates that in modal Knowledge representation of users and items.

6. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (3) constructs a user co-occurrence graph based on the user's knowledge representation and the number of common interactions between users, including: Introducing Virtual Relationships in User Co-occurrence Graphs , given the user-item interaction matrix , the user co-occurrence graph with virtual relations is defined as: ，（12）； in, Represents the user, Represents a defined set of users; Selection and user co-occurrence sorting set related, Indicates the sorting number; For each user, keep the top users and generate them according to the sorting number A virtual relationship.

7. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: In (3), based on the user co-occurrence graph, the hyperbolic relationship-aware graph neural network is used again to capture the hierarchical and adjacency information, and the final co-occurrence representation of users and the final representation of items under different modalities are obtained, including: Use hyperbolic relation-aware graph neural networks to capture hierarchical and adjacency information: ，（13）； in, Indicates user The first-order co-occurrence representation in hyperbolic space is, Indicates user Knowledge representation in hyperbolic space, Representing relationships Embedding representation in hyperbolic space; The hyperbolic relationship-aware graph neural network is recursively executed L times to obtain an L-order co-occurrence representation. The user’s modality can be obtained by accumulating all L layers. The final co-occurrence representation is: , (14); ，（15）； in, Indicates user The L-order co-occurrence representation in hyperbolic space is, Indicates user The L-1 order co-occurrence representation in hyperbolic space is, Representing relationships The embedding representation in hyperbolic space is, Indicates user exist Final co-occurrence representation under modality; Place users and projects in The embedding representations under the modality are mapped back to the Euclidean space and given a unified symbolic representation: ，（16）； ，（17）； in, For users exist The final representation under the modal, For Project exist Final representation under modality.

8. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (4) conducts cross-modal comparative learning on the final co-occurrence representation of the user and the final representation of the project in different modalities to mine the potential information of the user and project embedded in different modalities, including: Collaborative Knowledge Graph and user co-occurrence graph After the hyperbolic relation-aware GCN in the above dataset is aggregated, the representation of users and items in the visual modality is obtained. and , and representation of users and items in text mode and , mapping them into the contrastive loss space: ，（18）； ，（19）； ，（20）； ，（21）； in, represents the representation of users and items after feature transformation in visual and textual modalities, represents the trainable parameter matrix, represents the bias vector; The representation of the same node learned in the corresponding other modality is defined as a positive sample pair, and the samples other than itself in the two modalities are defined as negative samples. There are same-modal negative sample pairs and cross-modal negative sample pairs; the following contrast loss is defined: ，（22）； ，（23）； ，（24）； ，（25）； in and Respectively represent the user's visual mode and text modal The contrast loss calculated from the perspective, and Respectively represent the visual mode of the project and text modal The contrast loss calculated from the perspective, Represents the cosine similarity calculation function, represents the temperature coefficient; Finally, the total contrastive loss for cross-modal contrastive learning is Defined as: ，（26）； in Indicates the number of users, Indicates the number of items.

9. The knowledge recommendation method based on cross-modal comparison and Riemannian hyperbolic geometry as claimed in claim 2, characterized in that: The (5) outputs a recommendation result based on the final co-occurrence representation of the user in different modalities and the final representation of the item, including: Based on the final representations of users and items in different modalities, they are concatenated to obtain the final user representation and project characterization , and then predict the probability score through the inner product : ，（27）； ，（28）； ，（29）； in (·) represents the Sigmoid function, It will be used for click-through rate prediction tasks and TopK recommendation tasks; Next, the binary cross entropy function is used to calculate the recommendation loss: ，（30）； in is the true label, represents the set of training samples; The final joint optimization loss is as follows: ，（31）； in is the recommendation loss, is the contrast loss, is the regularization term, and are hyperparameters, which are used to adjust the weight of contrast loss and regularization term respectively.

10. A knowledge recommendation system based on cross-modal comparison and Riemannian hyperbolic geometry, characterized by: include: An acquisition module is configured to: acquire a bipartite graph of user-item interactions; Obtain basic knowledge graph; Combining the user-item interaction bipartite graph with the basic knowledge graph to obtain a collaborative knowledge graph; Setting a multimodal feature for each item in the collaborative knowledge graph, wherein the multimodal feature includes: a visual modality feature and a textual modality feature; The recommendation module is configured to: input the multimodal features of the collaborative knowledge graph and the project into the trained knowledge recommendation model, and recommend projects to users through the trained knowledge recommendation model; the trained knowledge recommendation model uses cross-modal contrastive learning to further explore the potential information between different modalities, introducing additional supervisory signals for the recommendation task.

Images