Personalized recommendation method based on multi-modal knowledge graph

Abstract

This invention discloses a personalized recommendation method based on a multimodal knowledge graph. The method includes: acquiring a public dataset and corresponding text, image, and audio files for items; extracting feature vectors for each modality; adding multimodal sub-entities for each item and establishing associations; constructing intramodal similarity relationships to form a multimodal knowledge graph; mapping each modal feature vector to a unified dimension through a fully connected layer; determining dynamic weights; and weighted summing to obtain fused features; based on a diffusion model, using a linear noise scheduler to progressively add noise; using the fused features as semantic guidance; iteratively denoising through a denoising network; and obtaining a high-quality knowledge graph through Top-K filtering; combining the high-quality knowledge graph with user-item interaction data; aggregating neighborhood entity features through RGAT to obtain item embeddings; injecting these into the user-item interaction graph; capturing high-order collaborative filtering signals through GCN; calculating the inner product score between users and items; and outputting a recommendation list.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer software and information technology, and in particular to a personalized recommendation method based on a multi-modal knowledge graph. BACKGROUND

[0002] In the current digital age of information explosion, recommendation systems have become the core link connecting users and massive information. However, the traditional recommendation paradigm based on collaborative filtering (CF) mainly relies on the interaction ID of users and items, and faces serious sparsity problems, and cannot understand the semantic content of the items themselves. Although the introduction of knowledge graphs (KG) alleviates the sparsity to some extent and enhances the explainability, most existing knowledge graph recommendation methods are based on static and deterministic graph structures for reasoning, ignoring the structural noise and long-tail distribution problems commonly existing in real-world graphs. In addition, with the explosion of multimedia content, single structured knowledge is not enough to describe the rich attributes of items. Text, images, audio and other multi-modal information contain valuable user preference clues, but there is a significant "semantic gap" and "heterogeneity" between different modalities, making it difficult for traditional algorithms to capture users' deep personalized needs, resulting in insufficient recommendation accuracy, and even being helpless in cold start scenarios.

[0003] In the existing research on multi-modal knowledge graph-based recommendation technology, the fusion of multi-modal features and knowledge graphs is mostly shallow concatenation, and there is a lack of effective graph denoising methods, further exacerbating the problems of insufficient recommendation accuracy and robustness. Therefore, how to effectively fuse multi-modal information and knowledge graphs, filter structural noise in graphs, alleviate the long-tail distribution and data sparsity problems, and improve the accuracy, robustness and explainability of recommendation systems, has become a technical problem to be solved in the current recommendation technology field.

[0004] In multi-modal recommendation, early methods mostly use simple feature concatenation or fixed weight weighting to fuse multi-modal information, which cannot effectively solve the problem of modal heterogeneity and cannot fully exploit the complementary value between different modalities. Subsequent improved methods based on attention mechanisms or graph neural networks have static modal correlation weights that are difficult to adapt to the dynamic changes in the importance of different scenarios, and some models also have the defect of disconnection between fusion targets and recommendation preferences, resulting in insufficient feature practicality. Existing knowledge graph recommendation technologies generally ignore the structural noise and long-tail distribution problems of graphs, and the fusion of multi-modal features and knowledge graph structures is mostly shallow concatenation, which cannot fully play the guiding role of multi-modal semantics in graph reasoning. In recent years, although some technologies have introduced diffusion models into the recommendation field, related models have not achieved collaborative optimization with multi-modal knowledge graphs, and there are problems such as insufficient alignment between diffusion processes and recommendation targets and lack of semantic support, which cannot fully exploit the technical advantages of diffusion models in graph denoising and completion. SUMMARY

[0005] The main purpose of the present application is to provide a personalized recommendation method based on a multi-modal knowledge graph.

[0006] Another purpose of the present application is to provide a personalized recommendation device based on a multi-modal knowledge graph.

[0007] The third purpose of the present application is to provide a computer device.

[0008] The fourth purpose of the present application is to provide a non-transitory computer readable storage medium.

[0009] To achieve the above purpose, the first aspect of the present application provides a personalized recommendation method based on a multi-modal knowledge graph, comprising:

[0010] S1, obtaining public data sets and text, image, audio files corresponding to the items, filtering invalid files, extracting modal feature vectors through BERT, ResNet, VGGish models, adding multi-modal sub-entities to each item and establishing a correlation relationship, constructing intra-modal similarity relationship based on cosine similarity, and forming a multi-modal knowledge graph; S2, mapping the modal feature vectors to a unified dimension through a full connection layer, determining dynamic weights through a cross-modal attention mechanism and a gate network, and obtaining a fusion feature through weighted summation; S3, based on the diffusion model, gradually adding noise to the multi-modal knowledge graph using a linear noise scheduler, taking the fusion feature as a semantic guide, and iteratively denoising through a denoising network to obtain a high-quality knowledge graph through Top-K screening; S4, combining the high-quality knowledge graph and user-item interaction data, aggregating neighborhood entity features through RGAT to obtain item embeddings, injecting them into the user-item interaction graph, capturing high-order collaborative filtering signals through GCN, calculating the inner product score of users and items, and outputting a recommendation list.

[0011] In one embodiment of the present application, the multi-modal sub-entities are added to each item and the correlation relationship is established, the intra-modal similarity relationship is constructed based on the cosine similarity, and the multi-modal knowledge graph is formed, comprising: Text sub-entities, image sub-entities, and audio sub-entities are added to each item, and sub-entity IDs are generated using the basic entity ID+modal offset rule; The entity-sub-entity correlation relationship between the item entity and the text sub-entity, image sub-entity, and audio sub-entity is established to realize the structured correlation between the item and the multi-modal features; For text, image, and audio three types of modalities, the cosine similarity of the same modal feature vectors of any two items is calculated; Screening the top K feature pairs of cosine similarity under each type of modality, establishing SIM_TEXT, SIM_IMG, SIM_AUDIO type of intra-modal similarity relationship, and enhancing the semantic association between cross items; Integrating the entity-subentity association relationship, intra-modal similarity relationship and original entity relationship of the item to form a multi-modal knowledge graph.

[0012] In an embodiment of the present application, the mapping of the modality feature vectors to a unified dimension through a full connection layer, the determination of dynamic weights through a cross-modal attention mechanism and a gate network, and the weighted summation to obtain the fusion features, include: The original features of the three types of modalities of text, image and audio are respectively nonlinearly mapped through independent full connection layers, the GeLU activation function is used to process the output of the full connection layer, the features of each modality are uniformly converted to a 128-dimensional latent semantic space, and the projected three types of modalities are obtained; The stacked and combined projected three types of modalities are input into a multi-head self-attention mechanism, the stacked features are taken as query Q, key K and value V for inter-modal deep interaction calculation, and a feature matrix containing global multi-modal context information is obtained; The feature matrix containing global multi-modal context information is flattened and input into a gate network, the dynamic weights of each modality are obtained through full connection layer calculation and Softmax normalization; Based on the dynamic weights, the weighted summation of the projected three types of modalities is performed to obtain the fusion features.

[0013] In an embodiment of the present application, based on the diffusion model, a linear noise scheduler is used to gradually add noise to the multi-modal knowledge graph, the fusion features are used as semantic guidance, and a high-quality knowledge graph is obtained through iterative denoising of the denoising network and Top-K screening, including: A linear noise scheduler is used to gradually add standard Gaussian noise and modality feature projection noise to the adjacency matrix of the multi-modal knowledge graph in T time steps, following the rules of Markov chain, and the noise intensity increases linearly with time steps; The fusion features are input into the denoising network as conditional constraints, starting from the noise adjacency matrix of the Tth step, the noise is gradually predicted and removed through the denoising network, and the contribution of the prediction results under the guidance of the multi-modal features and the prediction results without conditions in the denoising network prediction output is balanced through the dynamic gate weight of the time step; Through the reverse denoising process, a complete adjacency matrix with element relationship confidence is obtained, the Top-K screening strategy is used to select the top K relationships with the highest confidence, and a high-quality knowledge graph is constructed.

[0014] In one embodiment of the present invention, the step of combining the high-quality knowledge graph with user-item interaction data, aggregating neighborhood entity features through RGAT to obtain item embeddings, injecting them into the user-item interaction graph, capturing high-order collaborative filtering signals through GCN, calculating the inner product score between users and items, and outputting a recommendation list includes: On high-quality knowledge graphs, RGAT is used for any triples. Calculate the attention coefficient under the correspondence, and apply the Softmax function to the center node. The attention coefficients within the neighborhood are normalized to obtain the aggregation weights; wherein, in the triples As the central node, For neighboring nodes, It is a relation type; Based on the aggregation weights, the features of neighboring nodes are aggregated by weighted summation, and residual connections and normalization and dropout operations are introduced to update the representation of the central node, resulting in an item embedding rich in knowledge semantics. The items are embedded into the user-item interaction graph, and a GCN layer is used to perform graph convolution operations on the bipartite graph to aggregate the multi-hop neighbor features of users or items and capture high-order collaborative filtering signals. Calculate the inner product score between users and items, and output a recommendation list based on the scores.

[0015] To achieve the above objectives, a second aspect of the present invention provides a personalized recommendation device based on a multimodal knowledge graph, comprising: The data processing and multimodal knowledge graph construction module is used to obtain public datasets and text, image, and audio files corresponding to items. After filtering out invalid files, it extracts feature vectors of each modality using BERT, ResNet, and VGGish models respectively, adds multimodal sub-entities for each item and establishes association relationships, and constructs intramodal similarity relationships based on cosine similarity to form a multimodal knowledge graph. The dynamic weighted multimodal fusion module is used to map the feature vectors of each modality to a unified dimension through a fully connected layer, determine the dynamic weights through a cross-modal attention mechanism and a gating network, and obtain the fused features by weighted summation; The diffusion model denoising and completion module is used to gradually add noise to the multimodal knowledge graph based on the diffusion model using a linear noise scheduler, guided by the fused features, and iteratively denoising through a denoising network, and obtain a high-quality knowledge graph through Top-K screening; The graph network representation learning module combines the high-quality knowledge graph with user-item interaction data, aggregates neighborhood entity features through RGAT to obtain item embeddings, injects them into the user-item interaction graph, captures high-order collaborative filtering signals through GCN, calculates the inner product score between users and items, and outputs a recommendation list.

[0016] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing a personalized recommendation method based on a multimodal knowledge graph as described in the first aspect embodiment.

[0017] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a personalized recommendation method based on a multimodal knowledge graph as described in the first aspect embodiment.

[0018] The embodiments of the present invention have the following beneficial effects: This invention employs a collaborative architecture of dynamically weighted multimodal fusion and diffusion-denoised multimodal knowledge graphs. This architecture enables modal weights to adapt to the semantic characteristics of different items, addressing the modal heterogeneity problem and improving feature representation accuracy. Simultaneously, the diffusion-denoised mechanism filters structural noise in the knowledge graph, completes long-tail entity associations, and alleviates data sparsity. This embodiment, combined with graph network reasoning using relational graph attention, enhances the transmission efficiency of entity semantic associations and improves the interpretability of recommendation results. It performs better in cold-start scenarios, ultimately achieving significant improvements over existing technologies in terms of accuracy, robustness, and scenario adaptability. Attached Figure Description

[0019] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating the personalized recommendation method based on multimodal knowledge graphs provided in this embodiment of the invention; Figure 2 An architecture diagram of the personalized recommendation method based on multimodal knowledge graph provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multimodal knowledge graph structure provided in an embodiment of the present invention; Figure 4 This is a structural diagram of a personalized recommendation device based on a multimodal knowledge graph provided in an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The research objective of this invention is to construct a personalized recommendation model (DiffMKG) based on diffusion model and multimodal knowledge graph. By integrating multimodal semantic information and diffusion denoising capabilities, it solves the problems of multimodal heterogeneity, knowledge graph noise and sparsity in existing recommendation systems, and provides high-quality semantic support and efficient technical solutions for personalized recommendation tasks.

[0023] The following describes a personalized recommendation method based on a multimodal knowledge graph according to an embodiment of the present invention, with reference to the accompanying drawings.

[0024] Example 1 This embodiment provides a personalized recommendation method based on multimodal knowledge graphs. For example... Figure 1 and Figure 2 As shown, the method includes the following steps: S1. Obtain the publicly available dataset and the corresponding text, image, and audio files of the items. After filtering out invalid files, extract the feature vectors of each modality using BERT, ResNet, and VGGish models respectively. Add multimodal sub-entities for each item and establish association relationships. Construct intramodal similarity relationships based on cosine similarity to form a multimodal knowledge graph.

[0025] This step is the multimodal data preprocessing stage. We select publicly available datasets such as Movielens and Amazon-book to organize the text descriptions (e.g., product introductions, plot summaries), image files (e.g., posters, covers, illustrations), and audio files (e.g., movie clips, background music) corresponding to the items. We establish a one-to-one mapping table between items and modal files to clarify the data relationships. Invalid modal files (e.g., corrupted images, empty text, and audio files of abnormal length) are filtered to lay the foundation for subsequent feature extraction.

[0026] Specifically, image feature extraction uses the ResNet model. After inputting an object image, visual features are extracted through multi-layer convolution and pooling operations, ultimately outputting a 2048-dimensional image feature vector. Text feature extraction uses a BERT pre-trained model, performing word segmentation, masking, language model training, and encoding on the object text description to generate a 768-dimensional semantic vector. Audio feature extraction: Audio features are extracted based on the VGGish model. The audio signal is processed through a Mel filter bank that simulates human auditory perception, outputting a 128-dimensional audio feature vector.

[0027] Specifically, S1 also includes the following steps: S11, add text sub-entities, image sub-entities, and audio sub-entities for each item, and generate sub-entity IDs using the basic entity ID + modal offset rule.

[0028] Specifically, at the entity level, text, image, and audio sub-entities are added for each item. Sub-entity IDs are generated using a "base entity ID + modal offset" rule to ensure the uniqueness of each entity. The base entity ID is the unique identifier of the item in the original dataset, and the modal offset is used to distinguish different modal sub-entities. For example, the offset for text sub-entities is set to 1000, for image sub-entities to 2000, and for audio sub-entities to 3000. A globally unique sub-entity ID is generated by simply concatenating these values, avoiding entity identifier conflicts and providing a unified and standardized entity identification system for subsequent graph relationship construction and entity retrieval.

[0029] S12, establish entity-sub-entity association relationships between item entities and text sub-entities, image sub-entities, and audio sub-entities to realize the structured association between items and multimodal features.

[0030] Specifically, at the relational level, new entity-sub-entity relationships are added, connecting item entities with their corresponding text, image, and audio sub-entities, establishing structured associations between items and multimodal features. These relationships represent exclusive bindings between item entities and their respective modal sub-entities, clarifying the attribution logic of item-modal features. This integrates scattered text, image, and audio features into the knowledge graph in the form of entity relationships, achieving structured storage and unified management of multimodal information, and laying the relational foundation for subsequent collaborative utilization of modal features and graph reasoning.

[0031] S13 calculates the cosine similarity of feature vectors of any two items of the same modality for the three modalities of text, image, and audio.

[0032] Specifically, the similarity of features within the same modality is calculated using the cosine similarity formula, which is:

[0033] in, , Let be the feature vectors of two items in the same modality. Cosine similarity measures the semantic similarity between features in the same modality by measuring the cosine of the angle between the feature vectors in high-dimensional space. The value ranges from [-1, 1]. The closer the value is to 1, the more similar the features are semantically, and the closer it is to -1, the greater the semantic difference. This index can effectively capture the semantic association between items in text, image, and audio modalities, and provide a quantitative basis for constructing intramodal similarity relationships.

[0034] S14: Select the top K feature pairs with the highest cosine similarity in each modality, establish intramodal similarity relationships for SIM_TEXT, SIM_IMG, and SIM_AUDIO, and enhance semantic association across items.

[0035] Specifically, new intra-modal similarity relationships are added, including SIM_IMG, SIM_TEXT, and SIM_AUDIO. Top-K similarity feature pairs are selected to construct intra-modal similarity relationships, enhancing semantic associations across items. For text, image, and audio modalities, the cosine similarity of all item pairs is sorted in descending order, and the top K item pairs are selected to establish similarity relationships for the corresponding modalities. This approach preserves the most semantically relevant item associations in each modality, avoids redundant relationships, and distinguishes the semantic association characteristics of different modalities through modal-specific similarity relationships, achieving refined modeling of multimodal semantic associations.

[0036] S15, integrate the entity-sub-entity association, intra-modal similarity relationship and the original entity relationship of the item to form a multimodal knowledge graph.

[0037] Specifically, entity-sub-entity relationships, intra-modal similarity relationships, and existing entity relationships of items are integrated to form a complete multimodal knowledge graph. Based on existing item entities and traditional entity relationships (such as item-category, item-attribute), newly added multimodal sub-entities, entity-sub-entity relationships, and intra-modal similarity relationships are incorporated to construct a heterogeneous knowledge graph covering item entities, multimodal sub-entities, and multiple types of relationships. This achieves the unified fusion of item semantic information, multimodal feature information, and association information, such as... Figure 3 The diagram shown is a schematic of the structure of a multimodal knowledge graph.

[0038] S2, the feature vectors of each modality are mapped to a unified dimension through a fully connected layer, and dynamic weights are determined through a cross-modal attention mechanism and a gating network. The weighted sum is then used to obtain the fused features.

[0039] In this embodiment of the invention, the core objective of this step is to solve the inherent heterogeneity problem of multimodal data (i.e., the differences in dimension, semantic expression, and distribution characteristics of different modal features), and to achieve deep collaboration and adaptive fusion of text, image, and audio modal information, so as to provide comprehensive and targeted semantic features for subsequent knowledge graph denoising and completion and recommendation reasoning.

[0040] Specifically, S2 includes the following steps: S21 uses independent fully connected layers to perform nonlinear mapping on the original features of the three modalities of text, image, and audio respectively. The GeLU activation function is used to process the output of the fully connected layer, and the features of each modality are uniformly transformed into a 128-dimensional latent semantic space to obtain the projected features of the three modalities.

[0041] In this embodiment, due to the significant differences in the dimensionality of the original multimodal features (image features are 2048-dimensional, text features are 768-dimensional, and audio features are 128-dimensional), direct fusion would lead to high-dimensional features dominating semantic expression, while the value of low-dimensional features would be obscured. Therefore, a separate fully connected layer is used to perform nonlinear mapping on various modal features, uniformly transforming them into a 128-dimensional latent semantic space. This fully connected layer uses the GeLU activation function to introduce a nonlinear transformation, which can preserve the core semantic information of each modality while compressing dimensions, achieving comparability and fusion of different modal features at the same semantic scale. The mapping process can be represented as follows:

[0042] in, and These are the weight matrix and bias terms for the corresponding modal fully connected layer. For original modal features, Features after mapping and projection ( , representing text, image, and audio respectively.

[0043] S22, the three types of modal features after projection are stacked and combined, and input into a multi-head self-attention mechanism. The stacked features themselves are used as query Q, key K, and value V to perform deep interaction calculation between modalities, and a feature matrix containing global multimodal context information is obtained.

[0044] Specifically, this step captures semantic associations between modalities through a cross-modal attention mechanism. The core logic of the attention mechanism is "focusing on important information." The cross-modal attention mechanism introduced here generates a context feature matrix that integrates cross-modal interaction information by calculating the semantic similarity between features of different modalities. The mapped three types of modal features undergo self-attention computation. The relevance between different modalities is measured by the dot product similarity between the query and the key. Softmax is used to normalize the attention distribution, thereby weighted aggregation of the value. This allows semantically relevant modal features to reinforce each other and weakens the interference of irrelevant or redundant modal information.

[0045] S23, after flattening the feature matrix containing global multimodal context information, input it into the gated network, calculate the dynamic weights of each modality through the fully connected layer and normalize them with Softmax.

[0046] In this embodiment, this step learns and outputs dynamic fusion weights for each modality through a gating network. The core function of the gating network is to adaptively adjust the weight ratios of different modalities based on the current input multimodal features and the relationships between modalities, avoiding the limitations of traditional static weighted fusion (such as fixed weight averaging) which cannot adapt to different items and scenarios. Specifically, the context feature matrix output by the cross-modal attention mechanism is used to... The vectors are flattened and then input into a gated network. The dynamic weights for each mode are calculated through a fully connected layer and normalized using Softmax. The calculation formula is as follows:

[0047] in, and For the weights and bias parameters of the gated network, This is a temperature coefficient used to adjust the smoothness of the weight distribution. The smaller the value, the more concentrated the weight distribution, and the stronger the model's ability to focus on important modes; The larger the value, the more evenly the weight distribution, which can prevent the model from over-relying on a single modality.

[0048] S24. Based on the dynamic weights, the three types of modal features after projection are weighted and summed to obtain the fused features.

[0049] Specifically, this step performs adaptive weighted fusion of multimodal features. Based on the dynamic weights output by the gated network, the projected features of the three modalities are weighted and summed to obtain the final fused features. The fusion formula is:

[0050] in, Dynamic weights for text, images, and audio. These are the modal features after projection. This fusion process can dynamically adjust the contribution of each modality according to the modal feature characteristics of different items, achieving precise coordination of multimodal information and providing reliable semantic guidance for the subsequent diffusion denoising process.

[0051] S3. Based on the diffusion model, noise is gradually added to the multimodal knowledge graph using a linear noise scheduler. With the fused features as semantic guidance, noise is iteratively removed through a denoising network, and a high-quality knowledge graph is obtained through Top-K screening.

[0052] Specifically, this step is based on the core framework of the diffusion model, which is "forward noise addition and reverse noise reduction", and combines multimodal semantic constraints to achieve targeted noise reduction and accurate completion.

[0053] Specifically, S3 includes the following steps: S31 employs a linear noise scheduler that, following the Markov chain rule, gradually adds standard Gaussian noise and modal feature projection noise to the adjacency matrix of the multimodal knowledge graph over T time steps. The noise intensity increases linearly with time steps.

[0054] Specifically, this step is a forward noise addition process. A linear noise scheduler is used to progressively add noise to the adjacency matrix of the knowledge graph, including standard Gaussian noise and modal feature projection noise. The forward process follows the Markov chain rule, adding noise to the adjacency matrix step by step over T time steps, with the noise intensity varying with time steps. Linearly increasing. The core purpose of this process is to simulate the relationship corruption process in knowledge graphs in the real world, allowing the model to learn the ability to recover real relationships from noisy data of varying degrees during training.

[0055] S32, with the fusion features as the conditional constraints input to the denoising network, starting from the noise adjacency matrix at step T, the noise is gradually predicted and removed through the denoising network. By dynamically gating the weights at each time step, the contributions of the prediction results guided by the multimodal features and the prediction results without conditions to the prediction output of the denoising network are balanced.

[0056] Specifically, this step is a reverse denoising process. It involves introducing multimodal fusion features. As a semantic guide, it achieves targeted denoising and relation recovery. The reverse process is the inverse of the forward denoising process, and its goal is to start from the first... Noise adjacency matrix of the step Starting from the denoising network (Parameters are) The neural network progressively predicts and removes noise, ultimately recovering a high-quality true adjacency matrix. Unlike traditional unguided diffusion denoising, this stage uses the multimodal fusion features obtained in stage 2. As a conditional input to the denoising network, it guides the denoising direction to align with the multimodal semantic characteristics of the items, ensuring that the recovered entity relationships conform to objective semantic logic (e.g., avoiding incorrect association between the entity "comedy movie" and the entity "horror actor"). Its core prediction formula is:

[0057] in, This represents the unconditional predicted output of the denoising network. This represents the predicted output of the denoising network guided by multimodal features. Indicates time step The determined dynamic gating weights balance the "diversity" of unconditional generation with the "accuracy" of conditional generation—in the initial stage of noise addition ( Smaller size, less noise) Larger values ​​enhance the weight of multimodal guidance, ensuring that core real relationships are not mistakenly deleted; in the later stages of noise addition ( Larger size, more noise) Smaller values ​​increase the weight of unconditionally generated data, enhancing the model's ability to complete missing relationships.

[0058] S33. A complete adjacency matrix with relation confidence levels is obtained through a reverse denoising process. The top-K relations with the highest confidence levels are selected using a Top-K screening strategy to construct a high-quality knowledge graph.

[0059] Specifically, this step involves knowledge graph completion and quality screening. A reverse denoising process is used to obtain the denoised complete adjacency matrix. The elements in this matrix are continuous values ​​(representing the confidence level of relationships between entities). To obtain structured knowledge graph relationships, a Top-K filtering strategy is adopted, selecting the top K relationships with the highest confidence in the adjacency matrix as the final completion result, constructing a high-quality multimodal knowledge graph. This filtering process effectively filters out low-confidence noisy relationships (such as invalid associations and incorrectly recovered relationships) and retains high-confidence valid relationships (including original true relationships and missing relationships in accurate completion). Through this diffusion denoising and completion mechanism, not only is the noise interference problem of the original knowledge graph solved, but the data sparsity defect caused by the lack of long-tail entity associations is also compensated for. This provides high-quality data support for subsequent entity embedding learning and user preference propagation based on graph neural networks, thereby improving the accuracy and robustness of the recommendation model.

[0060] S4. Combining the high-quality knowledge graph and user-item interaction data, the item embedding is obtained by aggregating neighborhood entity features through RGAT. After being injected into the user-item interaction graph, the high-order collaborative filtering signal is captured by GCN, the inner product score between users and items is calculated, and the recommendation list is output.

[0061] Specifically, this step combines RGAT and GCN to achieve dual feature capture. RGAT is used to model heterogeneous relationships in the knowledge graph, assigning differentiated weights to different types of relationships through an attention mechanism, accurately aggregating neighborhood entity features, and highlighting the impact of important relationships on entity representation; GCN is used to capture user-item interaction patterns, aggregating neighborhood interaction features of users or items through simple and efficient graph convolution operations, capturing high-order collaborative filtering signals, and mining potential user preferences.

[0062] Specifically, S4 includes the following steps: S41, on a high-quality knowledge graph, uses RGAT for any triple. Calculate the attention coefficient under the correspondence, and apply the Softmax function to the center node. The attention coefficients within the neighborhood are normalized to obtain the aggregation weights; wherein, in the triples As the central node, For neighboring nodes, It is a relation type.

[0063] Specifically, in the denoised knowledge graph reconstructed by the diffusion model The model employs RGAT (Relational Graph Attention Network) to aggregate neighborhood features. RGAT can handle the heterogeneity in the graph, that is, it assigns differentiated attention weights to different types of relations (such as "has_image", "has_text", or ordinary entity relations). For any triple in the graph... in As the central node, For neighboring nodes, For relation types, RGAT first calculates the attention coefficient for a specific relation. The formula is:

[0064] in, Represents a node Features Represents a node Features For relational embedding, This represents vector concatenation. This is a learnable projection matrix. The Softmax function is used to optimize the neighborhood. The attention coefficients within the cluster are normalized to obtain the final aggregate weights. .

[0065] S42, based on the aggregation weight, the features of neighboring nodes are aggregated by weighted summation, and residual connections and normalization and dropout operations are introduced to update the representation of the central node, resulting in an item embedding rich in knowledge semantics.

[0066] In this embodiment, neighbor information is aggregated using a weighted summation method, and residual connections are introduced to prevent gradient vanishing. The node representation is then updated using the following formula:

[0067] RGAT output of knowledge-rich semantic item embeddings That is, the item is embedded.

[0068] S43, the item is embedded into the user-item interaction graph, and a GCN layer is used to perform graph convolution operation on the bipartite graph to aggregate the multi-hop neighbor features of users or items and capture high-order collaborative filtering signals.

[0069] Specifically, embedding items Injected into the user-item interaction graph, a GCN (Graph Convolutional Network) layer is used to capture high-order collaborative filtering signals. By performing lightweight graph convolution operations on the bipartite graph, multi-hop neighbor features of users or items are aggregated.

[0070] S44: Calculate the inner product score between users and items, and output a recommendation list based on the scores.

[0071] Specifically, based on the user embeddings and item embeddings obtained from the above steps, the matching score between the user and each candidate item is calculated through inner product operation. The higher the score, the stronger the user's preference for that item. The candidate items are sorted from high to low according to their scores, and the top-ranked items are selected to form the final recommendation list, thus achieving accurate personalized recommendations based on multimodal knowledge graphs.

[0072] Example 2 This invention also provides a personalized recommendation device based on a multimodal knowledge graph, such as... Figure 4 As shown, the device 10 includes: The data processing and multimodal knowledge graph construction module 100 is used to obtain public datasets and text, image and audio files corresponding to items. After filtering invalid files, it extracts feature vectors of each modality through BERT, ResNet and VGGish models respectively, adds multimodal sub-entities for each item and establishes association relationships, and constructs intramodal similarity relationships based on cosine similarity to form a multimodal knowledge graph. The dynamic weighted multimodal fusion module 200 is used to map the feature vectors of each modality to a unified dimension through a fully connected layer, determine dynamic weights through a cross-modal attention mechanism and a gating network, and obtain the fused features by weighted summation; The diffusion model denoising and completion module 300 is used to gradually add noise to the multimodal knowledge graph based on the diffusion model using a linear noise scheduler, guided by the fusion features, and iteratively denoising through a denoising network, and obtain a high-quality knowledge graph through Top-K screening. The graph network representation learning module 400 is used to combine the high-quality knowledge graph with user-item interaction data, aggregate neighborhood entity features through RGAT to obtain item embeddings, inject them into the user-item interaction graph, capture high-order collaborative filtering signals through GCN, calculate the inner product score between users and items, and output a recommendation list.

[0073] Example 3 To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the methods described above.

[0074] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A personalized recommendation method based on multimodal knowledge graphs, characterized in that, Includes the following steps: S1. Obtain the public dataset and the text, image and audio files corresponding to the items. After filtering out invalid files, extract the feature vectors of each modality using BERT, ResNet and VGGish models respectively. Add multimodal sub-entities for each item and establish association relationships. Construct intramodal similarity relationships based on cosine similarity to form a multimodal knowledge graph. S2, the feature vectors of each modality are mapped to a unified dimension through a fully connected layer, and dynamic weights are determined through a cross-modal attention mechanism and a gating network. The weighted sum is then used to obtain the fused features. S3, Based on the diffusion model, noise is gradually added to the multimodal knowledge graph using a linear noise scheduler. With the fusion features as semantic guidance, noise is iteratively removed through a denoising network, and a high-quality knowledge graph is obtained through Top-K screening. S4. Combining the high-quality knowledge graph and user-item interaction data, the item embedding is obtained by aggregating neighborhood entity features through RGAT. After being injected into the user-item interaction graph, the high-order collaborative filtering signal is captured by GCN, the inner product score between users and items is calculated, and the recommendation list is output.

2. The method according to claim 1, characterized in that, The process involves adding a multimodal sub-entity to each item and establishing associations, constructing intra-modal similarity relationships based on cosine similarity, and forming a multimodal knowledge graph, including: Add text sub-entities, image sub-entities, and audio sub-entities for each item, and generate sub-entity IDs using the base entity ID + modal offset rule; Establish entity-sub-entity relationships between item entities and text sub-entities, image sub-entities, and audio sub-entities to achieve structured association between items and multimodal features; For text, image, and audio modalities, calculate the cosine similarity of feature vectors of any two items of the same modality. Select the top K feature pairs with the highest cosine similarity in each modality, and establish intramodal similarity relationships for SIM_TEXT, SIM_IMG, and SIM_AUDIO to enhance semantic association across items; By integrating the entity-sub-entity relationships, intramodal similarity relationships, and original entity relationships of items, a multimodal knowledge graph is formed.

3. The method according to claim 1, characterized in that, The process of mapping the feature vectors of each modality to a unified dimension through a fully connected layer, determining dynamic weights through a cross-modal attention mechanism and a gating network, and then weighted summing to obtain the fused features includes: The original features of text, image and audio modalities are nonlinearly mapped by independent fully connected layers. The GeLU activation function is used to process the output of the fully connected layers, and the features of each modality are uniformly transformed into a 128-dimensional latent semantic space to obtain the projected features of the three modalities. The three types of modal features after projection are stacked and combined, and input into a multi-head self-attention mechanism. The stacked features themselves are used as query Q, key K, and value V to perform deep interaction calculation between modalities, resulting in a feature matrix containing global multimodal context information. The feature matrix containing global multimodal context information is flattened and then input into the gated network. The dynamic weights of each modality are calculated through a fully connected layer and normalized by Softmax. Based on the dynamic weights, the three types of modal features after projection are weighted and summed to obtain the fused features.

4. The method according to claim 1, characterized in that, The diffusion-based model involves progressively adding noise to the multimodal knowledge graph using a linear noise scheduler. Guided by the fused features, a denoising network iteratively removes noise, and a high-quality knowledge graph is obtained through Top-K filtering, including: A linear noise scheduler is used to gradually add standard Gaussian noise and modal feature projection noise to the adjacency matrix of the multimodal knowledge graph within T time steps, following the Markov chain rule. The noise intensity increases linearly with time step. The fusion features are used as conditions to constrain the input denoising network. Starting from the noise adjacency matrix at step T, the noise is predicted and removed step by step through the denoising network. The contribution of the prediction results guided by multimodal features and the prediction results without conditions to the prediction output of the denoising network is balanced by the dynamic gating weights at time steps. The reverse denoising process yields a complete adjacency matrix whose elements are relation confidence scores. A Top-K selection strategy is then used to select the top K relations with the highest confidence scores to construct a high-quality knowledge graph.

5. The method according to claim 1, characterized in that, The process involves combining the high-quality knowledge graph with user-item interaction data, aggregating neighborhood entity features using RGAT to obtain item embeddings, injecting these embeddings into the user-item interaction graph, capturing high-order collaborative filtering signals via GCN, calculating the inner product score between users and items, and outputting a recommendation list, including: On high-quality knowledge graphs, RGAT is used for any triples. Calculate the attention coefficient under the correspondence, and apply the Softmax function to the center node. The attention coefficients within the neighborhood are normalized to obtain the aggregation weights; wherein, in the triples As the central node, For neighboring nodes, It is a relation type; Based on the aggregation weights, the features of neighboring nodes are aggregated by weighted summation, and residual connections and normalization and dropout operations are introduced to update the representation of the central node, resulting in an item embedding rich in knowledge semantics. The items are embedded into the user-item interaction graph, and a GCN layer is used to perform graph convolution operations on the bipartite graph to aggregate the multi-hop neighbor features of users or items and capture high-order collaborative filtering signals. Calculate the inner product score between users and items, and output a recommendation list based on the scores.

6. A personalized recommendation device based on a multimodal knowledge graph, characterized in that, include: The data processing and multimodal knowledge graph construction module is used to obtain public datasets and text, image, and audio files corresponding to items. After filtering out invalid files, it extracts feature vectors of each modality using BERT, ResNet, and VGGish models respectively, adds multimodal sub-entities for each item and establishes association relationships, and constructs intramodal similarity relationships based on cosine similarity to form a multimodal knowledge graph. The dynamic weighted multimodal fusion module is used to map the feature vectors of each modality to a unified dimension through a fully connected layer, determine the dynamic weights through a cross-modal attention mechanism and a gating network, and obtain the fused features by weighted summation; The diffusion model denoising and completion module is used to gradually add noise to the multimodal knowledge graph based on the diffusion model using a linear noise scheduler, guided by the fused features, and iteratively denoising through a denoising network, and obtain a high-quality knowledge graph through Top-K screening; The graph network representation learning module combines the high-quality knowledge graph with user-item interaction data, aggregates neighborhood entity features through RGAT to obtain item embeddings, injects them into the user-item interaction graph, captures high-order collaborative filtering signals through GCN, calculates the inner product score between users and items, and outputs a recommendation list.

7. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a personalized recommendation method based on multimodal knowledge graph as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a personalized recommendation method based on a multimodal knowledge graph as described in any one of claims 1-5.

Images