A user side multi-modal recommendation method based on knowledge fusion agent

Abstract

This invention relates to the field of recommender system technology and provides a user-side multimodal recommendation method based on a knowledge fusion intelligent agent. The method includes: acquiring user-item interaction data, a collaboration graph, and original multimodal data of items, and encoding them to obtain item multimodal embeddings; constructing a user-side interest knowledge graph and an item-side category knowledge graph, fusing them to form a user-side knowledge graph, an item-side knowledge graph, and a knowledge-enhanced collaboration knowledge graph, and encoding them to obtain corresponding views; fusing the collaboration graph and item multimodal embeddings to obtain a multimodal collaboration view, performing hierarchical reconstruction processing and self-supervised alignment to obtain a multimodal knowledge view; and performing adaptive heterogeneous fusion processing on the user-side view, item-side view, and multimodal knowledge view to obtain a heterogeneous fused view, based on which a preference prediction score is calculated to complete the recommendation. This invention can improve the granularity of user representation, enhance the robustness of multimodal representation, and achieve closed-loop optimization of the fusion strategy.

Description

Technical Field

[0001] This invention belongs to the field of recommendation system technology and provides a user-side multimodal recommendation method based on knowledge fusion intelligent agents. Background Technology

[0002] With the rapid development of internet technology and the explosive growth of digital content, personalized recommendation systems have become a core technology for solving information overload, improving user experience, and driving platform commercial value. They are widely used in e-commerce, short videos, news and information, online education and many other fields.

[0003] Early recommender systems relied on collaborative filtering, which predicted user preferences by mining historical interaction data between users and items. This approach was widely adopted due to its simplicity and stable performance. However, collaborative filtering inherently suffers from data sparsity and cold-start limitations; recommendation accuracy drops sharply when user interaction data is insufficient. To address this, multimodal recommender technology emerged. This technology integrates multimodal content information such as images, text, and audio of items, providing rich semantic supplementation to the recommender system and effectively mitigating the data sparsity problem. In recent years, knowledge graph technology has been introduced into the recommender field. By explicitly modeling the semantic relationships between users, items, and various entities, it further improves the accuracy and interpretability of recommendations, forming the mainstream research direction of knowledge-enhanced multimodal recommender.

[0004] However, existing knowledge-enhanced multimodal recommender systems still suffer from a fundamental unresolved technical problem: the collaborative modeling and adaptive fusion capabilities of multi-source heterogeneous information are severely insufficient. Specifically, collaborative information generated by user-item interactions, multimodal content information of items, and semantic knowledge information composed of user interests and item categories belong to different information spaces, with different data structures, feature distributions, and semantic connotations. The inability to dynamically adjust the fusion weights of each information source according to the actual scenario makes it difficult for the comprehensive performance of the recommender system to meet the requirements in complex real-world scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a user-side multimodal recommendation method based on a knowledge fusion intelligent agent. By constructing a user-side interest knowledge graph and an item-side category knowledge graph, and combining a hierarchical diffusion model with a knowledge fusion intelligent agent, user-side multimodal personalized recommendations are achieved.

[0006] The technical solution of this invention includes: acquiring the original multimodal data of all items, user-item interaction data, and user-item collaboration graph; encoding the original multimodal data of the items to obtain item multimodal embeddings; constructing user-side interest knowledge graphs and item-side category knowledge graphs based on the user-item interaction data and item multimodal embeddings respectively; fusing the user-side interest knowledge graphs and user-item collaboration graphs to form a user-side knowledge graph; fusing the item-side category knowledge graphs and user-item collaboration graphs to form an item-side knowledge graph; and fusing the user-side interest knowledge graphs, item-side category knowledge graphs, and user-item collaboration graphs together to form a knowledge-enhanced collaborative knowledge graph; encoding the user-side knowledge graphs, item-side knowledge graphs, and knowledge-enhanced collaborative knowledge graphs to obtain user-side views, item-side views, and knowledge-enhanced collaborative views; and encoding the user-item collaboration graphs to obtain user-side views, item-side views, and knowledge-enhanced collaborative views. A multimodal collaborative view is obtained by fusing the user-item collaborative graph with multimodal embeddings of the same graph and items; a collaborative view is obtained by independently encoding the user-item collaborative graph; a multimodal collaborative aggregated representation is obtained by aggregating the collaborative view with the multimodal collaborative view; a reconstructed multimodal collaborative view is obtained by performing hierarchical reconstruction processing on the multimodal collaborative view; the multimodal collaborative aggregated representation and the reconstructed multimodal collaborative view are self-supervised aligned, and features are fused with the user-side view, item-side view and knowledge-enhanced collaborative view to obtain a multimodal knowledge view; the user-side view, item-side view and multimodal knowledge view are adaptively heterogeneously fused to obtain a heterogeneous fused view; global user representation and global item representation are extracted from the heterogeneous fused view; based on the global user representation and global item representation, the target user's preference prediction score for each candidate item is calculated to complete the user-side multimodal personalized recommendation.

[0007] Optionally, the encoding process includes: splitting the original multimodal data of all items to obtain visual modal raw data and text modal raw data corresponding to each item; wherein, the visual modal raw data is item image data, and the text modal raw data is item description text data; using a pre-trained visual encoder to extract features and map dimensions of the item image data to generate a fixed-dimensional item visual embedding vector; using a pre-trained text encoder to extract features and map dimensions of the item description text data to generate an item text embedding vector aligned with the dimension of the item visual embedding vector; performing mean aggregation on the item visual embedding vector and item text embedding vector corresponding to the same item to generate a unified item multimodal embedding for that item; and traversing all items to generate item multimodal embeddings corresponding to all items.

[0008] Optionally, the method for constructing the user-side interest knowledge graph includes: extracting the rating weights of target users for the items they interact with based on the user-item interaction data; performing normalized weighted aggregation on the multimodal embeddings of the items corresponding to the user's interactive items based on the rating weights to generate single-user preference features corresponding to the user; traversing all users to generate a full set of user preference features; performing clustering processing on the full set of user preference features using a clustering function to divide it into multiple user interest categories; and constructing a user-side interest knowledge graph representing the association between users and corresponding interest categories using users and user interest categories as basic semantic units.

[0009] Optionally, the method for constructing the item-side category knowledge graph includes: performing clustering processing on the multimodal embeddings of all items using a clustering function to divide them into multiple item category entities; and constructing an item-side category knowledge graph representing the association between items and corresponding category entities using items and item category entities as basic semantic units.

[0010] Optionally, the hierarchical reconstruction processing method includes: performing a forward diffusion process using a hierarchical diffusion model, gradually adding Gaussian noise to the multimodal collaborative view according to a preset noise scale to generate noise perturbation views at different time steps until a view conforming to a pure noise distribution is obtained; performing a reverse diffusion process using a hierarchical diffusion model, predicting the mean and variance of the noise distribution using a parameterized neural network, and gradually recovering from the pure noise distribution view to obtain a denoised reconstructed multimodal collaborative view; using a reconstruction loss to constrain the reconstructed multimodal collaborative view to retain the core feature information of the original multimodal collaborative view, and using a semantic transformation loss to constrain the reconstructed multimodal collaborative view to align with the semantic space of the collaborative view.

[0011] Optionally, the self-supervised alignment and feature fusion include: constructing a similarity constraint between a multimodal collaborative aggregation representation based on cosine similarity and a reconstructed multimodal collaborative view to obtain a self-supervised alignment loss; performing multi-view semantic alignment between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view with the goal of minimizing the self-supervised alignment loss; and performing feature fusion by combining the knowledge-enhanced collaborative view, the user-side view, and the item-side view to obtain a multimodal knowledge view.

[0012] Optionally, the adaptive heterogeneous fusion processing includes: concatenating the user-side view and the multimodal knowledge view into a matrix and inputting them into a first multilayer perceptron to calculate user-side gating weights; concatenating the item-side view and the multimodal knowledge view into a matrix and inputting them into a second multilayer perceptron to calculate item-side gating weights; performing weighted aggregation on the user-side view and the multimodal knowledge view according to the user-side gating weights to generate a preliminary user-side fusion representation; performing weighted aggregation on the item-side view and the multimodal knowledge view according to the item-side gating weights to generate a preliminary item-side fusion representation; concatenating the preliminary user-side fusion representation and the preliminary item-side fusion representation and obtaining a preliminary fusion representation through a feedforward layer; introducing a feature attention module to perform fine-grained feature adaptive fusion on the user-side view, the item-side view, the multimodal knowledge view, and the preliminary fusion representation to obtain a fine-grained fusion result; and sequentially performing linear projection and layer normalization processing on the fine-grained fusion result to output a heterogeneous fusion view.

[0013] Optionally, multimodal recommendation on the user side is performed through a recommendation model. Before performing multimodal recommendation on the user side, the recommendation model is pre-trained offline jointly. The recommendation model includes a graph neural network encoding module, a hierarchical diffusion module, and a knowledge fusion agent. The offline joint training method includes: constructing an end-to-end full-link training network, which is used to generate user-side views, item-side views, knowledge-enhanced collaborative views, multimodal knowledge views, and heterogeneous fusion views; and constructing a joint total loss function, which includes Bayesian personalized ranking loss, reconstruction loss, and self-supervised loss. Alignment loss, semantic transformation loss, consistency constraint term between heterogeneous fused view and multimodal knowledge view, oversmoothing regularization term, and parameter regularization term; based on user-item interaction training data, item multimodal data, and user-side interest knowledge graph and item-side category knowledge graph constructed accordingly, with minimizing the joint total loss function as the optimization objective, all learnable parameters of the graph neural network encoding module, hierarchical diffusion module, and knowledge fusion agent are jointly trained; when the joint total loss function converges to a preset threshold or reaches a preset number of iterations, iterative training is stopped, and the parameters of each module after training are fixed.

[0014] Optionally, the training method of the knowledge fusion agent is as follows: the multimodal knowledge view is defined as the environment state, the fusion weight adjustment action of the knowledge fusion agent is defined as the policy action, and a reward function is constructed based on the cosine similarity and semantic consistency between the multimodal knowledge view and the heterogeneous fusion view; the policy network generates the policy action for the current round based on the current environment state, executes the policy action to complete the adaptive adjustment of the fusion weight and generate the corresponding heterogeneous fusion view, and calculates the reward value corresponding to the policy action for the current round according to the reward function; with the goal of maximizing the cumulative reward value, all learnable parameters of the policy network and the value network are updated by jointly using the policy loss and value loss, and the policy action for the next round is generated based on the updated network, forming a closed-loop iterative link to continuously optimize the adaptive heterogeneous fusion strategy of the knowledge fusion agent; when the cumulative reward value converges to a preset threshold, the iteration stops, and the training of the knowledge fusion agent is completed.

[0015] As one implementation, the self-supervised alignment loss includes a similarity transition loss constructed based on the cosine similarity between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view.

[0016] The technical solution provided by this invention has the following advantages compared with the prior art: This invention constructs and merges a user-side interest knowledge graph and an item-side category knowledge graph to form a user-side knowledge graph, an item-side knowledge graph, and a knowledge-enhanced collaborative knowledge graph. This achieves unified collaborative modeling of user interests, item category semantic knowledge, and user-item collaborative information, effectively solving the problem of imbalance between user-side and item-side information modeling in existing technologies and significantly improving the granularity of user representation. By fusing the user-item collaborative graph with multimodal item embeddings to obtain a multimodal collaborative view, and performing hierarchical reconstruction and self-supervised alignment, the robustness of multimodal and collaborative information fusion is enhanced, effectively suppressing the interference of noise in multimodal data on recommendation results. By performing adaptive heterogeneous fusion processing on the user-side view, item-side view, and multimodal knowledge view, the limitations of the static and fixed fusion mechanism in existing technologies are broken. It can dynamically adjust the fusion weights of each information source according to different scenarios, greatly improving the scenario adaptability of the recommendation system. Overall, a complete framework for multi-source heterogeneous information collaborative modeling and fusion is constructed, comprehensively improving the overall recommendation performance of the recommendation system in complex real-world scenarios.

[0017] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0020] Figure 2 for Figure 1 This is a diagram illustrating the knowledge-aware multimodal classifier in [the context of the text].

[0021] Figure 3 for Figure 1 A diagram illustrating the knowledge fusion intelligent agent.

[0022] Figure 4 for Figure 1 A diagram illustrating the construction of a multimodal knowledge view in [the context of the diagram]. Detailed Implementation

[0023] The following detailed description of a specific embodiment of the present invention is provided in conjunction with the accompanying drawings. However, it should be understood that the scope of protection of the present invention is not limited to the specific embodiment.

[0024] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the technical solution of this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] In the description of the embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more.

[0026] Before describing the embodiments, let's define: The user-item collaboration graph is defined as follows: ,in Represents a set of users. ; Represents a collection of items. Interaction matrix Defined as , among which, if and , indicating user By rating For items An evaluation was conducted; and This indicates that there is no interaction between the two.

[0027] Furthermore, each item is associated with multimodal content (such as images and text), and its multimodal embedding (for each item, there is a corresponding modal feature vector for each modality; modality refers to different information sources such as images and text. Then, for the same item, the feature vectors of all its modalities are averaged to obtain the unified multimodal embedding of that item) can be represented as follows: ,in Indicates the modal type.

[0028] In the formula This represents the feature vector of the j-th item in the m-th modality. Indicates the total number of modes. This represents the summation and aggregation of all modal features of the item, resulting in the final value. The unified multimodal embedding of an item is the average representation of the features of each modality.

[0029] Example 1: like Figures 1 to 4 As shown, this embodiment provides a user-side multimodal recommendation method based on a knowledge fusion intelligent agent. This method constructs a user-side interest knowledge graph and an item-side category knowledge graph, combining a hierarchical diffusion model and a knowledge fusion intelligent agent to achieve user-side multimodal personalized recommendations. This addresses the technical problems of insufficient user representation refinement, rigid heterogeneous information fusion mechanisms, and weak noise resistance in existing multimodal recommendation methods. Specifically, it includes the following steps:

[0030] Acquire user-item interaction data, user-item collaboration graph, and raw multimodal data of all items, and encode the raw multimodal data of the items to obtain item multimodal embedding.

[0031] Specifically, user-item interaction data records users' historical behaviors towards items (such as clicks, purchases, and ratings). The user-item collaboration graph is a graph structure built upon this interaction data, where nodes represent users or items, and edges represent interaction relationships. The raw multimodal data includes visual modal data (such as product display images and detail images) and textual modal data (such as product titles and detailed descriptions). Encoding aims to transform unstructured multimodal data into a computer-processable vector form. For example, for visual modal data, pre-trained convolutional neural networks (such as ResNet and ViT) can be used to extract image features; for textual modal data, pre-trained language models (such as BERT) can be used to extract textual features. Through encoding, each item obtains an embedding vector with one or more dimensions, providing foundational data support for subsequent construction of knowledge graphs and views.

[0032] Based on the user-item interaction data and item multimodal embedding, a user-side interest knowledge graph and an item-side category knowledge graph are constructed respectively. The user-side interest knowledge graph and the user-item collaboration graph are fused to form a user-side knowledge graph, and the item-side category knowledge graph and the user-item collaboration graph are fused to form an item-side knowledge graph. The user-side interest knowledge graph, the item-side category knowledge graph, and the user-item collaboration graph are then fused together to form a knowledge-enhanced collaboration knowledge graph. The user-side knowledge graph, the item-side knowledge graph, and the knowledge-enhanced collaboration knowledge graph are encoded to obtain a user-side view, an item-side view, and a knowledge-enhanced collaboration view.

[0033] Specifically, the core of this step lies in constructing and fusing knowledge graphs from different dimensions. The user-side interest knowledge graph mines users' potential interest categories through methods such as clustering, establishing "user-interest category" relationships to explicitly characterize user preferences; the item-side category knowledge graph mines item category attributes, establishing "item-category entity" relationships. The fusion process is not a simple graph stitching, but rather the addition of the mined knowledge nodes (such as interest categories and item categories) and their associated edges to the original user-item collaboration graph. The user-side knowledge graph focuses on modeling interest relationships from the user's perspective, the item-side knowledge graph focuses on modeling category relationships from the item's perspective, and the knowledge-enhanced collaboration knowledge graph uniformly models multiple entities such as users, items, interest categories, and item categories, as well as their complex interactions. Subsequently, graph neural networks (such as LightGCN) are used to encode the above three graphs respectively, aggregating neighborhood information through multi-layer propagation, thereby obtaining user-side views, item-side views, and knowledge-enhanced collaboration views containing rich semantic information. It should be understood that although this embodiment uses a graph neural network for encoding, other graph encoding models can be used in other embodiments, as long as graph structure features can be extracted. By constructing a multi-dimensional view, this embodiment achieves fine-grained modeling of user preferences and item attributes, providing a rich source of features for subsequent fusion.

[0034] The user-item collaboration graph is fused with the item multimodal embedding to obtain a multimodal collaboration view; the user-item collaboration graph is independently encoded to obtain a collaboration view; the collaboration view and the multimodal collaboration view are aggregated to obtain a multimodal collaboration aggregate representation; the multimodal collaboration view is subjected to hierarchical reconstruction processing to obtain a reconstructed multimodal collaboration view; the multimodal collaboration aggregate representation and the reconstructed multimodal collaboration view are self-supervised aligned, and feature fusion is performed with the user side view, item side view and knowledge-enhanced collaboration view to obtain a multimodal knowledge view.

[0035] Specifically, this step aims to construct a robust multimodal knowledge view. First, the structural information of the collaboration graph is fused with the multimodal content information of the items to generate a multimodal collaboration view, which includes both collaboration signals and content semantics. Simultaneously, the collaboration graph is independently encoded to obtain a collaboration view containing only collaboration signals. To mitigate the noise interference commonly found in multimodal data, this embodiment introduces a hierarchical diffusion model to reconstruct the multimodal collaboration view. The hierarchical diffusion model progressively adds noise through a forward process and then progressively removes noise through a reverse process, thereby generating a more robust reconstructed multimodal collaboration view. The self-supervised alignment process calculates the similarity constraints between the multimodal collaboration aggregation representation and the reconstructed multimodal collaboration view, forcing them to align in the semantic space, further filtering noise and preserving core features. Finally, the aligned features are fused with the user-side view, item-side view, and knowledge-enhanced collaboration view to generate the final multimodal knowledge view. This view not only integrates multimodal content information but also introduces knowledge-enhanced semantic information and possesses strong noise resistance.

[0036] The user-side view, item-side view, and multimodal knowledge view are adaptively heterogeneously fused to obtain a heterogeneous fused view. The global user representation and global item representation are extracted from the heterogeneous fused view. Based on the global user representation and global item representation, the target user's preference prediction score for each candidate item is calculated to complete the user-side multimodal personalized recommendation.

[0037] Specifically, since the user-side view, item-side view, and multimodal knowledge view originate from different construction processes, their feature distributions and semantic spaces differ, making simple concatenation or weighting insufficient to achieve optimal results. Therefore, this embodiment employs a knowledge fusion agent for adaptive heterogeneous fusion. The knowledge fusion agent dynamically adjusts the fusion weights of each view based on the currently input view features, generating a unified heterogeneous fused view. In the heterogeneous fused view, each user and item corresponds to a global representation vector. The recommendation process calculates the similarity (e.g., dot product) between the global representation of the target user and the global representation of candidate items, obtains a preference prediction score, and generates a recommendation list based on the score. Through adaptive heterogeneous fusion, this embodiment can dynamically balance the importance of collaborative information, knowledge information, and multimodal information according to the characteristics of different users and scenarios, thereby significantly improving recommendation accuracy and scenario adaptability.

[0038] Example 2: This embodiment, based on Embodiment 1, provides a detailed explanation of the encoding and processing procedures for multimodal data, as well as the specific construction methods for user-side interest knowledge graphs and item-side category knowledge graphs. Through specific encoding and graph construction techniques, the original unstructured data is transformed into structured knowledge representations, providing accurate data support for subsequent view fusion and recommendation.

[0039] The original multimodal data of all items is split into visual modal raw data and text modal raw data for each item; the visual modal raw data is the item image data, and the text modal raw data is the item description text data.

[0040] Specifically, in e-commerce platforms or content recommendation scenarios, an item typically contains information in multiple modalities. For example, a product may be associated with product images showcasing its appearance, as well as text details describing its specifications, materials, and functions. This step first decouples this mixed data, separating the pure image data and text data, so that specialized encoding models can be used to process the data according to the characteristics of different modalities. It should be understood that although this embodiment uses visual and text modalities as examples, in other embodiments, the original multimodal data may also include audio data, video data, etc., and this invention does not limit this.

[0041] A pre-trained visual encoder is used to extract features and map dimensions of the object image data to generate a fixed-dimensional object visual embedding vector; a pre-trained text encoder is used to extract features and map dimensions of the object description text data to generate an object text embedding vector that is aligned with the dimensions of the object visual embedding vector.

[0042] The pre-trained visual encoder can employ a convolutional neural network model (such as ResNet or VGG) or a visual Transformer model (such as ViT) pre-trained on a large image dataset (such as ImageNet). The pre-trained text encoder can employ a language model (such as BERT or RoBERTa) pre-trained on a large text corpus. Since the original feature spaces of different modalities differ significantly—for example, image features might be high-dimensional pixel matrices, while text features might be discrete word vectors—direct fusion would lead to information imbalance. Therefore, this embodiment uses dimensionality mapping to project features from different modalities onto the same dimensional space (e.g., 768 dimensions), achieving feature space alignment. This alignment operation eliminates the heterogeneity barrier between modalities, giving subsequent aggregation operations practical physical meaning, i.e., measuring the contribution of different modal information at the same semantic scale.

[0043] The visual embedding vector and text embedding vector of the same item are averaged and aggregated to generate a unified multimodal embedding for that item; the entire set of items is traversed to generate the multimodal embeddings of all items.

[0044] Specifically, mean aggregation calculates the average of the visual embedding vector and the text embedding vector to obtain a comprehensive representation. This processing method is simple and efficient, and can preserve the main semantic information of both modalities. Of course, in addition to mean aggregation, weighted averaging, max pooling, and other aggregation methods can also be used. The generated multimodal embeddings of items not only include the visual appearance features of the items, but also integrate textual semantic features, forming the basic node features for subsequent construction of the knowledge graph.

[0045] Based on user-item interaction data and item multimodal embedding, construct user-side interest knowledge graphs and item-side category knowledge graphs respectively.

[0046] This step is a crucial one in building a knowledge-enhanced view, and it includes two sub-processes: building a user-side interest knowledge graph and building an item-side category knowledge graph.

[0047] The construction of user-side interest knowledge graphs specifically includes: First, based on user-item interaction data, the rating weights of target users for the items they interact with are extracted. Based on the rating weights, the multimodal embeddings of the items corresponding to the user's interactive items are normalized and weighted to generate the single-user preference features corresponding to the user. Then, all users are traversed to generate a full set of user preference features.

[0048] Among them, the rating weight reflects the user's degree of liking for the item. For example, in explicit feedback scenarios, the rating weight can be the specific score given by the user (such as 1-5 points); in implicit feedback scenarios, the rating weight can be the normalized value of behaviors such as the number of clicks and the duration of dwell time. The process of normalized weighted aggregation can be expressed as formula (1):

[0049] Formula Explanation: This formula is used to calculate the target user's preference feature vector in user-side multimodal recommendation, obtained by weighted aggregation of the multimodal embeddings of their interacted items. Where: For users The preference feature vector, For users A collection of interacted items For users For items The scoring weights, For items Multimodal embedding. Through weighted aggregation, user preference features can more accurately reflect the differences in users' attention to different items, rather than simply treating them as averages.

[0050] Secondly, the entire set of user preference features is clustered using a clustering function to divide it into multiple user interest categories; using users and user interest categories as basic semantic units, a user-side interest knowledge graph representing the relationship between users and their corresponding interest categories is constructed.

[0051] The clustering process of the clustering function on the full set of user preference features can be expressed as formula (2): In the formula: For clustering functions, For the user-side multimodal preference feature set, For the number of interest categories, For the division A set of user interest categories .

[0052] Clustering functions can employ algorithms such as K-means and DBSCAN. Through clustering, users with similar preference characteristics are grouped into the same interest category. For example, users with similar preference characteristics might be clustered as "digital enthusiasts" or "beauty experts." In the constructed user-side interest knowledge graph, nodes include user nodes and interest category nodes, with edges representing users belonging to that interest category. This explicit graph structure transforms implicit user preferences into explicit semantic associations, effectively solving the problem of inaccurate representation caused by the sparsity of user behavior data.

[0053] The construction of the item-side category knowledge graph specifically includes: Clustering is performed on the multimodal embeddings of all items using a clustering function to divide them into multiple item category entities. Using items and item category entities as basic semantic units, an item-side category knowledge graph representing the relationship between items and their corresponding category entities is constructed.

[0054] The clustering function for the multimodal embedding of all items can be expressed as formula (3): In the formula, For clustering functions, For the multimodal embedding set on the item side, For the quantity of item categories, For the division A collection of item categories .

[0055] Similarly, clustering functions group items with similar features into one category, forming item category entities (such as "cell phone" and "dress"). The nodes of the item-side category knowledge graph include item nodes and category nodes, with edges indicating that an item belongs to that category. This graph supplements semantic information from the item side, enhances the correlation between items, and helps to uncover potential similarities between items.

[0056] This embodiment achieves the transformation from raw multimodal data to a structured knowledge graph through the above steps. Pre-trained encoders and dimensional alignment ensure feature quality, weighted aggregation highlights key user preferences, and clustering algorithms uncover potential semantic categories. The constructed bilateral knowledge graph not only enriches the semantic information of the data but also provides a solid foundation of entities and relationships for subsequent knowledge-enhanced collaborative knowledge graph construction, thereby significantly improving the recommendation system's ability to understand user interests and item attributes.

[0057] Example 3: This embodiment, based on Embodiment 1, provides a detailed explanation of the construction process of a multimodal knowledge view. Specifically, this embodiment uses a hierarchical diffusion model to reconstruct and semantically align the multimodal collaborative view, thereby constructing a multimodal knowledge view with strong noise resistance and high semantic consistency.

[0058] The user-item collaboration graph is fused with the multimodal embedding of items to obtain a multimodal collaboration view; the user-item collaboration graph is independently encoded to obtain the collaboration view.

[0059] Specifically, the construction of the multimodal collaborative view aims to fuse collaborative signals (user-item interaction relationships) with content signals (visual and textual features of items) at an early stage. This embodiment uses a graph neural network to propagate multimodal embeddings of items into the user-item collaborative graph, ensuring that nodes in the graph contain not only structural information but also rich multimodal semantic information. The collaborative view, on the other hand, is a purely structured encoding of the original user-item collaborative graph, retaining only the topological relationships between users and items and excluding multimodal content information. This dual-view design provides a foundation for subsequent semantic alignment: the multimodal collaborative view contains rich semantics but is susceptible to noise, while the collaborative view has a clear structure but relatively sparse semantics.

[0060] Aggregating the collaborative view and the multimodal collaborative view yields a multimodal collaborative aggregate representation, and performing hierarchical reconstruction processing on the multimodal collaborative view yields a reconstructed multimodal collaborative view.

[0061] One of the core innovations of this embodiment is the hierarchical reconstruction process, which aims to mitigate noise interference in multimodal data by leveraging the generative capabilities of the diffusion model. Specifically, it includes both forward and reverse diffusion processes.

[0062] During the forward diffusion process, Gaussian noise is gradually added to the multimodal collaborative view according to a preset noise scale using a hierarchical diffusion model to generate noise perturbation views at different time steps until a view conforming to the pure noise distribution is obtained.

[0063] The conditional probability distribution of the forward diffusion process can be expressed as formula (4): Formula Explanation: This formula describes the conditional probability distribution of the forward noise addition process in a diffusion model, i.e., from the initial state... By gradually adding noise, the first Step-in-noise state The distribution of . Where: For the first Noisy data samples of the step, This is a sample of the original input data. Represents the Gaussian distribution function. This is the cumulative signal-to-noise ratio coefficient. For the first Noise scheduling parameters for each step It is the identity matrix. Indicates a time step. Represents the identity matrix. Control the first The Gaussian noise scale is added step by step. When hour, It will converge to a pure noise distribution.

[0064] Specifically, the diffusion forward process is a gradual destruction of the data structure. Assuming a multimodal collaborative view as the initial state, at each time step, the model adds Gaussian noise to the view according to a preset noise scale. As the time steps increase, the features of the original view are gradually masked by the noise, eventually becoming a pure noise view conforming to a standard Gaussian distribution. The physical significance of this process is that, through controlled noise addition, the original data is mapped to a noise space that is easy to model, providing a starting point for subsequent inverse recovery. It should be understood that the preset noise scale can be adjusted according to the noise level of the dataset; the larger the noise scale, the stronger the model's robustness to noise, but the more difficult it is to recover the original information.

[0065] In the diffusion reverse process, the mean and variance of the noise distribution are predicted by a parameterized neural network through a hierarchical diffusion model, and the noise distribution is gradually recovered from the pure noise distribution view to obtain the reconstructed multimodal collaborative view after noise reduction.

[0066] The conditional probability distribution of the reverse diffusion process can be expressed as formula (5): Formula Explanation: This formula describes the conditional probability distribution of the inverse denoising process in a diffusion model, i.e., from the current noisy state... Gradually reduce noise and restore the previous state. The distribution of . Where: Indicates that given the current state The state at the previous moment under the condition The conditional probability density, Represents the Gaussian distribution function. This is the denoised mean vector predicted by the neural network. The covariance matrix predicted by the neural network. Represents the learnable neural network parameters, mean. and variance By parameters Neural network prediction, The mean is The covariance matrix is The Gaussian distribution.

[0067] Specifically, the diffusion inverse process is the reverse of the forward process and is also a generative process. The model, through a trained parametric neural network (such as U-Net), predicts the noise distribution in the current view at each time step and performs denoising accordingly, gradually recovering the features of the original view. Through this "noise addition-denoising" cycle, the model can learn the core feature distribution of the multimodal collaborative view, thereby automatically filtering out random noise and irrelevant interference in the original data when generating the reconstructed view, generating a more robust feature representation.

[0068] To constrain the quality of the reconstructed multimodal collaborative view, this embodiment introduces a dual loss function: the reconstruction loss constrains the reconstructed multimodal collaborative view to retain the core feature information of the original multimodal collaborative view, and the semantic transformation loss constrains the reconstructed multimodal collaborative view to align with the semantic space of the collaborative view.

[0069] The dual loss function can be expressed as formula (6): Formula Explanation: This formula is used to calculate the contrastive learning loss of the diffusion model, optimizing the model parameters by minimizing the mean squared error between the denoising network's predictions and the actual initial data. Where: To compare the learning loss function, Indicates time step In the interval Find the expectation of a uniform distribution on the surface. Indicates by parameters Defined denoising neural network, For raw multimodal data In the A noisy version at each time step. The true distribution of the original data, For collaborative view As a semantic alignment target This represents the square of the Euclidean distance. The first term... It is a reconstruction loss term used to approximate the multimodal collaborative view. To prevent information loss; the second item It is a semantic transformation term, in which It represents a collaborative view, guiding the reconstruction results to align with the collaborative semantic space.

[0070] Specifically, the reconstruction loss ensures that the reconstructed view does not deviate too far from the original view, preserving the detailed information of the multimodal content; the semantic transformation loss guides the reconstructed view to converge towards the semantic space of the collaborative view, ensuring that the semantic information in the reconstructed view is consistent with the collaborative signals. This dual constraint mechanism not only guarantees the integrity of information but also achieves semantic alignment, effectively solving the problem of semantic drift in multimodal data.

[0071] The multimodal collaborative aggregation representation and reconstructed multimodal collaborative view are self-supervised aligned, and feature fusion is performed with the user side view, item side view and knowledge-enhanced collaborative view to obtain a multimodal knowledge view.

[0072] This step further enhances the consistency between views through self-supervised learning. Specifically, it includes: constructing a multimodal collaborative aggregation representation based on cosine similarity and obtaining a self-supervised alignment loss by constraining the similarity between the reconstructed multimodal collaborative view; performing multi-view semantic alignment between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view with the goal of minimizing the self-supervised alignment loss; and fusing features from the knowledge-enhanced collaborative view, user-side view, and item-side view to obtain a multimodal knowledge view.

[0073] The similarity transition loss in self-supervised alignment loss can be expressed as formula (7): Formula Explanation: This formula calculates the similarity transition loss between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view, achieving semantic alignment of the multi-view through cosine similarity constraints. Where: For the first Similarity transition loss between modalities This is the cosine similarity calculation function. For entity feature vectors in multimodal collaborative aggregation representation, For entity features in the first Mapping vectors under various modes For image feature vectors in multimodal collaborative aggregation representation, To reconstruct the multimodal collaborative view in the first Predicted mapping vectors for each modality.

[0074] By minimizing this loss, the model encourages multimodal collaborative aggregation representations and reconstructed multimodal collaborative views to increase their similarity, thereby effectively addressing data sparsity.

[0075] Specifically, the core idea of ​​self-supervised alignment is to narrow the feature distance of the same object across different views. Multimodal collaborative aggregation representations fuse collaborative signals and multimodal content signals, while the reconstructed multimodal collaborative view is a robust feature processed by a diffusion model for noise reduction. By calculating the cosine similarity between the two and constructing a self-supervised alignment loss, the model can be forced to learn feature representations that are both semantically rich and noise-resistant. When the self-supervised alignment loss is minimized, the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view are highly aligned in the semantic space, thus achieving complementarity and enhancement of multi-view information. Finally, this aligned feature is fused with the user-side view, the item-side view, and the knowledge-enhanced collaborative view to generate the final multimodal knowledge view. This view not only integrates multi-dimensional information such as user preferences, item attributes, collaborative interactions, and multimodal content, but also effectively suppresses noise interference through the diffusion model and self-supervised alignment mechanism, providing high-quality data input for subsequent knowledge fusion agents.

[0076] This embodiment constructs a high-quality multimodal knowledge view through a hierarchical diffusion model and a self-supervised alignment mechanism. The generative capability of the diffusion model effectively mitigates noise interference in multimodal data, the dual loss function ensures the integrity of information and semantic consistency, and the self-supervised alignment further enhances the feature associations between views. This process significantly improves the robustness and recommendation accuracy of the recommendation system in scenarios with sparse data and noise interference.

[0077] Example 4: This embodiment, based on Embodiment 1, provides a detailed explanation of the adaptive heterogeneous fusion processing procedure for the knowledge fusion intelligent agent. Specifically, this embodiment achieves dynamic adaptive fusion of the user side view, the item side view, and the multimodal knowledge view through a combination of gating mechanism and feature attention module, solving the problems of information redundancy and semantic conflicts caused by direct splicing of heterogeneous features.

[0078] The user-side view and the multimodal knowledge view are matrix-concatenated and input into the first multilayer perceptron to calculate the user-side gating weights; the item-side view and the multimodal knowledge view are matrix-concatenated and input into the second multilayer perceptron to calculate the item-side gating weights.

[0079] Specifically, since the user-side view focuses on modeling the user's explicit interests and preferences, the item-side view focuses on modeling the item's category attributes, and the multimodal knowledge view integrates collaborative signals and multimodal content information, the feature distributions and semantic spaces of the three differ significantly. Using simple vector concatenation or average pooling can lead to mutual interference between information from different views, even obscuring key features. Therefore, this embodiment introduces a gating mechanism to dynamically control the flow rate of information. The calculation process of the gating weight can be expressed as formula (8):

[0080] Formula Explanation: This formula is used to calculate the gating weights on the user side and the object side in a knowledge fusion agent, dynamically controlling the information flow ratio through a multilayer perceptron and a sigmoid activation function. Where: For user-side gating weights, For item-side gating weights, It is the Sigmoid activation function. and These are the first multilayer perceptron and the second multilayer perceptron, respectively. This represents a matrix concatenation operation. For the user's side view, This is a side view of the item. It is a multimodal knowledge view.

[0081] In the formula, , , These represent the user side view, the item side view, and the multimodal knowledge view, respectively. This represents a matrix concatenation operation. and These represent the first multilayer perceptron and the second multilayer perceptron, respectively. The sigmoid activation function maps the output to the (0,1) interval. It should be understood that although this embodiment uses a multilayer perceptron (MLP) as the gating network, other neural network structures can be used in other embodiments, such as single-layer fully connected networks or deep networks containing residual connections, as long as a non-linear mapping from spliced ​​features to scalar weights can be achieved. The physical meaning of the gating weights is to automatically determine the importance of the multimodal knowledge view to the user side or the item side based on the specific content of the current input view. For example, for cold-start users, where user-side view information is relatively sparse, the gating mechanism may assign higher weights to the multimodal knowledge view to supplement content information; for active users with rich behavior, the gating mechanism may focus more on the historical preference information of the user-side view.

[0082] Based on the user-side gating weights, the user-side view and the multimodal knowledge view are weighted and aggregated to generate a preliminary user-side fused representation; based on the item-side gating weights, the item-side view and the multimodal knowledge view are weighted and aggregated to generate a preliminary item-side fused representation.

[0083] Specifically, the weighted aggregation process can be expressed as formula (9): Formula Explanation: This formula is used to calculate the initial fused representations of the user side and the item side in a knowledge fusion agent. It uses gating weights to weighted aggregate the original view and the multimodal knowledge view. Where: This represents the initial integration on the user side. This represents the initial fusion of the items. For user-side gating weights, For item-side gating weights, For the user's side view, This is a side view of the item. It is a multimodal knowledge view.

[0084] In the formula, This represents the initial integration on the user side. This provides an initial fusion representation for the item side. In this way, the model can adaptively balance the contributions of user-side (or item-side) features and multimodal knowledge features. When the gating weight is close to 1, it indicates a greater tendency to retain features from the original view; when the gating weight is close to 0, it indicates a greater tendency to adopt features from the multimodal knowledge view. This dynamic weighting mechanism effectively avoids the shortcomings of static fusion strategies that cannot adapt to different data distribution scenarios.

[0085] The preliminary fusion representations on the user side and the item side are spliced ​​together, and a preliminary fusion representation is obtained through a feedforward layer.

[0086] The calculation process of the feedforward layer can be expressed as formula (10): Formula Explanation: This formula is used to calculate the updated embedding representation after fusing user-side and item-side features. A feedforward layer maps the concatenated high-dimensional vector to a unified feature space. Where: This is a preliminary fused representation of the feedforward layer output. To modify the activation function of the linear unit, For normalization operations (such as LayerNorm). For learnable weight matrix, For bias vectors, This represents the initial integration on the user side. This represents the initial fusion of the items. This indicates a vector concatenation operation.

[0087] Specifically, the feedforward layer typically includes linear transformations and nonlinear activation functions (such as ReLU). Its role is to map the concatenated high-dimensional vectors to a unified feature space and further extract the interaction features between the user and the item. This step integrates the preliminary fusion information from the user side and the item side, providing a foundation for subsequent fine-grained fusion.

[0088] A feature attention module is introduced to perform fine-grained feature adaptive fusion on the user side view, the item side view, the multimodal knowledge view, and the preliminary fusion representation, so as to obtain fine-grained fusion results.

[0089] Specifically, steps S401 to S403 mainly perform macro-level fusion at the view level, while the feature attention module further delves into the feature dimensions for micro-level fusion. Different feature dimensions in different views contribute differently to the final recommendation. For example, some dimensions may primarily encode the user's long-term interests, while others may primarily reflect the visual appeal of the item. The feature attention module calculates attention weights and performs a weighted summation of the feature dimensions; its calculation process can be expressed as formula (11):

[0090] Formula Explanation: This formula is used to calculate the weighted aggregate representation of the feature attention module in a knowledge fusion agent. It performs fine-grained adaptive fusion of features from multiple views using attention weights normalized by Softmax. Where: The fused feature vector For the first The characteristic representation of a view For the first Attention weights for each view For normalization function, For the output layer, a multilayer perceptron. For the user's side view, This is a side view of the item. For a multimodal knowledge view, This is a preliminary fused representation of the feedforward layer output. This represents a matrix concatenation operation. The Softmax function ensures that the sum of all weights is 1, achieving normalized weighting of the feature dimensions. Through the feature attention module, the model can automatically identify and strengthen the feature dimensions most valuable for the current recommendation task, suppressing the interference of noise dimensions, thereby achieving more refined feature fusion.

[0091] Linear projection and layer normalization are performed sequentially on the fine-grained fusion results to output a heterogeneous fusion view.

[0092] The calculation process of linear projection and layer normalization can be expressed as formula (12): Formula Explanation: This formula is used to calculate the feature representation of the heterogeneous fused view after linear projection, layer normalization, and activation function processing. Where: This is the output feature vector of the heterogeneous fused view. To modify the activation function of the linear unit, For layer normalization operation, For learnable weight matrix, The input feature matrix for the fine-grained fusion result. This is the bias vector.

[0093] Specifically, linear projection maps the fused features to a dimension that matches the output layer of the recommendation model, while layer normalization stabilizes the feature distribution, accelerates model convergence, and improves generalization ability. The final output is a heterogeneous fused view. It includes adaptively fused global representations of users and items. This view not only integrates heterogeneous information from multiple sources, but also achieves dynamic filtering and enhancement of information through gating and attention mechanisms, significantly improving the expressive power and recommendation accuracy of the recommendation system.

[0094] This embodiment achieves adaptive fusion of heterogeneous views through a knowledge fusion intelligent agent. A gating mechanism solves the weight allocation problem at the view level, while a feature attention module solves the fine-grained filtering problem at the feature level. Together, they form a complete "macro-micro" adaptive fusion system. This mechanism enables the recommendation system to dynamically adjust the fusion strategy of different information sources based on specific user states and item attributes, thus maintaining high recommendation accuracy even in complex scenarios such as data sparsity and noise interference.

[0095] Example 5: This embodiment, building upon embodiments 1 to 4, provides a detailed explanation of the offline joint training process of the recommendation model and the optimization strategy for the knowledge fusion agent. Specifically, this embodiment constructs an end-to-end training network and joint loss function, and introduces a reinforcement learning mechanism to perform closed-loop optimization of the knowledge fusion agent, thereby ensuring that the recommendation model has optimal parameter configuration and fusion strategy before deployment.

[0096] An end-to-end full-link training network is constructed to generate user-side views, item-side views, knowledge-enhanced collaborative views, multimodal knowledge views, and heterogeneous fusion views.

[0097] Specifically, this end-to-end full-link training network integrates the graph neural network encoding module, hierarchical diffusion module, and knowledge fusion agent from the aforementioned embodiments. During the training phase, user-item interaction training data and multimodal item data serve as inputs, flowing sequentially through each module for forward propagation: first, various knowledge views are generated through encoding and graph construction; then, robust multimodal knowledge views are generated through the hierarchical diffusion module; finally, the knowledge fusion agent outputs a heterogeneous fused view and calculates the prediction score. This end-to-end architecture design allows gradients to propagate backward from the final recommendation loss to the initial encoding layer, achieving coordinated updates of parameters across modules and avoiding information loss and error accumulation caused by staged training.

[0098] A joint total loss function is constructed, which includes Bayesian personalized ranking loss, reconstruction loss, self-supervised alignment loss, semantic transformation loss, consistency constraint term between heterogeneous fused view and multimodal knowledge view, oversmoothing regularization term, and parametric regularization term.

[0099] Specifically, the Bayesian Personalized Ranking Loss (BPR Loss), as the primary objective function for the recommendation task, aims to maximize the difference in predicted scores between positive samples (items interacted with by the user) and negative samples (items not interacted with by the user), directly optimizing recommendation ranking performance. Reconstruction loss and semantic transformation loss are used to constrain the hierarchical diffusion model, ensuring that the reconstructed view retains original information while aligning with the collaborative semantic space. Self-supervised alignment loss is used to narrow the distance between the multimodal collaborative aggregation representation and the reconstructed view. Consistency constraints encourage the heterogeneous fused view and the multimodal knowledge view to maintain consistency in the semantic space, preventing information distortion during the fusion process. Oversmoothing regularization is used to alleviate the problem of excessive feature smoothing during graph neural network propagation, while parameter regularization is used to prevent model overfitting. It should be understood that although this embodiment lists the above loss terms, in practical applications, the weight coefficients of each term can be adjusted or some loss terms can be added or removed according to specific scenario requirements; this invention does not impose any limitations on this.

[0100] The joint total loss function can be expressed as formula (13): Formula Explanation: This formula is used to calculate the total loss function of a user-side multimodal recommendation model based on knowledge fusion intelligent agents, jointly optimizing ranking, similarity, semantic transformation, embedding consistency, contrastive learning, and regularization constraints. Where: For the joint total loss function, For Bayesian personalized ranking loss, , , , , These are the hyperparameter weight coefficients for each sub-loss term. This is the sum of the similarity transition losses for all modalities. For semantic transformation loss, To embed consistency constraint weights, and These are the embedding matrices for the heterogeneous fused view and the multimodal knowledge view, respectively. Represents the square of the L2 norm. To compare learning loss, For oversmoothing regularization terms, This is the set of trainable parameters for the model.

[0101] Wherein, the main objective function The Bayesian personalized ranking loss can be expressed as formula (14): Formula Explanation: This formula is used to optimize recommendation models based on pairwise ranking loss (BPR), by maximizing user... For items that have already been interacted with Items that are not interacted with The model is trained based on the differences in preferences. Where: For Bayesian personalized ranking loss, Denotes a triple, where For users, For positive sample items, For negative sample items, The set of all observed positive and negative sample triples, and Users predicted by the model For items and Preference scores, It is the Sigmoid activation function. This is the binary cross-entropy loss term.

[0102] This pairwise loss maximizes the difference between positive and negative interactions, directly optimizing sorting performance.

[0103] Based on user-item interaction training data, item multimodal data, and the user-side interest knowledge graph and item-side category knowledge graph constructed accordingly, the graph neural network encoding module, hierarchical diffusion module, and knowledge fusion agent are jointly trained with the goal of minimizing the joint total loss function.

[0104] During training, stochastic gradient descent or its variants (such as the Adam optimizer) are used to iteratively update the model parameters. In each iteration, the model calculates the values ​​of each loss term based on the current parameters, sums them in a weighted manner to obtain the joint total loss, and calculates the gradient backpropagation to update the parameters. Through this multi-task joint optimization approach, the model can simultaneously consider recommendation accuracy, view reconstruction quality, semantic alignment, and model generalization ability, thereby training a recommendation model with better performance.

[0105] When the joint total loss function converges to a preset threshold or reaches a preset number of iterations, the iterative training stops, and the parameters of each module after training are fixed.

[0106] This step marks the end of the offline training phase. The model with fixed parameters will be used in the subsequent online inference phase, i.e., the recommendation process described in Examples 1 to 4. The separation of offline training and online inference ensures the responsiveness of the recommendation system in practical applications, eliminating the need to retrain the model for each recommendation.

[0107] Within the framework of the aforementioned joint training, the fusion strategy parameters (such as gating weights and attention weights) of the knowledge fusion agent will also be initially optimized. However, to further enhance the agent's adaptability under different data distributions, this embodiment also introduces a reinforcement learning mechanism to specifically optimize the knowledge fusion agent. Specifically, this includes the following steps:

[0108] The multimodal knowledge view is defined as the environment state, and the fusion weight adjustment action of the knowledge fusion agent is defined as the policy action. The reward function is constructed based on the cosine similarity and semantic consistency between the multimodal knowledge view and the heterogeneous fusion view.

[0109] The reward function can be expressed as formula (15): Formula Explanation: This formula is used to calculate the reward value during the reinforcement learning optimization process in a knowledge fusion agent. The quality of the fusion effect is measured by the reconstruction error and cosine similarity. Where: As a reward value, For the embedding matrix of heterogeneous fused views, For the embedding matrix of the multimodal knowledge view, This represents the squared L2 norm (reconstruction error term). This is the cosine similarity calculation function. This represents a heterogeneous fusion view (i.e., a strategy action). This represents a multimodal knowledge view (i.e., the environmental state). The first term is the reconstruction error term, which measures the difference between the fusion result and the original input; the second term is the similarity reward term, which measures the semantic consistency between the fusion result and the original input. If the fused heterogeneous view can better retain the key information of the original view and improve semantic consistency, a positive reward is given; otherwise, a negative reward is given.

[0110] Specifically, in the reinforcement learning framework, the environment state reflects the environmental information of the agent. The multimodal knowledge view, as the input to the fusion process, contains comprehensive features of the current user, items, and multimodal content, and is therefore defined as the environment state. The policy action is the agent's response to the environment. The knowledge fusion agent determines how to combine the user-side view, item-side view, and multimodal knowledge view by outputting fusion weights; therefore, adjusting the fusion weights is defined as the policy action. The reward function evaluates the quality of the action. The reward function constructed in this embodiment consists of two parts: cosine similarity measures the consistency between the fusion result and the original input, while semantic consistency measures the degree to which the fusion result preserves semantics in the recommendation task. If the fused heterogeneous view can better retain the key information of the original view and improve semantic consistency, a positive reward is given; otherwise, a negative reward is given. This design allows the agent to autonomously learn the optimal fusion strategy with the goal of maximizing long-term cumulative rewards.

[0111] The policy network generates the policy action for the current round based on the current environment state, executes the policy action to complete the adaptive adjustment of the fusion weights and generate the corresponding heterogeneous fusion view, and calculates the reward value corresponding to the policy action for the current round according to the reward function.

[0112] The action generation process of the policy network can be expressed as formula (16): Formula Explanation: This formula defines the action generation process of the policy network in a knowledge fusion agent, that is, adjusting actions based on the current environmental state and the fusion weights output from the user-side and item-side views. Where, For parameters The policy network, For the user's side view, This is a side view of the item. This is a strategy action (i.e., weight adjustment). For parameters The policy network, This refers to the environmental state (i.e., a multimodal knowledge view). For the user's side view, This is a side view of the item.

[0113] Policy networks are typically implemented using deep neural networks. Their input is the environment state (a multimodal knowledge view), and their output is the policy action (the distribution of fusion weights). In each round of interaction, the policy network samples or selects a policy action based on the currently input multimodal knowledge view. This action determines the specific fusion weight parameters. Subsequently, the agent executes this action, generating a heterogeneous fused view and calculating an immediate reward based on a reward function. This process simulates the agent continuously trying different fusion strategies in complex and ever-changing recommendation scenarios.

[0114] With the goal of maximizing the cumulative reward value, the policy loss and value loss are combined to update all learnable parameters of the policy network and the value network. Based on the updated network, the policy action for the next round is generated, forming a closed-loop iterative link to continuously optimize the adaptive heterogeneous fusion strategy of the knowledge fusion agent.

[0115] The joint optimization of strategy loss and value loss can be expressed as formula (17): Formula Explanation: This formula is used to calculate the policy loss and value loss during the reinforcement learning optimization process in a knowledge fusion agent. It updates the parameters of the policy network and value network by minimizing the policy error and value error. Where: To strengthen the joint loss of learning, For the expected operation, For strategic actions, To reward scaling factor, As a reward value, For value network to state The value estimate, The value loss coefficient, These are the learnable parameters of the value network. By minimizing... The knowledge fusion intelligent agent adaptively adjusts the fusion strategy based on environmental feedback signals to achieve more effective fusion of multimodal knowledge views with user-side views and item-side views.

[0116] Specifically, this embodiment employs an Actor-Critic architecture for optimization. The policy network acts as the Actor, responsible for generating actions; the value network acts as the Critic, responsible for evaluating the value of the current state and guiding the policy network's updates. By minimizing policy loss and value loss, the two networks alternately update, continuously improving the quality of the policy. This closed-loop iterative chain enables the knowledge fusion agent to continuously adjust its own policy based on environmental feedback (reward value), thereby achieving continuous optimization of the fusion strategy. Compared with traditional static fusion methods, this reinforcement learning-based dynamic optimization mechanism can better adapt to the differences in data distribution among different users and in different scenarios, significantly improving the robustness and scenario adaptability of the recommendation system.

[0117] When the accumulated reward value converges to the preset threshold, the iteration stops, and the training of the knowledge fusion agent is completed.

[0118] When the agent's policy stabilizes and the cumulative reward value no longer increases significantly, it indicates that the agent has learned a better fusion policy. At this point, iteration stops, and the trained policy network parameters are saved. These parameters will serve as the initial policy for the knowledge fusion agent in actual recommendation tasks, or be directly used for online inference.

[0119] Furthermore, in the aforementioned training process, the self-supervised alignment loss plays a crucial constraining role. Specifically, the self-supervised alignment loss includes a similarity transition loss constructed based on the cosine similarity between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view.

[0120] This loss function forces the model to narrow the gap between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view in the feature space by maximizing the cosine similarity between them. The similarity transition loss not only enhances the semantic consistency between views but also effectively mitigates representation bias caused by data sparsity or noise interference, providing a more reliable feature foundation for subsequent fusion and recommendation tasks.

[0121] To further improve the fusion quality, this embodiment also introduces semantic transformation loss as a self-supervised signal, encouraging the heterogeneous fused view to maintain semantic consistency with the user-side view and the item-side view, while enhancing the similarity between the heterogeneous fused view and the multimodal knowledge view. The semantic transformation loss can be expressed as formula (18):

[0122] Formula Explanation: This formula is used to calculate the semantic transformation loss. It optimizes model robustness and semantic consistency by measuring the sum of squared Euclidean distances between the original embedding matrix and the reconstructed embedding matrix under different perturbations. Where: For semantic transformation loss, For heterogeneous fusion view, The L2 norm squared represents the model's encouragement of aligning heterogeneous merged views with the user's side view and the item's side view, thereby improving the merging effect.

[0123] This embodiment constructs a complete closed loop for model training and optimization through offline joint training and reinforcement learning optimization. The joint loss function ensures the collaborative work of each module in the model, while the reinforcement learning mechanism endows the knowledge fusion agent with the ability to dynamically adapt. This training strategy not only improves the recommendation accuracy of the model, but more importantly, it solves the technical problem in existing technologies where the fusion strategy is statically fixed and cannot be dynamically adjusted according to environmental feedback, providing a strong guarantee for the stable operation of the recommendation system in complex real-world scenarios.

[0124] Example 6: This embodiment applies the user-side multimodal recommendation method based on knowledge fusion intelligent agents described in Embodiments 1 to 5 above to a specific personalized recommendation scenario on an e-commerce platform. Through the description of this specific application scenario, the feasibility and significant beneficial effects of the technical solution of this invention in actual business processes are further verified.

[0125] Specifically, the e-commerce platform is assumed to have millions of registered users and hundreds of thousands of online products. Each product is associated with raw multimodal data, including the product's display image (visual modal raw data) and the product's detailed description text (text modal raw data). User behavior data includes browsing history, click history, purchase history, and displayed rating data (such as five-star reviews). A typical technical challenge faced by the system is the "cold start" problem for new users, namely, how to accurately recommend products that a newly registered user with very few interaction records might be interested in.

[0126] In this scenario, the recommended method of the present invention is specifically implemented as follows: Acquire user-item interaction data, user-item collaboration graph, and raw multimodal data of all items, and encode the raw multimodal data of items to obtain item multimodal embeddings.

[0127] For example, the system acquires limited interaction data from the target user "User_A," such as User_A only clicking on a "smartphone" and a "sports headphones." Simultaneously, the system acquires the raw multimodal data for all items. For a "smartwatch," its raw multimodal data includes a high-resolution product image and a text description containing keywords such as "heart rate monitoring" and "waterproof." The system uses a pre-trained visual encoder (such as ResNet-50) to extract features from the image, generating a 2048-dimensional visual feature vector, which is then compressed to 128 dimensions through dimensionality mapping. Simultaneously, a pre-trained text encoder (such as BERT) encodes the text description, also generating a 128-dimensional text feature vector. Subsequently, the system performs mean aggregation on these two vectors to generate a unified 128-dimensional item multimodal embedding for the "smartwatch." This process is repeated for all products on the platform to generate a multimodal embedding matrix for all items.

[0128] Based on user-item interaction data and item multimodal embedding, construct user-side interest knowledge graphs and item-side category knowledge graphs respectively.

[0129] Specifically, the system uses multimodal embeddings of all items and clusters items with similar features into classes using clustering functions (such as the K-means algorithm). For example, all electronic devices with similar appearance and functions are clustered into the "digital products" category, and all clothing items are clustered into the "fashion outfits" category, thus constructing an item-side category knowledge graph. In this graph, the "smartwatch" node and the "digital product" category node are connected by edges, representing their affiliation.

[0130] For the user side, the system extracts User_A's click behavior on "smartphone" and "sports headphones" as scoring weights (e.g., clicks are weighted as 1, purchases as weight 5). Based on these weights, the system performs normalized weighted aggregation on the multimodal embeddings of "smartphone" and "sports headphones" to generate single-user preference features for User_A. Since both products belong to the digital category and have technological attributes, User_A's preference feature vector has high values ​​in the "technology" and "electronics" feature dimensions. The system traverses all user-generated preference feature sets and uses clustering to group users with similar preference features into the same interest category. For example, User_A and users who purchase a lot of electronic products and follow technology news are clustered together into the "technology enthusiast" interest category. The system constructs a user-side interest knowledge graph with users and interest categories as nodes, where the User_A node is connected to the "technology enthusiast" category node. This step effectively maps User_A's limited behavioral data into a knowledge network with rich semantic associations, providing crucial knowledge support for its cold-start recommendation.

[0131] The user-side interest knowledge graph is fused with the user-item collaboration graph to form a user-side knowledge graph, and the item-side category knowledge graph is fused with the user-item collaboration graph to form an item-side knowledge graph. The three are then fused together to form a knowledge-enhanced collaborative knowledge graph, and encoded to obtain the user-side view, item-side view, and knowledge-enhanced collaborative view.

[0132] The system merges the constructed graph with the original user-item collaboration graph (which records the click and purchase topology relationships of all users). The merged knowledge-enhanced collaborative knowledge graph not only includes the interaction edges between User_A and "Smartphone", but also the belonging edges between User_A and the "Tech Enthusiast" category, as well as the belonging edges between "Smartphone" and the "Digital Products" category. The system uses a graph neural network (such as LightGCN) to perform multi-layer propagation encoding on this graph. During the propagation process, the node representation of User_A not only aggregates the information of its interacted items, but also aggregates the preference information of other highly active users in the "Tech Enthusiast" category through the "Tech Enthusiast" category node, thus greatly enriching the feature representation of User_A and effectively alleviating the data sparsity problem. The final output includes a user side view, an item side view, and a knowledge-enhanced collaborative view containing rich semantics.

[0133] Construct a multimodal knowledge view.

[0134] The system fuses the user-item collaboration graph with the item multimodal embedding to obtain a multimodal collaboration view, and independently encodes the collaboration view. To address noise interference such as poor lighting in e-commerce images and keyword stuffing in text descriptions, the system initiates a hierarchical diffusion model. The diffusion model performs forward noise addition and reverse noise reduction on the multimodal collaboration view to reconstruct a reconstructed multimodal collaboration view. This reconstructed view filters background noise from the original image and irrelevant modifiers from the text, while retaining core features such as "screen resolution" and "wearing comfort." The system uses a self-supervised alignment mechanism to semantically align the reconstructed view with the collaboration view, and combines the aforementioned knowledge-enhanced collaboration view to generate the final multimodal knowledge view. This view contains noise-resistant multimodal content information and integrates semantic information from collaboration signals and knowledge graphs.

[0135] The user side view, item side view, and multimodal knowledge view are adaptively fused to obtain a heterogeneous fused view. Preference prediction scores are then calculated to complete the recommendation.

[0136] The system inputs User_A's user-side view (emphasizing explicit interests), item-side view (emphasizing item attributes), and multimodal knowledge view (emphasizing content and collaboration) into the knowledge fusion agent. The agent dynamically calculates the fusion weights of each view through a gating mechanism and a feature attention module. Since User_A is a cold-start user, their user-side view information is relatively sparse. The agent adaptively assigns higher weights to the multimodal knowledge view to supplement User_A's features by utilizing the common preferences and multimodal content information of the "tech enthusiast" group in the knowledge graph. Finally, a heterogeneous fusion view is generated, extracting User_A's global user representation and the global item representations of each candidate item. The system calculates the preference prediction score between User_A and the candidate item "new smart bracelet" (e.g., through vector dot product). Because "new smart bracelet" belongs to "digital products" in the item-side category knowledge graph, and its multimodal features are highly similar to User_A's preference for "smartphones," and User_A is categorized as a "tech enthusiast," this prediction score is significantly higher than other unrelated items (e.g., "dress"). The system sorts users by score and recommends the "new smart bracelet" and related accessories to User_A.

[0137] Through application scenario verification in this embodiment, the method of the present invention successfully solves the recommendation problem in the cold start scenario for new users. By constructing a user-side interest knowledge graph, sparse user behavior is mapped to a dense semantic category space, achieving accurate expansion of user preferences; a hierarchical diffusion model effectively suppresses noise in e-commerce multimodal data; and the adaptive mechanism of the knowledge fusion agent dynamically balances the contribution of different information sources in the cold start scenario. This entire process demonstrates the efficient operation logic and significant technical effects of the technical solution of the present invention in a real e-commerce platform.

[0138] It should be noted that any parts not disclosed or specifically described in this invention are existing technology or conventional configurations, and their specific structures and working principles will not be elaborated further. In this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0139] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for the present invention. Other modifications can be readily implemented by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and examples shown and described herein.

Claims

1. A user-side multimodal recommendation method based on a knowledge fusion intelligent agent, characterized in that, include: Acquire the raw multimodal data of all items, user-item interaction data, and user-item collaboration graph, and encode the raw multimodal data to obtain the item multimodal embedding; Based on the user-item interaction data and item multimodal embedding, a user-side interest knowledge graph and an item-side category knowledge graph are constructed respectively. The user-side interest knowledge graph and the user-item collaboration graph are fused to form a user-side knowledge graph. The item-side category knowledge graph and the user-item collaboration graph are fused to form an item-side knowledge graph. The user-side interest knowledge graph, the item-side category knowledge graph, and the user-item collaboration graph are then fused together to form a knowledge-enhanced collaborative knowledge graph. The user-side knowledge graph, the item-side knowledge graph, and the knowledge-enhanced collaborative knowledge graph are encoded to obtain a user-side view, an item-side view, and a knowledge-enhanced collaborative view. The user-item collaboration graph is fused with the item multimodal embedding to obtain a multimodal collaboration view; The user-item collaboration graph is independently encoded to obtain a collaboration view; The collaborative view and the multimodal collaborative view are aggregated to obtain a multimodal collaborative aggregated representation, and the multimodal collaborative view is subjected to hierarchical reconstruction processing to obtain a reconstructed multimodal collaborative view; The multimodal collaborative aggregation representation is self-supervised and aligned with the reconstructed multimodal collaborative view, and features are fused with the user side view, item side view and knowledge-enhanced collaborative view to obtain a multimodal knowledge view; The user-side view, item-side view, and multimodal knowledge view are adaptively heterogeneously fused to obtain a heterogeneous fused view. The global user representation and global item representation are extracted from the heterogeneous fused view. Based on the global user representation and global item representation, the target user's preference prediction score for each candidate item is calculated to complete the user-side multimodal personalized recommendation.

2. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The encoding processing method includes: The original multimodal data of all items is split to obtain the visual modal raw data and text modal raw data corresponding to each item; wherein, the visual modal raw data is the item image data, and the text modal raw data is the item description text data; A pre-trained visual encoder is used to extract features and map dimensions of the object image data to generate a fixed-dimensional object visual embedding vector; a pre-trained text encoder is used to extract features and map dimensions of the object description text data to generate an object text embedding vector aligned with the dimensions of the object visual embedding vector. The visual embedding vector and text embedding vector of the same item are aggregated by mean to generate a unified multimodal embedding of the item. Iterate through all items and generate multimodal embeddings for each item.

3. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The method for constructing the user-side interest knowledge graph includes: Based on the user-item interaction data, extract the target user's rating weight for the items they interact with. Based on the rating weight, perform normalized weighted aggregation on the multimodal embedding of the items corresponding to the user's interactive items to generate the single-user preference feature corresponding to the user. Iterate through all users to generate a full set of user preference features. The full set of user preference features is clustered using a clustering function to divide it into multiple user interest categories. Using users and user interest categories as basic semantic units, a user-side interest knowledge graph representing the relationship between users and their corresponding interest categories is constructed.

4. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The method for constructing the item-side category knowledge graph includes: Clustering is performed on the multimodal embeddings of all items using a clustering function to divide them into multiple item category entities. Using items and item category entities as basic semantic units, an item-side category knowledge graph is constructed to represent the relationship between items and their corresponding category entities.

5. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The method for the hierarchical reconstruction process includes: The diffusion process is carried out through a hierarchical diffusion model. Gaussian noise is gradually added to the multimodal collaborative view according to a preset noise scale to generate noise perturbation views at different time steps until a pure noise distribution view is obtained. The diffusion inverse process is carried out through a hierarchical diffusion model. The mean and variance of the noise distribution are predicted by a parameterized neural network. The noise distribution is gradually recovered from the pure noise distribution view to obtain the reconstructed multimodal collaborative view after noise reduction. The reconstruction loss constrains the reconstructed multimodal collaborative view to retain the core feature information of the original multimodal collaborative view, and the semantic transformation loss constrains the reconstructed multimodal collaborative view to align with the semantic space of the collaborative view.

6. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The self-supervised alignment and feature fusion method includes: A self-supervised alignment loss is obtained by constructing a multimodal collaborative aggregation representation based on cosine similarity and reconstructing a multimodal collaborative view based on similarity constraints. With the goal of minimizing the self-supervised alignment loss, multi-view semantic alignment is performed between the multimodal collaborative aggregation representation and the reconstructed multimodal collaborative view; A multimodal knowledge view is obtained by combining the knowledge-enhanced collaborative view, the user side view, and the item side view through feature fusion.

7. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The adaptive heterogeneous fusion processing method includes: The user-side view and the multimodal knowledge view are matrix-concatenated and input into the first multilayer perceptron to calculate the user-side gating weight; the item-side view and the multimodal knowledge view are matrix-concatenated and input into the second multilayer perceptron to calculate the item-side gating weight. Based on the user-side gating weights, the user-side view and the multimodal knowledge view are weighted and aggregated to generate a preliminary user-side fused representation; based on the item-side gating weights, the item-side view and the multimodal knowledge view are weighted and aggregated to generate a preliminary item-side fused representation. The user-side preliminary fusion representation and the item-side preliminary fusion representation are spliced ​​together, and a preliminary fusion representation is obtained through a feedforward layer; A feature attention module is introduced to perform fine-grained feature adaptive fusion on the user side view, item side view, multimodal knowledge view, and preliminary fusion representation to obtain a fine-grained fusion result; The fine-grained fusion results are sequentially subjected to linear projection and layer normalization processing to output a heterogeneous fusion view.

8. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, Multimodal recommendations on the user side are achieved through a recommendation model. Before performing multimodal recommendations on the user side, the recommendation model is pre-trained offline jointly. The recommendation model includes a graph neural network encoding module, a hierarchical diffusion module, and a knowledge fusion intelligent agent. The offline joint training method includes: Construct an end-to-end full-link training network, which is used to generate user side view, item side view, knowledge-enhanced collaborative view, multimodal knowledge view and heterogeneous fusion view; Construct a joint total loss function, which includes Bayesian personalized ranking loss, reconstruction loss, self-supervised alignment loss, semantic transformation loss, consistency constraint term between heterogeneous fused view and multimodal knowledge view, oversmoothing regularization term, and parametric regularization term; Based on user-item interaction training data, item multimodal data, and user-side interest knowledge graphs and item-side category knowledge graphs constructed from these data, the graph neural network encoding module, hierarchical diffusion module, and knowledge fusion agent are jointly trained with the goal of minimizing the joint total loss function. When the joint total loss function converges to a preset threshold or reaches a preset number of iterations, the iterative training stops, and the parameters of each module after training are fixed.

9. A user-side multimodal recommendation method based on a knowledge fusion intelligent agent according to claim 8, characterized in that, The training method for the knowledge fusion intelligent agent is as follows: The multimodal knowledge view is defined as the environment state, the fusion weight adjustment action of the knowledge fusion agent is defined as the policy action, and the reward function is constructed based on the cosine similarity and semantic consistency between the multimodal knowledge view and the heterogeneous fusion view. The policy network generates the policy action for the current round based on the current environment state, executes the policy action to complete the adaptive adjustment of the fusion weight and generate the corresponding heterogeneous fusion view, and calculates the reward value corresponding to the policy action for the current round according to the reward function. With the goal of maximizing the cumulative reward value, the policy loss and value loss are combined to update all learnable parameters of the policy network and the value network. Based on the updated network, the policy action for the next round is generated, forming a closed-loop iterative link to continuously optimize the adaptive heterogeneous fusion strategy of the knowledge fusion agent. When the accumulated reward value converges to the preset threshold, the iteration stops, and the training of the knowledge fusion agent is completed.

10. The user-side multimodal recommendation method based on knowledge fusion intelligent agents according to claim 1, characterized in that, The method for constructing the self-supervised alignment loss includes: A similarity transition loss is constructed based on the cosine similarity between multimodal collaborative aggregation representation and reconstructed multimodal collaborative view.

Images