A knowledge-enhanced multi-modal knowledge dynamic recommendation method and system

Abstract

The application belongs to the technical field of intelligent recommendation, and relates to a multi-modal knowledge dynamic recommendation method and system based on knowledge enhancement. The application obtains a multi-modal feature tensor of a project; combines attention coefficients of modes to fuse the multi-modal feature tensor, and obtains a multi-modal feature fusion vector; generates an entity embedding vector according to a domain knowledge graph and a graph attention network; obtains a current user interest state according to the multi-modal feature fusion vector and the entity embedding vector, obtains interest prediction according to the current user interest state, the multi-modal feature fusion vector, a bias vector, a weight matrix and a nonlinear activation function; if the interest prediction is greater than or equal to an adaptive recommendation threshold, the application recommends the project to the user, otherwise, the application does not recommend; and generates a recommendation path explanation according to recommendation results of multiple projects. The application significantly improves the accuracy and explainability of long-tail content recommendation through knowledge-guided multi-modal semantic fusion.

Description

Technical Field

[0001] This invention belongs to the field of intelligent recommendation technology, and relates to a multimodal knowledge dynamic recommendation method and system based on knowledge enhancement. Background Technology

[0002] In the field of recommender systems, traditional methods have long been limited by their ability to process single-modal data and their shallow feature associations. Current mainstream collaborative filtering algorithms and content-based recommendation technologies mainly perform linear modeling on structured data (such as metadata and text tags), making it difficult to effectively capture the non-linear relationships between multimodal data (such as image visual features, audio spectral patterns, and video spatiotemporal information). Taking video recommendation scenarios as an example, existing systems typically perform simple associations between video metadata and user viewing records, neglecting the deep semantic connections between object recognition features implicit in video frames, audio sentiment, and user behavior patterns, leading to an increasingly prominent cross-modal semantic gap.

[0003] Existing technologies face three major bottlenecks in multimodal data processing:

[0004] First, there is the problem of fragmentation in heterogeneous feature representation. The distribution of data from different modalities in the feature space varies significantly, and traditional vector concatenation or weighted fusion methods are difficult to establish a semantically consistent mapping across modalities.

[0005] Secondly, dynamic interest modeling is lagging behind; static user profiles cannot capture the implicit preference migration patterns in multi-source behavioral data (such as live interactive bullet comments and short video scrolling trajectories) in real time.

[0006] Third, there is a lack of knowledge reasoning ability. Existing recommendation models lack the structured utilization of domain knowledge, making it difficult to achieve explainable recommendations in cold-start scenarios through entity relationship reasoning. Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal knowledge dynamic recommendation method and system based on knowledge enhancement, so as to solve the technical problems of fragmentation of heterogeneous feature representation, lag in dynamic interest modeling, and lack of knowledge reasoning ability in multimodal data processing.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] This invention discloses a multimodal knowledge dynamic recommendation method based on knowledge enhancement, comprising the following steps:

[0010] Obtain the multimodal feature tensor of the project;

[0011] By combining the attention coefficients of the modalities, the multimodal feature tensors are fused to obtain the multimodal feature fusion vector;

[0012] Generate entity embedding vectors based on domain knowledge graphs and graph attention networks;

[0013] The current user interest state is obtained based on the multimodal feature fusion vector and entity embedding vector. Interest prediction is obtained based on the current user interest state, multimodal feature fusion vector, bias vector, weight matrix, and nonlinear activation function.

[0014] If the interest prediction is greater than or equal to the adaptive recommendation threshold, then recommend items to the user; otherwise, do not recommend items.

[0015] An explanation of the recommended path is generated based on the recommendation results from multiple projects.

[0016] Furthermore, the formula for obtaining the multimodal feature tensor of the project is as follows:

[0017]

[0018] in, For the project Visual data, For visual feature extraction networks, For the project audio data, For audio spectrum encoders, For the project Text data, For text embedding models, Indicates feature concatenation operation; The dimension of the multimodal feature vector; for A multi-modal feature vector space; For the project The multimodal feature tensor.

[0019] Furthermore, the attention coefficients of the combined modalities are used to fuse the multimodal feature tensors to obtain a multimodal feature fusion vector, as detailed below:

[0020] The attention coefficients for each modality of the project are obtained by combining the projection matrix, multimodal feature tensor, bias vector, and learnable query vector with a normalized exponential function. The specific formula for obtaining the attention coefficients for each modality is as follows:

[0021]

[0022] in, It is a normalized exponential function; It is the hyperbolic tangent function; For learnable query vectors, where For the hidden layer dimension; This is the transpose of the learnable query vector; For modality The projection matrix, where For visual data, audio data, or text data, For the hidden layer dimension, The dimension of the multimodal feature vector; For the project The multimodal feature tensor has a dimension of . ; It is the bias vector;

[0023] The multimodal feature fusion vector is obtained by taking the attention coefficient and feature tensor of each modality. The specific formula is as follows:

[0024]

[0025] in, For the project modality The characteristic tensor; For modality Attention coefficient For the project The multimodal feature fusion vector.

[0026] Furthermore, the generation of entity embedding vectors based on the domain knowledge graph and graph attention network is as follows:

[0027] A domain knowledge graph is constructed, and the specific formula for constructing the domain knowledge graph is as follows:

[0028]

[0029] in, For a collection of entities, For a set of relations;

[0030] The entity embedding vector of the previous layer is updated using a graph attention network to generate the entity embedding vector of the current layer. The specific formula for updating the entity embedding vector is as follows:

[0031]

[0032] in, A relation is a set of relations. Elements in; An entity is a collection of entities. Elements in; It is a non-linear activation function; For entities In relationship Below Attention coefficient; A specific transformation matrix for the relation; For entities The Layer embedding vector; For knowledge graphs A set of relationships; Representing entities In relationship The set of neighbors below; For entities The Layer embedding vector.

[0033] Furthermore, the process of obtaining the current user interest state based on the multimodal feature fusion vector and entity embedding vector is as follows:

[0034] The multimodal feature fusion vector and entity embedding vector are concatenated, and the current user interest state is obtained by combining the concatenation result with the user interest state of the previous time step and the gated recurrent unit.

[0035] The specific formula for obtaining the current user's interest status is as follows:

[0036]

[0037] in, For gated loop unit; This represents the user's interest status at the previous time step. For cross-modal fusion vectors; For entity embedding vectors; Indicates feature concatenation operation; This represents the user's interest status at the current time step.

[0038] Furthermore, the interest prediction formula is as follows:

[0039]

[0040] in, This is the weight matrix; It is the bias vector; This represents the user's interest status at the current time step. For interest prediction; For cross-modal fusion vectors; It is a non-linear activation function.

[0041] Furthermore, the method for obtaining the adaptive recommendation threshold is as follows:

[0042] Obtain the mean of users' historical preferences, the dynamic adjustment coefficient, and the variance of users' historical preferences;

[0043] The adaptive recommendation threshold is obtained based on the user's historical preference mean, dynamic adjustment coefficient, and user's historical preference variance.

[0044] The formula for obtaining the adaptive recommendation threshold is as follows:

[0045]

[0046] in, This represents the average of users' historical preferences. This is a dynamic adjustment coefficient; The variance of users' historical preferences; This is an adaptive recommendation threshold.

[0047] Furthermore, the process of generating a recommendation path explanation based on the recommendation results of multiple projects is as follows:

[0048] The recommended path explanation is generated based on the embedding vector of the head entity, the multimodal feature fusion vector of the item, the relational path in the knowledge graph, and the cosine similarity calculation function.

[0049] The formula for generating the recommended path explanation is as follows:

[0050]

[0051] in, This is the function for calculating cosine similarity. The embedding vector of the head entity; For the project The multimodal feature fusion vector; This refers to the relationship paths in the domain knowledge graph.

[0052] Furthermore, it also includes the following steps:

[0053] Obtain positive and negative interest prediction samples;

[0054] Construct a positive sampling sample set based on positive samples predicted by interest, and obtain a negative sampling distribution based on project popularity based on negative samples predicted by interest;

[0055] An objective function is constructed by combining positive interest prediction samples, negative interest prediction samples, sampled sample set, negative sampling distribution based on project popularity, and loss value. The model parameters are then updated based on the objective function.

[0056] Obtain the embedding vectors of the head entity, the tail entity, and the relation in the relation path of the domain knowledge graph;

[0057] A KL divergence constraint function is constructed based on the embedding vectors of the head entity, the tail entity, the relation, and the KL divergence value. The smoothness of the knowledge graph embedding is then constrained based on the KL divergence constraint function.

[0058] Based on the above method, the present invention also discloses a multimodal knowledge dynamic recommendation system based on knowledge enhancement, comprising:

[0059] The multimodal feature tensor acquisition module is used to acquire the multimodal feature tensors of the project.

[0060] The multimodal feature fusion vector acquisition module is used to fuse multimodal feature tensors by combining the attention coefficients of the modalities to obtain multimodal feature fusion vectors;

[0061] The entity embedding vector acquisition module is used to generate entity embedding vectors based on the domain knowledge graph and graph attention network.

[0062] The interest prediction acquisition module is used to obtain the current user interest state based on the multimodal feature fusion vector and entity embedding vector, and to obtain interest prediction based on the current user interest state, multimodal feature fusion vector, bias vector combined with weight matrix and nonlinear activation function;

[0063] The project recommendation module recommends projects to users if their interest prediction is greater than or equal to the adaptive recommendation threshold; otherwise, it does not recommend any projects.

[0064] The recommended path explanation generation module is used to generate recommended path explanations based on the recommendation results of multiple projects.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] This invention's method acquires multimodal feature tensors of items, achieving unified representation of heterogeneous data such as images, audio, and text. This covers diverse user-item interaction information, effectively addressing the problem of significant differences in the distribution of different modal data in the feature space, enhancing data representation capabilities, and facilitating the establishment of cross-modal semantic consistency mapping. This allows data from different modalities to be fused and reasoned within the same semantic framework, significantly improving the comprehensiveness and accuracy of data representation. The method combines modal attention coefficients to fuse multimodal feature tensors, obtaining a multimodal feature fusion vector. It uses an attention mechanism to dynamically allocate weights for different modalities, resolving information redundancy or conflict between modalities, improving the discriminative power of fused features, and enabling the recommendation system to more accurately capture user interests, thereby improving the accuracy of recommendation results and user satisfaction. Finally, it generates entity embedding vectors based on domain knowledge graphs and graph attention networks. Based on the graph attention network of the domain knowledge graph, it establishes a cross-modal semantic association reasoning path, realizing the comprehensive understanding of domain knowledge. This invention utilizes structured knowledge and dynamically models relationships between entities through graph attention networks, enabling the recommendation system to reason based on entity relationships. This enhances the system's knowledge reasoning capabilities and provides strong support for complex recommendation scenarios. The current user interest state is obtained based on multimodal feature fusion vectors and entity embedding vectors. Interest prediction is then obtained by combining the current user interest state, multimodal feature fusion vectors, bias vectors, weight matrices, and nonlinear activation functions. By combining temporal gating networks and adaptive threshold mechanisms, the dynamic migration patterns of user interests are captured, allowing the recommendation system to more accurately predict user interests. Recommendation triggering conditions are dynamically adjusted based on the user's interest state, thus adjusting the recommendation strategy promptly when user interests shift, improving the timeliness and accuracy of recommendations. If the interest prediction is greater than or equal to the adaptive recommendation threshold, an item is recommended to the user; otherwise, no recommendation is made. The recommendation triggering conditions are dynamically adjusted based on the interest prediction value to avoid over-recommendation or under-recommendation. An explanation of the recommendation path is generated based on the recommendation results of multiple items. This invention breaks through the reliance on explicit feedback in traditional recommendation systems, significantly improving the accuracy and interpretability of long-tail content recommendations through knowledge-guided multimodal semantic fusion.

[0067] The system of this invention includes: a multimodal feature tensor acquisition module, a multimodal feature fusion vector acquisition module, an entity embedding vector acquisition module, an interest prediction acquisition module, an item recommendation module, and a recommendation path interpretation generation module. The multimodal feature tensor acquisition module acquires the multimodal feature tensors of items. The multimodal feature fusion vector acquisition module fuses the multimodal feature tensors with attention coefficients of the modalities to obtain a multimodal feature fusion vector. The entity embedding vector acquisition module generates entity embedding vectors based on a domain knowledge graph and a graph attention network. The interest prediction acquisition module obtains the current user's interest state based on the multimodal feature fusion vector and the entity embedding vector, and obtains an interest prediction based on the current user's interest state, the multimodal feature fusion vector, a bias vector, a weight matrix, and a nonlinear activation function. The item recommendation module recommends items to the user if the interest prediction is greater than or equal to an adaptive recommendation threshold; otherwise, it does not recommend items. The recommendation path interpretation generation module generates a recommendation path interpretation based on the recommendation results of multiple items. This system overcomes the reliance of traditional recommendation systems on explicit feedback and significantly improves the accuracy and interpretability of long-tail content recommendation through knowledge-guided multimodal semantic fusion. Attached Figure Description

[0068] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0069] Figure 2 This is a system connection diagram according to an embodiment of the present invention;

[0070] Figure 3 This is a simplified flowchart of a method according to another embodiment of the present invention;

[0071] Figure 4 This is a simplified diagram of the domain knowledge graph according to an embodiment of the present invention;

[0072] Figure 5 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0073] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0074] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0075] The present invention will now be described in further detail with reference to the accompanying drawings:

[0076] Example 1:

[0077] See Figure 1 This embodiment discloses a multimodal knowledge dynamic recommendation method based on knowledge enhancement, including the following steps:

[0078] S1, Obtain the multimodal feature tensor of the project, as follows:

[0079]

[0080] in, For the project Visual data, For visual feature extraction networks, For the project audio data, For audio spectrum encoders, For the project Text data, For text embedding models, Indicates feature concatenation operation; The dimension of the multimodal feature vector; for A multi-modal feature vector space; For the project The multimodal feature tensor.

[0081] By acquiring the multimodal feature tensor of a project, a unified representation of heterogeneous data such as images, audio, and text can be achieved. This can cover diverse interaction information between users and projects, effectively solve the problem of significant differences in the distribution of different modal data in the feature space, enhance data representation capabilities, and facilitate the establishment of cross-modal semantic consistency mapping. This enables data from different modalities to be fused and reasoned within the same semantic framework, thereby significantly improving the comprehensiveness and accuracy of data representation.

[0082] S2, combining the attention coefficients of the modalities, fuses the multimodal feature tensors to obtain a multimodal feature fusion vector. The attention mechanism dynamically allocates weights for different modalities, resolving information redundancy or conflict between modalities, improving the discriminative power of the fused features, and enabling the recommendation system to more accurately capture user interests, thereby improving the accuracy of recommendation results and user satisfaction. Specifically:

[0083] The attention coefficients for each modality of the project are obtained by combining the projection matrix, multimodal feature tensor, bias vector, and learnable query vector with a normalized exponential function. The specific formula for obtaining the attention coefficients for each modality is as follows:

[0084]

[0085] in, It is a normalized exponential function; It is the hyperbolic tangent function; For learnable query vectors, where For the hidden layer dimension; This is the transpose of the learnable query vector; For modality The projection matrix, where For visual data, audio data, or text data, For the hidden layer dimension, The dimension of the multimodal feature vector; For the project The multimodal feature tensor has a dimension of . ; It is the bias vector;

[0086] The multimodal feature fusion vector is obtained by taking the attention coefficient and feature tensor of each modality. The specific formula is as follows:

[0087]

[0088] in, For the project modality The characteristic tensor; For modality Attention coefficient For the project The multimodal feature fusion vector.

[0089] S3 generates entity embedding vectors based on the domain knowledge graph and graph attention network, as follows:

[0090] A domain knowledge graph is constructed, and the specific formula for constructing the domain knowledge graph is as follows:

[0091]

[0092] in, For a collection of entities, For a set of relations;

[0093] The entity embedding vector of the previous layer is updated using a graph attention network to generate the entity embedding vector of the current layer. The specific formula for updating the entity embedding vector is as follows:

[0094]

[0095] in, A relation is a set of relations. Elements in; An entity is a collection of entities. Elements in; It is a non-linear activation function; For entities In relationship Below Attention coefficient; A specific transformation matrix for the relation; For entities The Layer embedding vector; For knowledge graphs A set of relationships; Representing entities In relationship The set of neighbors below; For entities The Layer embedding vector.

[0096] This invention establishes a cross-modal semantic association reasoning path based on a graph attention network of domain knowledge graph, realizing the structured utilization of domain knowledge. Moreover, by dynamically modeling the relationships between entities through the graph attention network, the recommendation system can perform reasoning based on entity relationships, enhancing the knowledge reasoning ability of the recommendation system and providing strong support for complex recommendation scenarios.

[0097] S4. Obtain the current user interest state based on the multimodal feature fusion vector and entity embedding vector. Obtain interest prediction based on the current user interest state, multimodal feature fusion vector, bias vector, weight matrix, and nonlinear activation function.

[0098] In this embodiment, the process of obtaining the current user interest state based on the multimodal feature fusion vector and entity embedding vector is as follows:

[0099] The multimodal feature fusion vector and entity embedding vector are concatenated, and the current user interest state is obtained by combining the concatenation result with the user interest state of the previous time step and the gated recurrent unit.

[0100] The specific formula for obtaining the current user's interest status is as follows:

[0101]

[0102] in, For gated loop unit; This represents the user's interest status at the previous time step. For cross-modal fusion vectors; For entity embedding vectors; Indicates feature concatenation operation; This represents the user's interest status at the current time step.

[0103] This invention combines a temporal gating network with an adaptive threshold mechanism to capture the dynamic migration patterns of user interests, enabling the recommendation system to more accurately predict user interests and dynamically adjust recommendation triggering conditions based on user interest states. This allows for timely adjustment of recommendation strategies when user interests shift, improving the timeliness and accuracy of recommendations.

[0104] S5. If the interest prediction is greater than or equal to the adaptive recommendation threshold, then recommend items to the user; otherwise, do not recommend.

[0105] In this embodiment, the interest prediction formula is as follows:

[0106]

[0107] in, This is the weight matrix; It is the bias vector; This represents the user's interest status at the current time step. For interest prediction; For cross-modal fusion vectors; It is a non-linear activation function.

[0108] In this embodiment, the method for obtaining the adaptive recommendation threshold is as follows:

[0109] Obtain the mean of users' historical preferences, the dynamic adjustment coefficient, and the variance of users' historical preferences;

[0110] The adaptive recommendation threshold is obtained based on the user's historical preference mean, dynamic adjustment coefficient, and user's historical preference variance.

[0111] The formula for obtaining the adaptive recommendation threshold is as follows:

[0112]

[0113] in, This represents the average of users' historical preferences. This is a dynamic adjustment coefficient; The variance of users' historical preferences; This is an adaptive recommendation threshold.

[0114] S6 generates a recommendation path explanation based on the recommendation results of multiple projects, as detailed below:

[0115] The recommended path explanation is generated based on the embedding vector of the head entity, the multimodal feature fusion vector of the item, the relational path in the knowledge graph, and the cosine similarity calculation function.

[0116] The formula for generating the recommended path explanation is as follows:

[0117]

[0118] in, This is the function for calculating cosine similarity. The embedding vector of the head entity; For the project The multimodal feature fusion vector; This refers to the relationship paths in the domain knowledge graph.

[0119] In this embodiment of the invention, the following steps are also included:

[0120] Obtain positive and negative interest prediction samples;

[0121] Construct a positive sampling sample set based on positive samples predicted by interest, and obtain a negative sampling distribution based on project popularity based on negative samples predicted by interest;

[0122] An objective function is constructed by combining positive interest prediction samples, negative interest prediction samples, sampled sample set, negative sampling distribution based on project popularity, and loss value. The model parameters are then updated based on the objective function.

[0123] Obtain the embedding vectors of the head entity, the tail entity, and the relation in the relation path of the domain knowledge graph;

[0124] A KL divergence constraint function is constructed based on the embedding vectors of the head entity, the tail entity, the relation, and the KL divergence value. The smoothness of the knowledge graph embedding is then constrained based on the KL divergence constraint function.

[0125] Based on the above method, this invention also discloses a knowledge-enhanced multimodal knowledge dynamic recommendation system, see [link to relevant documentation]. Figure 2It includes: a multimodal feature tensor acquisition module, a multimodal feature fusion vector acquisition module, an entity embedding vector acquisition module, an interest prediction acquisition module, an item recommendation module, and a recommendation path interpretation generation module;

[0126] The multimodal feature tensor acquisition module is used to acquire the multimodal feature tensors of the project.

[0127] The multimodal feature fusion vector acquisition module is used to fuse multimodal feature tensors by combining the attention coefficients of the modalities to obtain multimodal feature fusion vectors;

[0128] The entity embedding vector acquisition module is used to generate entity embedding vectors based on the domain knowledge graph and graph attention network.

[0129] The interest prediction acquisition module is used to obtain the current user interest state based on the multimodal feature fusion vector and entity embedding vector, and to obtain interest prediction based on the current user interest state, multimodal feature fusion vector, bias vector combined with weight matrix and nonlinear activation function;

[0130] The project recommendation module recommends projects to users if their interest prediction is greater than or equal to the adaptive recommendation threshold; otherwise, it does not recommend any projects.

[0131] The recommended path explanation generation module is used to generate recommended path explanations based on the recommendation results of multiple projects.

[0132] This invention breaks through the reliance of traditional recommendation systems on explicit feedback. By using knowledge-guided multimodal semantic fusion, it significantly improves the accuracy and interpretability of long-tail content recommendations.

[0133] Example 2:

[0134] See Figure 3 and Figure 5 This embodiment proposes a multimodal knowledge dynamic recommendation method based on knowledge enhancement. By constructing a cross-modal semantic alignment network and a dynamic interest evolution model, it achieves accurate recommendation of multi-source heterogeneous data. This method overcomes the limitations of traditional recommendation systems in modeling single-modal data, innovatively integrating knowledge graph reasoning and multimodal dynamic feature learning to form a "data-knowledge-behavior" triple-driven architecture. The layered and progressive design of the technical architecture is as follows:

[0135] The data perception layer, through the multimodal feature tensor construction module, achieves unified representation of heterogeneous data such as images, audio, and text;

[0136] The knowledge enhancement layer, based on a graph attention network of domain knowledge graph, establishes a reasoning path for cross-modal semantic association;

[0137] The dynamic decision-making layer combines a temporal gating network with an adaptive threshold mechanism to capture the dynamic migration patterns of user interests.

[0138] The incremental optimization layer achieves continuous evolution and updating of model parameters through online negative sampling and regularization constraints.

[0139] The specific implementation steps of this invention are as follows:

[0140] S1. Construction of multimodal feature tensors;

[0141] For each project Extracting the multimodal feature tensor, the specific formula is as follows:

[0142]

[0143] in, For the project Visual data, For visual feature extraction, this embodiment uses ResNet-50 (Residual Network-50, 50 layers).

[0144] For the project audio data, For the audio spectrum encoder, this embodiment uses Mel-CNN (Mel-Spectrogram Convolutional Neural Network).

[0145] For the project Text data, For the text embedding model, this embodiment uses BERT (Bidirectional Encoder Representations from Transformers).

[0146] Indicates feature concatenation operation;

[0147] The dimension of the multimodal feature vector;

[0148] for A multi-modal feature vector space;

[0149] For the project The multimodal feature tensor.

[0150] It should be noted that the project It's a relatively abstract concept, corresponding to the English word "item." It can be a product, a program, a training project, a competition, and so on. Each item contains data categorized by visual, audio, and text types (i.e., multimodal data).

[0151] The goal of this invention is to recommend items of interest to a user from multiple items using a recommendation model.

[0152] In this step, For input; This is the calculation model selected in this invention; This is the output.

[0153] S2. Fusion of multimodal feature tensors;

[0154] Multimodal feature fusion is achieved through a multi-level attention mechanism:

[0155]

[0156]

[0157] in, It is a normalized exponential function;

[0158] It is the hyperbolic tangent function;

[0159] For learnable query vectors, where For the hidden layer dimension;

[0160] This is the transpose of the learnable query vector;

[0161] For modality The projection matrix, where For visual data, audio data, or text data, For the hidden layer dimension, The dimension of the multimodal feature vector;

[0162] For the project The multimodal feature tensor has a dimension of . ; The result of S1;

[0163] Let be the bias vector, where For the hidden layer dimension;

[0164] For the project modality The characteristic tensor, where It can be visual data, audio data, or text data; that is... For the project The visual feature tensor, For the project The audio feature tensor, For the project The text feature tensor;

[0165] For modality The attention coefficient, where It can be visual data, audio data, or text data; that is... Visual attention coefficient For audio attention coefficient, The text attention coefficient;

[0166] For the project The multimodal feature fusion vector.

[0167] In this step, The result of S1 is also the input of S2; These are model parameters, determined through model training; This is the output.

[0168] S3. Knowledge graph embedding enhancement;

[0169] Building Domain Knowledge Graphs ,in For a collection of entities, This is a set of relations. Entity embedding vectors are generated using a graph attention network:

[0170]

[0171] in, A relation is a set of relations. Elements in;

[0172] An entity is a collection of entities. Elements in;

[0173] It is a non-linear activation function;

[0174] For entities In relationship Below Attention coefficient;

[0175] A specific transformation matrix for the relation;

[0176] For entities The Layer (sub-)embedding vectors;

[0177] For knowledge graphs A set of relationships;

[0178] Representing entities In relationship The set of neighbors below;

[0179] For entities The Layer (sub) embedding vector.

[0180] In S3: Domain Knowledge Graph For input;

[0181] See Figure 4 For a domain knowledge graph Example. Where, entity set. Relation set This knowledge graph is based on domain-related knowledge of recommended projects and is processed using knowledge graph construction tools. Therefore, the set in the formula... and the elements in the set It can be used as known input.

[0182] These are the model parameters for the knowledge graph (determined through model training).

[0183] In particular, It is both input and output, and is calculated through an iterative process.

[0184] S4. Temporal interest modeling;

[0185] Establish a dynamic user interest state equation:

[0186]

[0187]

[0188] in,

[0189] For gated loop unit;

[0190] It is a non-linear activation function;

[0191] This represents the user's interest status at the previous time step.

[0192] For cross-modal fusion vectors;

[0193] For entity embedding vectors;

[0194] Indicates feature concatenation operation;

[0195] This is the weight matrix;

[0196] It is the bias vector;

[0197] This represents the user's interest status at the current time step.

[0198] For interest prediction;

[0199] This is a transpose operation.

[0200] Gated Loop Unit The control mechanism is as follows:

[0201] Reset Door:

[0202] Update Gate:

[0203] Candidate hidden state:

[0204] Update hidden status:

[0205] in, Use the Sigmoid activation function;

[0206] It is the hyperbolic tangent function;

[0207] , and This is the weight matrix;

[0208] Input for the current time step;

[0209] Indicates the convolution operation;

[0210] To reset the door;

[0211] To update the door;

[0212] This represents the user's interest status at the previous time step.

[0213] Candidate state;

[0214] Update to the final state.

[0215] In this step, For input, This is the result of step 2. This is the result of step 3; It is both input and output, and is calculated through an iterative process; These are model parameters, determined through model training; This is the output, which is the final result of S4.

[0216] S5. Adaptive recommendation threshold generation;

[0217] Define the adaptive recommendation threshold generation function:

[0218]

[0219] in, This represents the average of users' historical preferences.

[0220] This is a dynamic adjustment coefficient;

[0221] The variance of users' historical preferences;

[0222] This is an adaptive recommendation threshold.

[0223] The final recommendation decision function is:

[0224]

[0225] In S5:

[0226] The input is the result of step 4;

[0227] The mean and variance were calculated from the historical data, respectively.

[0228] These are manually set parameters;

[0229] This is the output. When the value is 1, item k is recommended to the user; otherwise, it is not recommended.

[0230] S6. Incremental model optimization;

[0231] The model parameters are updated using an online negative sampling strategy:

[0232]

[0233] in,

[0234] It is a logarithmic function;

[0235] For the Sigmoid function;

[0236] For interest prediction; Interest prediction of positive samples Interest prediction of negative samples;

[0237] It should be noted that positive samples are those recommended to users and that users are interested in, or those not recommended to users and that users are not interested in. Negative samples are those recommended to users but that users are not interested in, or those not recommended to users but that users are interested in.

[0238] For positive sampling sample set;

[0239] This is a negative sampling distribution based on project popularity;

[0240] This represents the loss value of the objective function.

[0241] Smoothness of knowledge graph embeddings constrained by KL divergence:

[0242]

[0243] in,

[0244] The embedding vector of the head entity of the triple;

[0245] The embedding vector for the relation of the triples;

[0246] The embedding vector of the tail entity of the triple;

[0247] For domain knowledge graphs;

[0248] This represents the KL divergence value.

[0249] These are (entity, relation, entity) triples in the knowledge graph; for example, in the previous knowledge graph: (personnel e1, operation r1, weapons and supplies e2).

[0250] It represents the square of the 2-norm (vector length).

[0251] In S6:

[0252] Indicates interest prediction of positive samples. Indicates interest in predicting negative samples;

[0253] The objective function (loss function) is used to train the recommendation model constructed in steps 1-5;

[0254] The objective function for the embedding vector in the knowledge graph is used to train the recommendation model constructed in steps 1 to 5, especially to train the knowledge graph model in step 3, thereby optimizing the entity embedding vector.

[0255] S7. Explainable Recommendations

[0256] Explanation of the generated recommended path:

[0257]

[0258] in, This is the function for calculating cosine similarity.

[0259] The embedding vector of the head entity in the entity embedding vector;

[0260] For the project The multimodal feature fusion vector;

[0261] This refers to the relational paths in a domain knowledge graph. A relational path is a set of (entity, relation, entity) triples in the knowledge graph, which form a continuous path within the knowledge graph. For example, in the previous knowledge graph, the set P = {(person e1, job title r2, activity e3), (activity e3, element relation r3, material e2), (material e2, deployment location r8, location e5)} constitutes the relational path: Person-Activity-Material-Location.

[0262] The recommended path; the recommendation result obtained by the recommendation model, for example, Figure 4 In the knowledge graph: Activity Items - Materials - Location

[0263] Let be a function, representing the computation of such that Recommended path (set) to find the maximum value );

[0264] Each of these is a (entity, relation, entity) triple in the knowledge graph; for example... Figure 4 In the knowledge graph: (personnel e1, operation r1, material e2).

[0265] This invention innovatively constructs a multimodal dynamic recommendation framework that integrates multimodal fusion and knowledge enhancement. By introducing a domain knowledge graph as a semantic bridge, the system achieves hierarchical alignment of cross-modal features. In the feature extraction layer, a deep residual network is used to extract visual features, combined with temporal convolution to process audio signals, and a graph neural network is used to encode knowledge entity embeddings. In the fusion reasoning layer, a multi-level attention mechanism is designed to dynamically adjust the contribution weights of different modal features, while implicit interest reasoning is achieved through graph relationship path mining. In the dynamic update layer, a model optimization mechanism based on incremental learning is established to capture pattern evolution features in user cross-platform behavior data in real time. This method breaks through the dependence of traditional recommendation systems on explicit feedback, and significantly improves the accuracy and interpretability of long-tail content recommendation through knowledge-guided multimodal semantic fusion.

[0266] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A multimodal knowledge dynamic recommendation method based on knowledge enhancement, characterized in that, Includes the following steps: Obtain the multimodal feature tensor of the project; By combining the attention coefficients of the modalities, the multimodal feature tensors are fused to obtain the multimodal feature fusion vector; Generate entity embedding vectors based on domain knowledge graphs and graph attention networks; The current user interest state is obtained based on the multimodal feature fusion vector and entity embedding vector. Interest prediction is obtained based on the current user interest state, multimodal feature fusion vector, bias vector, weight matrix, and nonlinear activation function. The process of obtaining the current user interest state based on the multimodal feature fusion vector and entity embedding vector is as follows: The multimodal feature fusion vector and entity embedding vector are concatenated, and the current user interest state is obtained by combining the concatenation result with the user interest state of the previous time step and the gated recurrent unit. If the interest prediction is greater than or equal to the adaptive recommendation threshold, then recommend items to the user; otherwise, do not recommend items. An explanation of the recommended path is generated based on the recommendation results from multiple projects.

2. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The specific formula for obtaining the multimodal feature tensor of the project is as follows: in, For the project Visual data, For visual feature extraction networks, For the project audio data, For audio spectrum encoders, For the project Text data, For text embedding models, Indicates feature concatenation operation; The dimension of the multimodal feature vector; for A multi-modal feature vector space; For the project The multimodal feature tensor.

3. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The attention coefficients of the combined modalities are used to fuse the multimodal feature tensors to obtain a multimodal feature fusion vector, as detailed below: The attention coefficients for each modality of the project are obtained by combining the projection matrix, multimodal feature tensor, bias vector, and learnable query vector with a normalized exponential function. The specific formula for obtaining the attention coefficients for each modality is as follows: in, It is a normalized exponential function; It is the hyperbolic tangent function; For learnable query vectors, where For the hidden layer dimension; This is the transpose of the learnable query vector; For modality The projection matrix, where For visual data, audio data, or text data, For the hidden layer dimension, The dimension of the multimodal feature vector; For the project The multimodal feature tensor has a dimension of . ; It is the bias vector; The multimodal feature fusion vector is obtained by taking the attention coefficient and feature tensor of each modality. The specific formula is as follows: in, For the project modality The characteristic tensor; For modality Attention coefficient For the project The multimodal feature fusion vector.

4. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The generation of entity embedding vectors based on the domain knowledge graph and graph attention network is as follows: A domain knowledge graph is constructed, and the specific formula for constructing the domain knowledge graph is as follows: in, For a collection of entities, For a set of relations; The entity embedding vector of the previous layer is updated using a graph attention network to generate the entity embedding vector of the current layer. The specific formula for updating the entity embedding vector is as follows: in, A relation is a set of relations. Elements in; An entity is a collection of entities. Elements in; It is a non-linear activation function; For entities In relationship Below Attention coefficient; A specific transformation matrix for the relation; For entities The Layer embedding vector; For knowledge graphs A set of relationships; Representing entities In relationship The set of neighbors below; For entities The Layer embedding vector.

5. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The specific formula for obtaining the current user's interest status is as follows: in, For gated loop unit; This represents the user's interest status at the previous time step. For cross-modal fusion vectors; For entity embedding vectors; Indicates feature concatenation operation; This represents the user's interest status at the current time step.

6. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The formula for obtaining the interest prediction is as follows: in, This is the weight matrix; It is the bias vector; This represents the user's interest status at the current time step. For interest prediction; For cross-modal fusion vectors; It is a non-linear activation function.

7. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The method for obtaining the adaptive recommendation threshold is as follows: Obtain the mean of users' historical preferences, the dynamic adjustment coefficient, and the variance of users' historical preferences; The adaptive recommendation threshold is obtained based on the user's historical preference mean, dynamic adjustment coefficient, and user's historical preference variance. The formula for obtaining the adaptive recommendation threshold is as follows: in, This represents the average of users' historical preferences. This is a dynamic adjustment coefficient; The variance of users' historical preferences; This is an adaptive recommendation threshold.

8. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, The explanation for generating the recommendation path based on the recommendation results of multiple projects is as follows: The recommended path explanation is generated based on the embedding vector of the head entity, the multimodal feature fusion vector of the item, the relational path in the knowledge graph, and the cosine similarity calculation function. The formula for generating the recommended path explanation is as follows: in, This is the function for calculating cosine similarity. The embedding vector of the head entity; For the project The multimodal feature fusion vector; This refers to the relationship paths in the domain knowledge graph.

9. The multimodal knowledge dynamic recommendation method based on knowledge enhancement according to claim 1, characterized in that, It also includes the following steps: Obtain positive and negative interest prediction samples; Construct a positive sampling sample set based on positive samples predicted by interest, and obtain a negative sampling distribution based on project popularity based on negative samples predicted by interest; An objective function is constructed by combining positive interest prediction samples, negative interest prediction samples, sampled sample set, negative sampling distribution based on project popularity, and loss value. The model parameters are then updated based on the objective function. Obtain the embedding vectors of the head entity, the tail entity, and the relation in the relation path of the domain knowledge graph; A KL divergence constraint function is constructed based on the embedding vectors of the head entity, the tail entity, the relation, and the KL divergence value. The smoothness of the knowledge graph embedding is then constrained based on the KL divergence constraint function.

10. A multimodal knowledge dynamic recommendation system based on knowledge enhancement, characterized in that, include: The multimodal feature tensor acquisition module is used to acquire the multimodal feature tensors of the project. The multimodal feature fusion vector acquisition module is used to fuse multimodal feature tensors by combining the attention coefficients of the modalities to obtain multimodal feature fusion vectors; The entity embedding vector acquisition module is used to generate entity embedding vectors based on the domain knowledge graph and graph attention network. The interest prediction acquisition module is used to obtain the current user interest state based on the multimodal feature fusion vector and entity embedding vector, and to obtain interest prediction based on the current user interest state, multimodal feature fusion vector, bias vector combined with weight matrix and nonlinear activation function; The process of obtaining the current user interest state based on the multimodal feature fusion vector and entity embedding vector is as follows: The multimodal feature fusion vector and entity embedding vector are concatenated, and the current user interest state is obtained by combining the concatenation result with the user interest state of the previous time step and the gated recurrent unit. The project recommendation module recommends projects to users if their interest prediction is greater than or equal to the adaptive recommendation threshold; otherwise, it does not recommend any projects. The recommended path explanation generation module is used to generate recommended path explanations based on the recommendation results of multiple projects.

Images