Multimodal recommendation method based on knowledge graph enhancement

Abstract

The application belongs to the technical field of artificial intelligence and recommendation system, and discloses a multi-modal recommendation method based on knowledge graph enhancement, which comprises the following steps: step 1, generating user-single scene level data samples; step 2, after the generated user-single scene level data samples are subjected to structured identification processing, non-structured information processing and text-semantic alignment and hierarchical fusion, multi-modal action sequence features are obtained; step 3, constructing a field knowledge graph to obtain knowledge context features; and step 4, fusing the multi-modal action sequence features and the knowledge context features, and inputting the fused features into a recommendation prediction model for processing to obtain a final recommendation table. The application integrates multi-source heterogeneous data and a field knowledge graph, and realizes integrated recommendation of data complete representation-knowledge safety constraint-time sequence dynamic adaptation.

Description

Technical Field

[0001] This application belongs to the interdisciplinary field of artificial intelligence and recommendation systems, specifically involving a multimodal recommendation method based on knowledge graph enhancement. Background Technology

[0002] Multimodal data fusion and knowledge-driven recommendation are core research directions in the field of artificial intelligence. Their core value lies in providing accurate and reliable recommendation support for decision-making needs in different scenarios by integrating multi-source heterogeneous data and domain knowledge. In various recommendation scenarios, two core objectives must be simultaneously met: accurate matching and risk and security assurance. The former relies on multimodal data to comprehensively characterize the state of users and objects, while the latter relies on domain knowledge to constrain the rationality and security of recommendation results. Only by combining these two aspects can the practical value and user trust of the recommendation system be effectively improved.

[0003] Current general recommendation methods revolve around data utilization and knowledge integration, forming single-modal data-driven methods, knowledge graph-assisted recommendation methods, and preliminary attempts at multimodal fusion. Single-modal data-driven methods use structured data such as user IDs, item identifiers, and interaction timestamps, or single unstructured data such as text descriptions and image features as their core, capturing data features through models such as recurrent neural networks (RNNs), Transformers, and collaborative filtering. In the e-commerce field, time-series recommendation models based on users' historical purchase records often have an architecture of "identifier embedding - time-series modeling - probability output," where signals are transmitted from the discrete encoding input layer through hidden layers to the recommendation output layer. In the content domain, news recommendation models based solely on text keywords rely on bag-of-words models or basic pre-trained models to extract text features. However, these methods completely ignore the complementary information between multimodal data, resulting in one-sided representations of user and object states, insufficient data utilization, and difficulty in handling complex scenarios. Knowledge graph-assisted recommendations construct knowledge graphs of the "user-item-attribute" or "entity-relationship-entity" type and utilize techniques such as Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and path reasoning to uncover patterns in entity relationships. In e-commerce, a "user-product-category-brand" knowledge graph is constructed, supplementing product attribute relationships through path reasoning; in finance, a "user-financial product-risk level-investment period" knowledge graph is constructed to assist in screening compliant products. However, most solutions only cover basic relationships such as "associated with" and "belongs to" and lack integration of deep entity attributes and deep fusion with unstructured data modalities, resulting in recommendations lacking personalization. Some studies attempt to combine structured and unstructured data features, but due to the lack of cross-modal semantic alignment mechanisms, the complementarity of the two types of data is not fully utilized; furthermore, most methods do not incorporate domain knowledge constraints, leading to security vulnerabilities or compliance risks in the recommendation results.

[0004] In summary, existing general recommendation methods suffer from several shortcomings, including insufficient utilization of fragmented multimodal data, low recommendation accuracy due to a lack of domain knowledge association, poor scenario compliance and weak engineering reproducibility, and an inability to effectively integrate structured and unstructured data. Summary of the Invention

[0005] To address the aforementioned technical challenges, this application provides a knowledge graph-based multimodal recommendation method. This method integrates multi-source heterogeneous data with domain knowledge graphs to achieve integrated recommendation encompassing "complete data representation, knowledge security constraints, and dynamic temporal adaptation." It can be widely applied in various fields such as healthcare, e-commerce, content distribution, and fintech, and is particularly suitable for scenarios with high requirements for recommendation accuracy, security, and interpretability.

[0006] To achieve the above objectives, this application employs the following technical solution:

[0007] This application presents a knowledge graph-based multimodal recommendation method, which includes the following steps:

[0008] Step 1, Data Acquisition and Preprocessing: Acquire multi-source data, including structured and unstructured data, and preprocess the acquired multi-source data to generate user-single-scenario level data samples. The user-single-scenario level data samples include a scenario identifier table, a core attribute table, an interaction behavior table, an operation record table, unstructured information data, and an auxiliary mapping table.

[0009] Step 2, Multimodal Feature Processing: The user-single scene-level data samples generated in Step 1 are processed through structured labeling, unstructured information processing, and text-semantic alignment and hierarchical fusion to obtain multimodal action sequence features;

[0010] Step 3, Domain Knowledge Graph Construction and Enhancement: Construct a domain knowledge graph and use a graph attention network to perform embedding learning and reasoning on the domain knowledge graph to obtain knowledge context features;

[0011] Step 4, Multimodal and Knowledge Graph Collaborative Fusion and Recommendation Prediction: The multimodal action sequence features obtained in Step 2 and the knowledge context features obtained in Step 3 are fused together. The fused features are then input into the recommendation prediction model for processing to obtain the final recommendation table.

[0012] A further improvement of this application is that, in step 1, the preprocessing of the multi-source data includes the following steps:

[0013] Step 1.1, Structured Data Preprocessing: Convert non-standard data in the interaction behavior table into domain-specific data through the corresponding mapping table, filter out null or invalid identifier records in the core attribute table and operation record table, group and deduplicate by user unique identifier - scene unique identifier, and convert the core identifiers into ID sequences that the recommendation prediction model can recognize based on domain vocabulary.

[0014] Step 1.2, Unstructured Information Preprocessing: Filter the category records in the unstructured information data that match the recommendation requirements, filter out categories and characters that are not related to the recommendation requirements, and align the preprocessed unstructured data with the user scenario-related records generated based on the scenario identifier table, interaction behavior table, etc., according to the scenario time in the scenario identifier table to obtain aligned unstructured information;

[0015] Step 1.3, Data Integration: Using the user's unique identifier and the scene's unique identifier as dual identifiers, the ID sequence that the preprocessed recommendation prediction model can recognize is inner-joined with the aligned unstructured information obtained after step 1.2. Invalid samples without core attributes or unstructured information are removed to generate user-single scene-level data samples, and user scene auxiliary features are calculated at the same time.

[0016] A further improvement in this application is that the structured labeling process in step 2 involves encoding the preprocessed user-single-scene-level data samples and user-scene auxiliary features using a CodeBERT encoder. The CodeBERT encoder encoding includes a LayerNorm layer (adapting to the normalization of discrete features) and a Multi-HeadedAttention layer (strengthening semantic associations), specifically including the following steps:

[0017] Step 2.1.1 Input Construction: The core attribute table, operation record table, and interaction behavior table for a single scenario are processed into text, forming core attribute text sequences, operation record text sequences, interaction behavior text sequences, and user scenario auxiliary features, respectively. These are then combined in the following order: the identifier representing the start of classification, the core attribute text sequence, the identifier used for separation, the operation record text sequence, the identifier used for separation, the interaction behavior text sequence, the identifier used for separation, and the user scenario auxiliary features, to form the input content for subsequent processing.

[0018] Step 2.1.2, Embedding Learning: Construct trainable embedding vectors for the core attribute text sequence, operation record text sequence, interaction behavior text sequence, and user scenario auxiliary features, respectively, preserving the hierarchical structure and co-occurrence context of the core attribute text sequence, operation record text sequence, and interaction behavior text sequence; introduce a top-down parent node aggregation strategy to aggregate the constructed trainable embedding vectors hierarchically into a scene-level structured feature representation. .

[0019] A further improvement in this application is that, in step 2, unstructured information processing involves using a domain-specific pre-trained language model to label the pre-processed user-single-scene-level data samples, specifically including the following steps:

[0020] Step 2.2.1 Input Processing: The aligned unstructured information obtained in Step 1.2 is truncated or supplemented according to the maximum sequence length (supplementing specific identifiers) to generate an unstructured ID sequence;

[0021] Step 2.2.2, Feature Extraction: Using the encoder layer of the domain-pre-trained language model, unstructured semantic features are extracted from the generated unstructured ID sequence in Step 2.2.1. The pooling vector corresponding to the starting identifier of the ID sequence is taken as the scene-level unstructured feature representation. .

[0022] A further improvement in this application is that, in step 2, the text-semantic alignment and hierarchical fusion specifically include the following steps:

[0023] Step 2.3.1, Text-Code Alignment: Construct a text-data attention alignment layer and compute scene-level structured feature representations. and scene-level unstructured feature representation cosine similarity The formula is:

[0024]

[0025] Among them, cosine similarity The range of values ​​is cosine similarity The closer the value is to 1, the higher the semantic correlation between scene-level structured features and scene-level unstructured features, which is used to filter strongly related data fragments. For scene-level unstructured feature representation The One dimension, For scene-level structured features The One dimension, Using the feature vector dimension, based on similarity weights, unstructured fragments that are strongly related to user status and needs are highlighted, while noisy text is filtered out;

[0026] Step 2.3.2, Multimodal hierarchical fusion: including action fusion and sequence-level fusion, specifically:

[0027] Action fusion: aligning scene-level structured feature representations and scene-level unstructured feature representation The feature dimension is concatenated using a two-layer fully connected network and... The layers are used for feature fusion to obtain multimodal features of a single action. ;

[0028] Sequence-level fusion: The user's multi-turn actions are sorted by action time, and a temporal attention layer is constructed to calculate the feature representation of each action. The weights are calculated using the following formula:

[0029]

[0030]

[0031] in, For action sequence number, The number of times the action is performed. , It is a linear projection function. For bias, For the first Attention weights for sub-scene features, and multimodal features of a single action. We obtain the user's multimodal action sequence features by weighted summation. .

[0032] A further improvement in this application is that step 3 specifically includes the following steps:

[0033] Step 3.1, Domain Knowledge Graph Construction: Construct a domain knowledge graph containing four types of entities—core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes—as well as multiple types of relationships. ,in At the node level, For edge set, For a set of node types, Let the set be an edge type set; where the node set is... This includes core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes; the edge set... This includes hierarchical edges (parent-child or ancestor-descendant relationships between core entities, parent-child or ancestor-descendant relationships between related entities, parent-child or ancestor-descendant relationships between operation nodes), and cross-domain edges (adaptation relationships between core entities and related entities, interaction relationships between related entities, and adaptation relationships between core entities and operations); the node type set : {Core entity node, associated entity node, operation node, constraint rule node}; the edge type set {Core entity hierarchical relationship, associated entity hierarchical relationship, operational hierarchical relationship, core entity-associated entity adaptation relationship, associated entity-associated entity interaction relationship, core entity-operation adaptation relationship}; Edge weight assignment: Hierarchical edge weight is set to 1.0; The edge weight of the core entity-associated entity adaptation relationship is assigned based on the domain guide or quasi-recommendation degree; The edge weight of the associated entity-associated entity interaction relationship is assigned based on the risk value in the constraint rule table.

[0034] Step 3.2, Knowledge Graph Embedding and Graph Reasoning: Graph Attention Network (GAT) is used to embed the domain knowledge graph constructed in Step 3.1. Embedded learning and graph reasoning are performed. The graph attention network contains two layers of GATConv, and the knowledge dimension is set as a configurable hyperparameter.

[0035] Step 3.3: Knowledge Constraints and Security Controls.

[0036] A further improvement of this application is that step 3.2 specifically includes the following steps:

[0037] Step 3.2.1, Node Embedding Initialization: Align the initial features of core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes with the ID mapping in the domain's auxiliary mapping to generate configurable initial embedding vectors.

[0038] Step 3.2.2, Graph Attention Calculation: First layer GATConv, processing hierarchical edges: Calculate attention for hierarchical edges of core entity nodes, associated entity nodes, and operation nodes. The formula is:

[0039]

[0040]

[0041]

[0042] in, The central node currently of interest in the first layer. and its neighboring nodes Interaction features or embeddings between them For neighboring nodes Embedded vector, Represents the current central focus. represent The neighboring nodes, For a leaky linear rectified activation function, For nodes The initial embedding, This is the first layer weight matrix. This is the first layer attention vector. Indicates transpose. For nodes The hierarchical set of edge neighbors, For the attention weights of the hierarchical edges; The hierarchical association features output from the first layer. Indicates the previously concerned central node and its neighboring nodes Interaction features or embedded representations between them;

[0043] The second layer, GATConv, handles cross-domain edges: it performs attention calculations on cross-domain edges, which are core entity node-related entity node, related entity node-related entity node, core entity node-operation node, and related entity node-constraint rule node. The attention calculation formula is as follows:

[0044]

[0045]

[0046]

[0047] in, The central node currently of interest in the second layer. and its neighboring nodes Interaction features or embeddings between them This is the weight matrix for the second layer. This is the second layer attention vector. For nodes The set of cross-domain edge neighbors, For cross-domain edge attention weights, This forms the basis for the hierarchical association features output by the second layer, i.e., the knowledge context features;

[0048] Step 3.2.3, Knowledge Context Feature Generation: Based on Cross-Domain Edge Attention Weights Hierarchical association features of the second layer output Configure the number of inference paths, perform multi-hop inference on subgraphs related to the current user scenario, and output knowledge context features. The formula is:

[0049]

[0050] in, For the first The number of nodes on the path, For the node To the node Cumulative attention weight, Characteristics representing a single path, The path weight for each path, The number of configurable inference paths.

[0051] A further improvement of this application is that step 3.3 specifically includes the following steps:

[0052] Step 3.3.1, Training Phase Constraints: Introduce a penalty term into the loss function, with the following formula:

[0053]

[0054] in, For multi-label classification loss, To recommend prediction vectors, For the actual label vector, The penalty coefficient is... For the domain constraint penalty term, the calculation logic is as follows: Medium probability > The recommended target pair, if its field risk level is The cumulative penalty value is 1.0 for each high-risk pair. The formula is as follows:

[0055] in, This is an indicator function; it returns 1 if the condition is met, and 0 otherwise.

[0056] Step 3.3.2, Reasoning Stage Constraints: Based on the cross-domain edge relationships of the knowledge graph, filter the recommended candidate objects and remove objects that are not compatible with the user scenario or have high-risk interactions with historical interactions; re-rank the remaining candidate objects according to the core entity-related entity compatibility relationship weight and security weight, output the re-ranked security weight, and prioritize recommending objects with high rationality and low risk.

[0057] A further improvement in this application is that step 4 specifically includes the following steps:

[0058] Step 4.1, Collaborative Fusion: Combine the multimodal scene sequence features obtained in Step 2. With the knowledge context features obtained in step 3 By concatenating the features along the feature dimension, we obtain enhanced features that integrate multimodal data and knowledge. :

[0059]

[0060] Through a two-layer fully connected network Perform feature optimization and output the optimized fused features. ;

[0061] Step 4.2, Recommendation and Prediction: [The text abruptly ends here, likely due to an incomplete sentence Input the recommendation prediction model, with the loss function Training to optimize the target involves the following steps:

[0062] Step 4.2.1, Temporal Adaptation Layer: The recommendation prediction model's LSTM network is used to... Temporal modeling is performed. Based on the security weights output in step 3.3.2, low attention weights are assigned to the feature components of risk-related entities in the user's historical multimodal action sequence features, and temporal adaptation features are output. ;

[0063] Step 4.2.2, Classification Prediction Layer: Adapting Temporal Features The input consists of a two-layer fully connected classifier and a regressor. For classification, either a sigmoid or softmax activation function is used to output the recommendation probability of each object, removing entities that are irrelevant to the user scenario or have high-risk interactions. For regression, continuous values ​​are output directly, with a dynamic threshold set. Objects with a recommendation probability greater than the dynamic threshold are considered candidate recommendations, and the highest-probability objects are selected in descending order of probability and value. One is used as the final recommendation list, and the knowledge graph reasoning path is explained.

[0064] This application includes a recommendation prediction model, which comprises a CodeBERT encoder, a multimodal feature processing layer, a knowledge graph enhancement layer, and a collaborative fusion and recommendation prediction layer. The CodeBERT encoder of the recommendation prediction model includes a LayerNorm layer and a Multi-Headed Attention layer; the multimodal feature processing layer of the recommendation prediction model includes a text-data attention alignment layer, two fully connected layers, and... The recommendation prediction model includes a knowledge graph enhancement layer and a temporal attention layer; the knowledge graph enhancement layer of the recommendation prediction model includes a graph attention network (GAT), which contains two layers of GATConv; the collaborative fusion and recommendation prediction layer of the recommendation prediction model includes two fully connected networks, a temporal adaptation layer, and a classification prediction layer, wherein the temporal adaptation layer is an LSTM network, and the classification prediction layer includes two fully connected classifiers and a regressor.

[0065] The beneficial effects of this application are:

[0066] This application proposes a multimodal feature encoding method based on dual encoders and dynamic weight fusion, breaking down the modal barriers between structured and unstructured information. The CodeBERT encoder preserves the hierarchical relationships and co-occurrence logic of structured data, while a domain-pre-trained model extracts the deep semantics of unstructured information. Finally, a cross-modal attention alignment mechanism strengthens the semantic connections between the two types of data.

[0067] This application addresses compliance requirements across different domains by constructing a two-layer knowledge graph comprising core entities, related entities, operations, and constraint rules. It combines this with Graph Attention Networks (GAT) to learn complex relationships between entities and employs a two-way constraint mechanism—penalty during training and filtering during inference—to control risk. This mechanism effectively addresses the shortcomings of existing recommendation methods, such as a lack of domain knowledge constraints and high security risks, significantly improving the scenario adaptability and reliability of recommendation results.

[0068] This application designs a temporal attention layer based on LSTM / Transformer, which accurately captures the state changes of users and objects by dynamically calculating the weight contributions of multiple rounds of scenarios. This design effectively overcomes the shortcomings of existing technologies in capturing temporal information, making the recommendation results more in line with the dynamic needs of users and adaptable to complex scenarios of multi-round interactions in various fields. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating an embodiment of this application.

[0070] Figure 2 This is a flowchart illustrating the multimodal recommendation model of this application.

[0071] Figure 3 This is a flowchart of the multimodal drug recommendation model in an embodiment of this application. Detailed Implementation

[0072] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. That is, in some embodiments of the present invention, these practical details are not essential. In addition, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0073] This application presents a multimodal recommendation method based on knowledge graph enhancement. This multimodal recommendation method is implemented through a recommendation prediction model, such as... Figure 2 As shown, the recommendation prediction model includes a CodeBERT encoder, a multimodal feature processing layer, a knowledge graph enhancement layer, and a collaborative fusion and recommendation prediction layer. The CodeBERT encoder of the recommendation prediction model includes a LayerNorm layer and a Multi-Headed Attention layer; the multimodal feature processing layer of the recommendation prediction model includes a text-data attention alignment layer, two fully connected network layers, and... The recommendation prediction model includes a knowledge graph enhancement layer and a temporal attention layer; the knowledge graph enhancement layer of the recommendation prediction model includes a graph attention network (GAT), which contains two layers of GATConv; the collaborative fusion and recommendation prediction layer of the recommendation prediction model includes two fully connected networks, a temporal adaptation layer, and a classification prediction layer, wherein the temporal adaptation layer is an LSTM network, and the classification prediction layer includes two fully connected classifiers and a regressor. Specifically, the multimodal recommendation method of this application includes the following steps:

[0074] Step 1: Data Acquisition and Preprocessing: Acquire multi-source data, including structured and unstructured data. Preprocess the acquired multi-source data to generate user-single-scenario level data samples, which include a scenario identifier table, a core attribute table, an interaction behavior table, an operation record table, unstructured information data, and an auxiliary mapping table.

[0075] In step 1, the preprocessing of multi-source data includes the following steps:

[0076] Step 1.1, Structured Data Preprocessing: Non-standard data in the interaction behavior table is converted into domain-specific data through a corresponding mapping table. Null or invalid identifier records in the core attribute table and operation record table are filtered out. Deduplication is performed by grouping by user unique identifier - scene unique identifier. Based on domain vocabulary, the core identifiers are converted into ID sequences that the recommendation prediction model can recognize. The core identifiers mentioned here refer to the core attribute table, as well as the identifier information that represents key information and plays a core role in subsequent processing and analysis in the multi-source data such as the interaction behavior table, core attribute table, and operation record table, after conversion and encoding.

[0077] Step 1.2, Unstructured Information Preprocessing: Filter the category records in the unstructured information data that match the recommendation requirements, filter out categories and special characters that are not related to the recommendation requirements, and align the preprocessed unstructured data with the user scenario-related records generated based on the scenario identifier table, interaction behavior table, etc., according to the scenario time in the scenario identifier table to obtain aligned unstructured information;

[0078] Step 1.3, Data Integration: Using the user's unique identifier and the scene's unique identifier as dual identifiers, the ID sequence that the preprocessed recommendation prediction model can recognize is inner-joined with the aligned unstructured information obtained after step 1.2. Invalid samples without core attributes or unstructured information are removed to generate user-single scene-level data samples, and user scene auxiliary features are calculated at the same time.

[0079] Step 2, Multimodal Feature Processing: The user-single scene-level data samples generated in Step 1 are processed through structured labeling, unstructured information processing, and text-semantic alignment and hierarchical fusion to obtain multimodal action sequence features.

[0080] The structured labeling process in step 2 involves encoding the preprocessed user-single-scene-level data samples and user-scene auxiliary features using a CodeBERT encoder. The CodeBERT encoder includes a LayerNorm layer (adapting to the normalization of discrete features) and a Multi-Headed Attention layer (strengthening semantic associations), specifically comprising the following steps:

[0081] Step 2.1.1 Input Construction: The core attribute table, operation record table, and interaction behavior table for a single scenario are processed into text, forming core attribute text sequences, operation record text sequences, interaction behavior text sequences, and user scenario auxiliary features, respectively. These are then combined in the following order: the identifier representing the start of classification, the core attribute text sequence, the identifier used for separation, the operation record text sequence, the identifier used for separation, the interaction behavior text sequence, the identifier used for separation, and the user scenario auxiliary features, to form the input content for subsequent processing.

[0082] Step 2.1.2, Embedding Learning: Construct trainable embedding vectors for the core attribute text sequence, operation record text sequence, interaction behavior text sequence, and user scenario auxiliary features, respectively, preserving the hierarchical structure and co-occurrence context of the core attribute text sequence, operation record text sequence, and interaction behavior text sequence; introduce a top-down parent node aggregation strategy to aggregate the constructed trainable embedding vectors hierarchically into a scene-level structured feature representation. .

[0083] In step 2, unstructured information processing involves using a domain-specific pre-trained language model to label the pre-processed user-single-scenario-level data samples. This specifically includes the following steps:

[0084] Step 2.2.1 Input Processing: The aligned unstructured information obtained in Step 1.2 is truncated or supplemented according to the maximum sequence length (supplementing specific identifiers) to generate an unstructured ID sequence;

[0085] Step 2.2.2, Feature Extraction: Using the encoder layer of the domain-pre-trained language model, unstructured semantic features are extracted from the generated unstructured ID sequence in Step 2.2.1. The pooling vector corresponding to the starting identifier of the ID sequence is taken as the scene-level unstructured feature representation. .

[0086] Step 2, text-semantic alignment and hierarchical fusion, specifically includes the following steps:

[0087] Step 2.3.1, Text-Code Alignment: Construct a text-data attention alignment layer and compute scene-level structured feature representations. and scene-level unstructured feature representation cosine similarity The formula is:

[0088]

[0089] Among them, cosine similarity The range of values ​​is cosine similarity The closer the value is to 1, the higher the semantic correlation between scene-level structured features and scene-level unstructured features, which is used to filter strongly related data fragments. For scene-level unstructured feature representation The One dimension, For scene-level structured features The One dimension, Using the feature vector dimension, based on similarity weights, unstructured fragments that are strongly related to user status and needs are highlighted, while noisy text is filtered out;

[0090] Step 2.3.2, Multimodal hierarchical fusion: including action fusion and sequence-level fusion, specifically:

[0091] Action fusion: aligning scene-level structured feature representations and scene-level unstructured feature representation The feature dimension is concatenated using a two-layer fully connected network and... The layers are used for feature fusion to obtain multimodal features of a single action. ;

[0092] Sequence-level fusion: The user's multi-turn actions are sorted by action time, and a temporal attention layer is constructed to calculate the feature representation of each action. The weights are calculated using the following formula:

[0093]

[0094]

[0095] in, For action sequence number, The number of times the action is performed. , It is a linear projection function. For bias, For the first Attention weights for sub-scene features, and multimodal features of a single action. We obtain the user's multimodal action sequence features by weighted summation. .

[0096] Step 3, Domain Knowledge Graph Construction and Enhancement: Construct a domain knowledge graph and use a graph attention network to perform embedding learning and reasoning on the domain knowledge graph to obtain knowledge context features. This includes the following steps:

[0097] Step 3.1, Domain Knowledge Graph Construction: Construct a domain knowledge graph containing four types of entities—core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes—as well as multiple types of relationships. ,in At the node level, For edge set, For a set of node types, Let V be the set of edge types; where V is the node set, including core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes; E is the edge set, including hierarchical edges (core entity parent-child or ancestor-descendant, associated entity parent-child or ancestor-descendant, operation node parent-child or ancestor-descendant), and cross-domain edges (core entity-associated entity adaptation relationship, associated entity-associated entity interaction relationship, core entity-operation adaptation relationship); A is the node type set, consisting of {core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes}; R is the edge type set, consisting of {core entity hierarchical relationship, associated entity hierarchical relationship, operation hierarchical relationship, core entity-associated entity adaptation relationship, associated entity-associated entity interaction relationship, and core entity-operation adaptation relationship}; edge weight assignment: the weight of hierarchical edges is set to 1.0; the weight of core entity-associated entity adaptation relationship edges is assigned based on domain guidelines or quasi-recommendation degree; the weight of associated entity-associated entity interaction relationship edges is assigned based on the risk value in the constraint rule table.

[0098] Step 3.2, Knowledge Graph Embedding and Graph Reasoning: Graph Attention Network (GAT) is used to embed the domain knowledge graph constructed in Step 3.1. Embedded learning and graph reasoning are performed. The graph attention network contains two layers of GATConv, and the knowledge dimension is set as a configurable hyperparameter. The specific steps include:

[0099] Step 3.2.1, Node Embedding Initialization: Align the initial features of core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes with the ID mapping in the domain's auxiliary mapping to generate configurable initial embedding vectors.

[0100] Step 3.2.2, Graph Attention Calculation: First layer GATConv, processing hierarchical edges: Calculate attention for hierarchical edges of core entity nodes, associated entity nodes, and operation nodes. The formula is:

[0101]

[0102]

[0103]

[0104] in, The central node currently of interest in the first layer. and its neighboring nodes Some kind of interaction feature or embedding between them For neighboring nodes Embedded vector, Represents the current central focus. represent The neighboring nodes, For a leaky linear rectified activation function, For nodes The initial embedding, This is the first layer weight matrix. This is the first layer attention vector. Indicates transpose. For nodes The hierarchical set of edge neighbors, For the attention weights of the hierarchical edges; The hierarchical association features output from the first layer. Indicates the previously concerned central node and its neighboring nodes Some kind of interaction feature or embedded representation between them;

[0105] The second layer, GATConv, handles cross-domain edges: it performs attention calculations on cross-domain edges, which are core entity node-related entity node, related entity node-related entity node, core entity node-operation node, and related entity node-constraint rule node. The attention calculation formula is as follows:

[0106]

[0107] in, The central node currently of interest in the second layer. and its neighboring nodes Some kind of interaction feature or embedding between them This is the weight matrix for the second layer. This is the second layer attention vector. For nodes The set of cross-domain edge neighbors, For cross-domain edge attention weights, This forms the basis for the hierarchical association features output by the second layer, i.e., the knowledge context features;

[0108] Step 3.2.3, Knowledge Context Feature Generation: Based on Cross-Domain Edge Attention Weights Hierarchical association features of the second layer output Configure the number of inference paths, perform multi-hop inference on subgraphs related to the current user scenario, and output knowledge context features. The formula is:

[0109]

[0110] in, For the first The number of nodes on the path, For the node To the node Cumulative attention weight, Characteristics representing a single path, The path weight for each path, The number of configurable inference paths.

[0111] Step 3.3, Knowledge Constraints and Security Controls. This specifically includes the following steps:

[0112] Step 3.3.1, Training Phase Constraints: Introduce a penalty term into the loss function, with the following formula:

[0113]

[0114] in, For multi-label classification loss, To recommend prediction vectors, For the actual label vector, The penalty coefficient is... For the domain constraint penalty term, the calculation logic is as follows: Medium probability > For recommended pairs, if their domain risk level is "high", then the cumulative penalty value is 1.0 for each high-risk pair. The formula is as follows:

[0115] in, This is an indicator function; it returns 1 if the condition is met, and 0 otherwise.

[0116] Step 3.3.2, Reasoning Stage Constraints: Based on the cross-domain edge relationships of the knowledge graph, filter the recommended candidate objects and remove objects that are not compatible with the user scenario or have high-risk interactions with historical interactions; re-rank the remaining candidate objects according to the core entity-related entity compatibility relationship weight and security weight, output the re-ranked security weight, and prioritize recommending objects with high rationality and low risk.

[0117] Step 4: Multimodal and Knowledge Graph Collaborative Fusion and Recommendation Prediction: The multimodal action sequence features obtained in Step 2 and the knowledge context features obtained in Step 3 are fused. The fused features are then input into the recommendation prediction model for processing to obtain the final recommendation table. This includes the following steps:

[0118] Step 4.1, Collaborative Fusion: Combine the multimodal scene sequence features obtained in Step 2. With the knowledge context features obtained in step 3 By concatenating the features along the feature dimension, we obtain enhanced features that integrate multimodal data and knowledge. :

[0119]

[0120] Through a two-layer fully connected network Perform feature optimization and output the optimized fused features. ;

[0121] Step 4.2, Recommendation and Prediction: [The text abruptly ends here, likely due to an incomplete sentence Input the recommendation prediction model, with the loss function Training is performed to optimize the objective. This includes the following steps:

[0122] Step 4.2.1, Temporal Adaptation Layer: The recommendation prediction model's LSTM network is used to... Temporal modeling is performed. Based on the security weights output in step 3.3.2, low attention weights are assigned to the feature components of risk-related entities in the user's historical multimodal action sequence features, and temporal adaptation features are output. ;

[0123] Step 4.2.2, Classification Prediction Layer: Adapting Temporal Features The input consists of a two-layer fully connected classifier and a regressor. For classification, either a sigmoid or softmax activation function is used to output the recommendation probability of each object, removing entities that are irrelevant to the user scenario or have high-risk interactions. For regression, continuous values ​​are output directly, with a dynamic threshold set. Objects with a recommendation probability greater than the dynamic threshold are considered candidate recommendations, and the highest-probability objects are selected in descending order of probability and value. This serves as the final recommendation list and explanation of the knowledge graph reasoning path. The training process of the recommendation prediction model follows the loss function defined in step 3.3.1. To optimize the objective, the recommendation probability output by the classification prediction layer should both conform to the true label distribution and avoid high-risk combinations of recommended objects.

[0124] Example

[0125] Suppose Company A constructs a clinical drug recommendation model based on multimodal fusion and knowledge graph enhancement. This model recommends appropriate treatment drugs for the next stage of treatment to doctors based on patients' historical medical data, such as admission records, diagnoses, clinical notes, and medication history. The embodiments and features described herein can be combined with each other; and all model training is performed using Python programming on an Intel Core 13th generation 13400F processor and an NVIDIA RTX 4060Ti graphics card. The construction of this model requires following these implementation steps, and the system flow is as follows: Figure 1 Specifically, the process includes the following steps:

[0126] Step S1: Medical Data Collection and Preprocessing

[0127] 1. Data Collection: Acquire structured medical data and unstructured data, and simultaneously acquire the NDC-ATC4 drug code mapping table, ICD-9 diagnostic glossary, and drug glossary. Ensure that key fields, namely patient ID, admission ID, diagnosis code, drug code, and clinical note text, are complete and without missing information. Among them, the structured medical data includes the admission information table, diagnosis code table, prescription drug table, and surgical code table, while the unstructured data is the clinical note table.

[0128] 2. Data Preprocessing: Optimization operations are performed to address the heterogeneous nature of medical data: NDC codes in the prescription drug table are converted to ATC4-level codes via a mapping table; empty records in the diagnosis code table are filtered; and diagnosis, drug, and surgical codes are converted into model-recognizable integer IDs based on a vocabulary. Discharge summary texts in the clinical notes table are filtered to remove privacy markers, such as anonymized fields and special characters related to doctor's name and patient ID number; the text is aligned with the patient's medical records by admission time. Using patient ID and admission ID as association keys, the preprocessed structured data is joined with the text data to generate patient-single-visit level data samples, providing standardized input for subsequent multimodal coding.

[0129] Step S2: Construction of medical knowledge graph.

[0130] 1. Graph Entity and Relationship Definition: Taking "disease-drug-surgery" as the core entity, hierarchical relationships are defined based on the ICD-9 diagnostic tree and ATC drug tree, and cross-domain relationships are defined by combining clinical guidelines and drug interaction (DDI) data;

[0131] 2. Edge connection and weight assignment: A two-stage edge construction strategy is adopted. In the first stage, the direct parent-child edges of disease, drug and surgical entities are constructed. In the second stage, the long-distance edges from ancestor nodes to leaf nodes are constructed.

[0132] 3. Knowledge graph storage: The constructed knowledge graph is stored in the PyTorch Geometric Data object format, which includes node feature matrix, edge index matrix, and edge weight vector, to ensure that subsequent models can directly call it.

[0133] Step S3: Construction of a multimodal drug recommendation model, as follows Figure 3 As shown:

[0134] S3.1 Multimodal Feature Encoding: The structured medical code sequence is processed using the CodeBERT encoder; then the clinical text data is processed using the ClinicalBERT encoder, and key text fragments are enhanced through a "text-code attention alignment layer"; finally, the processed structured medical code sequence is concatenated with the clinical text data, and after regularization by the dropout layer, multimodal fusion features are obtained.

[0135] S3.2 Knowledge Graph Enhancement: The constructed knowledge graph is input into the Graph Attention Network (GAT), and the knowledge context features are learned through attention weight calculation. These features are then concatenated with multimodal fusion features to generate enhanced features for multimodal and knowledge aspects.

[0136] S3.3 Temporal Modeling and Prediction: Temporally align the enhanced features of patients' multiple visits, calculate the weight of each visit through the temporal attention layer, input the classification prediction layer to output the drug recommendation probability, combine the knowledge graph DDI relationship to filter high-risk drugs, and generate a Top 5 recommendation list.

[0137] Step S4: Model Training and Optimization

[0138] S4.1 Data partitioning: The preprocessed dataset is stratified and divided into training set, validation set and test set according to the proportion, to ensure that the patient visit distribution in the training set and the test set is consistent.

[0139] S4.2 Model Training: Start training via script, fix core parameters such as random seed, number of training rounds, batch size, and learning rate, and conduct multiple sets of experiments by traversing knowledge dimensions and the number of inference paths; use Adam optimizer, freeze the parameters of the first 10 layers of ClinicalBERT, use BCEWithLogitsLoss combined with DDI penalty term to optimize the model, and save the model weights that are optimal for F1 on the validation set.

[0140] S4.3 Hyperparameter Tuning: Evaluate the macro-average F1, AUC and other metrics of each experimental combination on the validation set to determine the optimal hyperparameter configuration.

[0141] Step S5: Drug Recommendation and Evaluation

[0142] S5.1 Recommendation Reasoning: Load the optimal model weights, input the test set patient data, and output a drug recommendation list with knowledge path explanations through multimodal coding, knowledge reasoning, and time-series modeling processes.

[0143] S5.2 Performance Evaluation: The F1, Accuracy, and AUC metrics are used to quantify the model performance and compare its advantages with existing single-modal model verification techniques.

[0144] Step S6: Clinical Deployment and Monitoring of the Model

[0145] Once the model's performance reaches a clinically acceptable level, it will be deployed to a real-world hospital application environment.

[0146] 1. Deployment method: The model can be encapsulated as an API interface and integrated into the hospital HIS system, supporting doctors to call the recommendation function through patient ID;

[0147] 2. Performance monitoring and updates: Real-time monitoring of model recommendation accuracy, doctor adoption rate, and the proportion of DDI risk combinations; fine-tuning of model parameters every 3 months using new clinical data; synchronous updates of the knowledge graph to adapt to the dynamic changes in medical data and knowledge.

[0148] Through the above implementation steps, Company A can construct a drug recommendation model based on multimodal fusion and knowledge graph enhancement, providing clinicians with integrated decision support that includes data support and knowledge interpretation. Hyperparameters can be flexibly adjusted during implementation to ensure the model is adaptable to different clinical scenarios.

[0149] Effect demonstration:

[0150] This embodiment selects the publicly available medical dataset MIMIC-III (Medical Information Mart for Intensive Care III) to verify the effectiveness of the proposed knowledge graph-enhanced multimodal medical drug recommendation method. The MIMIC-III dataset contains electronic health records of over 40,000 critically ill patients, covering rich structured medical codes such as ICD diagnostic codes and ATC drug codes, unstructured clinical texts such as medical records, discharge summaries, and long-term multi-round visit time-series information, which can fully simulate real-world clinical drug recommendation scenarios. The results of the comparative test are shown in Table 1.

[0151] Table 1 Comparison of Method Results

[0152]

[0153] In Table 1, LR is a traditional machine learning method that uses a binary correlation method for multi-label classification; RNN uses patient record sequences as input, learns the hidden representation of the patient, and performs binary classification based on the hidden state; Med-BERT adapts the BERT framework from the field of natural language processing to structured electronic medical records, defining serialized embeddings to represent the relative order of each code. The methods in this application are compared with other methods, demonstrating that this application outperforms other current models.

[0154] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A multimodal recommendation method based on knowledge graph enhancement, characterized in that: The multimodal recommendation method is implemented through a recommendation prediction model, which includes a CodeBERT encoder, a multimodal feature processing layer, a knowledge graph enhancement layer, and a collaborative fusion and recommendation prediction layer. The CodeBERT encoder of the recommendation prediction model includes a LayerNorm layer and a Multi-Headed Attention layer; the multimodal feature processing layer of the recommendation prediction model includes a text-data attention alignment layer, two fully connected layers, and... The recommendation prediction model includes a knowledge graph enhancement layer and a temporal attention layer; the knowledge graph enhancement layer of the recommendation prediction model includes a graph attention network, which contains two layers of GATConv; the collaborative fusion and recommendation prediction layer of the recommendation prediction model includes two fully connected networks, a temporal adaptation layer, and a classification prediction layer, wherein the temporal adaptation layer is an LSTM network, and the classification prediction layer includes two fully connected classifiers and a regressor. The specific multimodal recommendation method includes the following steps: Step 1: Data Acquisition and Preprocessing: Acquire multi-source data, including structured and unstructured data. Preprocess the acquired multi-source data to generate user-single-scenario level data samples and user-scenario auxiliary features. The user-single-scenario level data samples include a scenario identifier table, a core attribute table, an interaction behavior table, an operation record table, unstructured information data, and an auxiliary mapping table. The structured medical data includes an admission information table, a diagnosis code table, a prescription drug table, and a surgical code table. The unstructured data includes a clinical note table. Step 2, Multimodal Feature Processing: The user-single-scene-level data samples and user scene auxiliary features generated in Step 1 are processed through structured labeling, unstructured information processing, and text-semantic alignment and hierarchical fusion to obtain multimodal action sequence features; Step 3, Domain Knowledge Graph Construction and Enhancement: Construct a domain knowledge graph and use a graph attention network to perform embedding learning and reasoning on the domain knowledge graph to obtain knowledge context features. This includes the following steps: Step 3.1, Domain Knowledge Graph Construction: Construct a domain knowledge graph containing four types of entities—core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes—as well as multiple types of relationships. ,in At the node level, For edge set, For a set of node types, For edge type set; Step 3.2, Knowledge Graph Embedding and Graph Reasoning: A graph attention network is used to embed the domain knowledge graph constructed in Step 3.

1. Embedded learning and graph reasoning are performed. The graph attention network contains two layers of GATConv. The knowledge dimension is set as a configurable hyperparameter. Graph attention calculation: The first layer of GATConv processes hierarchical edges: attention calculation is performed on hierarchical edges of core entity nodes, associated entity nodes, and operation nodes. The second layer of GATConv processes cross-domain edges: attention calculation is performed on cross-domain edges, which are core entity node-associated entity node, associated entity node-associated entity node, core entity node-operation node, and associated entity node-constraint rule node. Step 3.3: Knowledge Constraints and Security Controls; Step 4, Multimodal and Knowledge Graph Collaborative Fusion and Recommendation Prediction: The multimodal action sequence features obtained in Step 2 and the knowledge context features obtained in Step 3 are fused together. The fused features are then input into the recommendation prediction model for processing to obtain the final recommendation table.

2. The multimodal recommendation method based on knowledge graph enhancement according to claim 1, characterized in that: In step 1, the preprocessing of multi-source data includes the following steps: Step 1.1, Structured Data Preprocessing: Convert non-standard data in the interaction behavior table into domain-specific data through the corresponding mapping table, filter out null or invalid identifier records in the core attribute table and operation record table, group and deduplicate by user unique identifier - scene unique identifier, and convert the core identifiers into ID sequences that the recommendation prediction model can recognize based on domain vocabulary. Step 1.2, Unstructured Information Preprocessing: Filter the category records in the unstructured information data that match the recommendation requirements, filter out categories and characters that are not related to the recommendation requirements, and align the preprocessed unstructured data with the user scene records generated based on the scene identifier table and the interaction behavior table according to the scene time in the scene identifier table to obtain aligned unstructured information. Step 1.3, Data Integration: Using the user's unique identifier and the scene's unique identifier as dual identifiers, the ID sequence that the preprocessed recommendation prediction model can recognize is internally connected with the aligned unstructured information obtained after aligning in Step 1.

2. Invalid samples are removed, and user-single scene-level data samples are generated. At the same time, user scene auxiliary features are calculated.

3. The multimodal recommendation method based on knowledge graph enhancement according to claim 1, characterized in that: Step 2 involves structured labeling, which encodes the preprocessed user-single-scene-level data samples and user-scene auxiliary features using a CodeBERT encoder. The CodeBERT encoder includes a LayerNorm layer adapted to encode discrete features within the domain and a Multi-Headed Attention layer. Specifically, it includes the following steps: Step 2.1.1 Input Construction: The core attribute table, operation record table, and interaction behavior table for a single scenario are processed into text, forming core attribute text sequences, operation record text sequences, interaction behavior text sequences, and user scenario auxiliary features, respectively. These are then combined in the following order: the identifier representing the start of classification, the core attribute text sequence, the identifier used for separation, the operation record text sequence, the identifier used for separation, the interaction behavior text sequence, the identifier used for separation, and the user scenario auxiliary features, to form the input content for subsequent processing. Step 2.1.2, Embedding Learning: Construct trainable embedding vectors for the core attribute text sequence, operation record text sequence, interaction behavior text sequence, and user scenario auxiliary features, respectively, preserving the hierarchical structure and co-occurrence context of the core attribute text sequence, operation record text sequence, and interaction behavior text sequence; introduce a top-down parent node aggregation strategy to aggregate the constructed trainable embedding vectors hierarchically into a scene-level structured feature representation. .

4. The multimodal recommendation method based on knowledge graph enhancement according to claim 1, characterized in that: In step 2, unstructured information processing involves using a domain-specific pre-trained language model to label the pre-processed user-single-scenario-level data samples. This specifically includes the following steps: Step 2.2.1 Input Processing: The aligned unstructured information obtained in Step 1.2 is truncated according to the maximum sequence length to generate an unstructured ID sequence; Step 2.2.2, Feature Extraction: Using the encoder layer of the domain-pre-trained language model, unstructured semantic features are extracted from the generated unstructured ID sequence in Step 2.2.

1. The pooling vector corresponding to the starting identifier of the ID sequence is taken as the scene-level unstructured feature representation. .

5. The multimodal recommendation method based on knowledge graph enhancement according to claim 1, characterized in that: Step 2, text-semantic alignment and hierarchical fusion, specifically includes the following steps: Step 2.3.1, Text-Code Alignment: Construct a text-data attention alignment layer and compute scene-level structured feature representations. and scene-level unstructured feature representation cosine similarity The formula is: ， Among them, cosine similarity The range of values ​​is cosine similarity The closer the value is to 1, the higher the semantic correlation between scene-level structured features and scene-level unstructured features, which is used to filter strongly related data fragments. For scene-level unstructured feature representation The One dimension, For scene-level structured features The One dimension, The dimension of the feature vector; Step 2.3.2, Multimodal hierarchical fusion: including action fusion and sequence-level fusion, specifically: Action fusion: aligning scene-level structured feature representations and scene-level unstructured feature representation The feature dimension is concatenated using a two-layer fully connected network and... The layers are used for feature fusion to obtain multimodal features of a single action. ; Sequence-level fusion: The user's multi-turn actions are sorted by action time, and a temporal attention layer is constructed to calculate the feature representation of each action. The weights are calculated using the following formula: ， ， in, For action sequence number, The number of times the action is performed. , It is a linear projection function. For bias, For the first Attention weights for sub-scene features, and multimodal features of a single action. We obtain the user's multimodal action sequence features by weighted summation. .

6. The multimodal recommendation method based on knowledge graph enhancement according to claim 1, characterized in that: Step 3.2 specifically includes the following steps: Step 3.2.1, Node Embedding Initialization: Align the initial features of core entity nodes, associated entity nodes, operation nodes, and constraint rule nodes with the ID mapping in the domain's auxiliary mapping to generate configurable initial embedding vectors. Step 3.2.2, the formula for calculating attention in the first layer is: ， ， ， in, The central node currently of interest in the first layer. and its neighboring nodes Interaction features or embeddings between them For neighboring nodes Embedded vector, Represents the current central focus. represent The neighboring nodes, For a leaky linear rectified activation function, For nodes The initial embedding, This is the first layer weight matrix. This is the first layer attention vector. Indicates transpose. For nodes The hierarchical set of edge neighbors, For the attention weights of the hierarchical edges; The hierarchical association features output from the first layer. Indicates the previously concerned central node and its neighboring nodes Interaction features or embedded representations between them; The formula for attention calculation in the second layer GATConv is: ， ， ， in, The central node currently of interest in the second layer. and its neighboring nodes Interaction features or embeddings between them This is the weight matrix for the second layer. This is the second layer attention vector. For nodes The set of cross-domain edge neighbors, For cross-domain edge attention weights, The hierarchical association features output from the second layer; Step 3.2.3, Knowledge Context Feature Generation: Based on Cross-Domain Edge Attention Weights Hierarchical association features of the second layer output Configure the number of inference paths, perform multi-hop inference on subgraphs related to the current user scenario, and output knowledge context features. The formula is: ， in, For the first The number of nodes on the path, For the node To the node Cumulative attention weight, Characteristics representing a single path, The path weight for each path, The number of configurable inference paths.

7. The multimodal recommendation method based on knowledge graph enhancement according to claim 6, characterized in that: Step 3.3 specifically includes the following steps: Step 3.3.1, Training Phase Constraints: Introduce a penalty term into the loss function, with the following formula: ， in, For multi-label classification loss, To recommend prediction vectors, For the actual label vector, The penalty coefficient is... For the domain constraint penalty term, the calculation logic is as follows: Medium probability > The recommended target pair, if its field risk level is Then the penalty value is accumulated. The formula is as follows: ， in, This is an indicator function; it returns 1 if the condition is met, and 0 otherwise. Step 3.3.2, Inference Stage Constraints: Based on the cross-domain edge relationships of the knowledge graph, filter the recommended candidate objects and remove objects that are not compatible with the user scenario or have risky interactions with historical interactions; reorder the remaining candidate objects according to the core entity-related entity compatibility relationship weight and security weight, and output the reordered security weight.

8. The multimodal recommendation method based on knowledge graph enhancement according to claim 7, characterized in that: Step 4 specifically includes the following steps: Step 4.1, Collaborative Fusion: Combine the multimodal scene sequence features obtained in Step 2. With the knowledge context features obtained in step 3 By concatenating the features along the feature dimension, we obtain enhanced features that integrate multimodal data and knowledge. : ， Through a two-layer fully connected network Perform feature optimization and output the optimized fused features. ; Step 4.2, Recommendation and Prediction: [The text abruptly ends here, likely due to an incomplete sentence Input the recommendation prediction model, with the loss function Training to optimize the objective involves the following steps: Step 4.2.1, Temporal Adaptation Layer: The recommendation prediction model's LSTM network is used to... Temporal modeling is performed. Based on the security weights output in step 3.3.2, low attention weights are assigned to the feature components of risk-related entities in the user's historical multimodal action sequence features, and temporal adaptation features are output. ; Step 4.2.2, Classification Prediction Layer: Adapting Temporal Features Input a two-layer fully connected classifier and a regressor. In the classification scenario, output the recommendation probability of each object and remove invalid entities. In the regression scenario, output continuous values ​​directly. Set a dynamic threshold; objects with a recommendation probability greater than the dynamic threshold are selected as candidate recommendations, and then ranked in descending order of recommendation probability and value. One of these will be included in the final recommendation list.

Images