Decision optimization method fusing enhanced multi-modal learning and knowledge graph

Abstract

The invention discloses a decision optimization method fusing enhanced multi-modal learning and a knowledge graph, and relates to the technical field of artificial intelligence and knowledge graphs. According to the decision optimization method for fusing enhanced multi-modal learning and the knowledge graph, dynamic fusion of multi-modal features is realized through a dynamic weight adjustment and semantic alignment constraint mode, semantic precision and interpretability are improved, the semantic deviation problem caused by traditional static feature fusion is effectively solved, and the method has the advantages of being high in robustness and high in reliability. And by recording an intelligent reasoning path selected by a hierarchical reinforcement learning agent, interactive graph structure display can be carried out, the advantage of transparency is achieved, meanwhile, the interpretability of the path is further improved in cooperation with a multi-target reward function, intelligent updating of the knowledge graph is carried out in cooperation with comprehensive confidence, intelligent growth of the knowledge graph is achieved, and the intellectual property of the knowledge graph is improved. And a fine-grained interpretable report is generated through an adversarial training mechanism and anti-factual reasoning, so that the false alarm rate of an output result is further reduced, and the decision transparency is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and knowledge graph, and specifically provides a decision optimization method integrating enhanced multi-modal learning and knowledge graph. Background Art

[0002] Knowledge graph, also known as scientific knowledge graph, is a concept in the field of library and information science, used to map, analyze and display the interconnections between subjects or academic research entities. It is a visualization tool for revealing the development process and structural relationships of scientific knowledge. In most cases, the knowledge graph is visually represented using a graph structure, with nodes representing authors, academic institutions, scientific literature or keywords, and connections representing the relationships between nodes.

[0003] Existing knowledge graphs do have problems with poor interpretability in applications. Especially when using complex models such as deep neural networks, the reasoning process is often regarded as a black box, lacking transparency. Moreover, multi-modal features often adopt the method of static feature fusion, which easily leads to semantic deviation and non-traceability. For example, the TransAE model uses weighted average or concatenation methods to fuse multi-modal data. And knowledge graphs usually adopt the method of manual supplementation, which not only has a large workload but also lacks intelligent growth.

[0004] In view of this, a decision optimization method integrating enhanced multi-modal learning and knowledge graph is specifically proposed. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a decision optimization method integrating enhanced multi-modal learning and knowledge graph, which solves the problems of poor interpretability, static feature fusion, easy semantic deviation and non-traceability, and lack of intelligent growth existing in the application of existing knowledge graphs.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A decision optimization method integrating enhanced multi-modal learning and knowledge graph specifically includes the following steps:

[0007] S1. Construct a multi-modal embedding space including text, images, and time-series data, and dynamically align the modal feature vectors through a meta-learning framework;

[0008] S2. Build a hierarchical reinforcement learning agent, whose state space includes the knowledge graph entity relationship topology structure and the multi-modal feature joint encoding vector;

[0009] S3. Define a dual action space: the main action space is used to control the selection of the knowledge graph relationship reasoning path, and the auxiliary action space is used to adjust the multi-modal data fusion weight;

[0010] S4. Construct a multi-objective reward function to drive the hierarchical reinforcement learning agent to generate an intelligent reasoning path;

[0011] S5. Construct an adversarial training mechanism to verify the rationality of the intelligent inference path, generate a differential explanation report through counterfactual reasoning, and perform incremental knowledge graph updates.

[0012] The present invention is further configured such that: in the S1, a gated attention mechanism is used to construct a cross-modal feature interaction channel to calculate the semantic correlation degree of text-image features. The calculation formula of the gated attention mechanism is:

[0013] α t，i =σ(W t ·h t +W i ·h i +b)

[0014] In the formula, α t，i is the attention weight of the text-image modality, α t，i ∈[0, 1]. When α t，i >0.7, the cross-modal attention mechanism is activated. σ is the sigmoid function, W t is the text trainable weight matrix for the linear transformation of text features, h t is the text feature vector, W i is the image trainable weight matrix for the linear transformation of image features, h i is the image feature vector, and b is the bias term parameter.

[0015] The present invention is further configured such that: the ways in which the auxiliary action space in the S3 is used to adjust the multi-modal data fusion weight include:

[0016]

[0017] In the formula, w m is the fusion weight of the m-th modality, exp is the exponential function for Softmax normalization, f m is the importance scoring function of the m-th modality feature, s is the current state vector, M is the total number of modalities, k is the modality traversal index, k∈[1, M], and f k is the importance scoring function of the k-th modality.

[0018] The present invention is further configured such that: the calculation formula of the multi-objective reward function in the S4 is:

[0019] R toal =R c +R u +R s

[0020] In the formula, R total is the total reward score, R cScore the path complexity penalty term, R u Score the user feedback reward item, R s Scoring semantic consistency measures;

[0021] The calculation formula of the path complexity penalty term is:

[0022] R c = -λ·log(1+n)

[0023] Where n is the number of path nodes, λ is the penalty weight coefficient;

[0024] The calculation formula for user feedback rewards is:

[0025] R u =β·G

[0026] Where G is the manual rating, G∈[0,1], β is the user feedback weight coefficient;

[0027] The calculation formula of semantic consistency measure is:

[0028] R s =γ·g

[0029] Where g is the intelligent reasoning path score and γ is the semantic weight coefficient.

[0030] The present invention is further configured as follows: the adversarial training mechanism in S5 includes:

[0031] Construct a discriminator network to distinguish between the intelligent reasoning path and the expert annotation path, and define the adversarial loss function:

[0032]

[0033] Where L is the adversarial training loss value, is the discriminator network, is a parameter, p r Marking paths for experts, p f It is the intelligent reasoning path.

[0034] The present invention is further configured such that the method of generating the difference explanation report by counterfactual reasoning in S5 includes:

[0035] The contribution of each triple to the prediction is calculated by the SHAP value.

[0036] Delete the triple with the highest contribution and observe the change in predicted probability.

[0037] If the predicted probability change value exceeds a threshold, the triple is marked as a decisive reasoning basis and a natural language explanation is generated.

[0038] The present invention is further configured such that the method for incrementally updating the knowledge graph in S5 includes:

[0039] When a new entity in the unstructured data is detected, calculate the semantic similarity between the new entity and the existing entities, screen the three candidate associated entities with the highest similarity, use a graph convolutional network to predict potential relationships, generate multiple candidate insertion paths based on a reinforcement learning agent, select the path with the highest comprehensive confidence, and when its comprehensive confidence exceeds a preset reference threshold, perform knowledge graph expansion.

[0040] The present invention is further configured such that the calculation method of the comprehensive confidence is:

[0041]

[0042] In the formula, the graph structure score is the rationality score for predicting potential relationships through a graph convolutional network.

[0043] The present invention provides a decision optimization method that fuses reinforcement multi-modal learning and a knowledge graph. It has the following beneficial effects:

[0044] (1) Through the methods of dynamic weight adjustment and semantic alignment constraints, the present invention realizes the dynamic fusion of multi-modal features, improves semantic accuracy and interpretability, effectively solves the semantic deviation problem caused by traditional static feature fusion, and by recording the intelligent reasoning paths selected by the hierarchical reinforcement learning agent, can perform interactive graph structure display, has the advantage of transparency, and further improves the interpretability of the path in cooperation with a multi-objective reward function, and performs intelligent update of the knowledge graph in cooperation with the comprehensive confidence, realizing the intelligent growth of the knowledge graph.

[0045] (2) Through the cooperation of an adversarial training mechanism and counterfactual reasoning, the present invention generates a fine-grained interpretable report, further reducing the false alarm rate of the output result and improving decision transparency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention.

[0048] Please refer to Figure 1 , the embodiments of the present invention provide the following technical solutions: A decision optimization method that fuses reinforcement multi-modal learning and a knowledge graph, specifically including the following steps:

[0049] S1. Construct a multi-modal embedding space including text, images, and time-series data, and dynamically align the modal feature vectors through a meta-learning framework. The processing methods include:

[0050] S11. Preprocessing of data

[0051] Text modality: Use the BERT model to extract the semantic vector of the entity description text, capture the global semantics through the hidden state of the last [CLS] token, generate a 768-dimensional sentence vector, and retain the semantic information;

[0052] Image modality: Use the CLIP model to extract visual features, output a 512-dimensional image feature vector, and capture visual details;

[0053] Time series data: Encode it into time series features through a bidirectional LSTM, adopt a sliding window mechanism to extract time-dependent patterns, set the window step size to 10 seconds, and output 128-dimensional time series features;

[0054] S12. Dynamic alignment of the meta-learning framework

[0055] Train independent autoencoders for the text modality, image modality, and time series data respectively, and pre-train the reconstruction loss;

[0056] Learn the initial parameters through the MAML algorithm, so that it can quickly adapt to new tasks after a small number of gradient updates;

[0057] During inference, the meta-learner dynamically adjusts the encoder parameters according to the current input, so that the modal feature vectors are semantically aligned in the angular space through cosine similarity, ensuring cross-modal semantic consistency.

[0058] Further explanation, S12 uses a gated attention mechanism for semantic alignment, constructs a cross-modal feature interaction channel, and calculates the semantic correlation degree of text-image features. The calculation formula of the gated attention mechanism is:

[0059] α t，i =σ(W t ·h t +W i ·h i +b)

[0060] In the formula, α t，i is the attention weight of the text-image modality, α t，i ∈[0, 1]. When α t，i >0.7, the cross-modal attention mechanism is activated. σ is the sigmoid function. W t is the trainable weight matrix for text, used for the linear transformation of text features. h t is the text feature vector. W i is the trainable weight matrix for images, used for the linear transformation of image features. h i is the image feature vector, and b is the bias term parameter.

[0061] S2. Build a hierarchical reinforcement learning agent, whose state space includes the topological structure of the knowledge graph entity relationships and the joint encoding vector of multi-modal features.

[0062] Among them, the construction method of the state space includes:

[0063] Encode the entity relationships into an adjacency matrix, concatenate the vectors corresponding to the text modality, image modality, and temporal data after dynamic alignment in S12, and perform joint representation.

[0064] Furthermore, the hierarchical reinforcement learning agent consists of an upper-layer agent and a lower-layer agent. Among them, the upper-layer agent is used to receive the task objective and output a sub-task sequence, and the lower-layer agent is used to execute specific actions for each sub-task.

[0065] S3. To separate path selection and modality fusion weight adjustment, and improve the flexibility and interpretability of decision-making, define a dual action space:

[0066] A. The main action space is used to control the path selection of knowledge graph relationship reasoning, including:

[0067] Action set: contains all possible relationships in the knowledge graph;

[0068] Policy network: uses a DQN network, inputs the state vector, and outputs the relationship selection probability;

[0069] B. The auxiliary action space is used to adjust the multi-modal data fusion weight, including:

[0070] Action set: defines the modality fusion weight vector;

[0071] Policy network: adopts a Softmax output layer, and encourages exploration diversity through entropy regularization.

[0072] Furthermore, the way the auxiliary action space adjusts the multi-modal data fusion weight includes:

[0073]

[0074] In the formula, w m is the fusion weight of the m-th modality, exp is the exponential function, used for Softmax normalization, f m is the importance scoring function of the m-th modality feature, s is the current state vector, M is the total number of modalities, k is the modality traversal index, k ∈ [1, M], f k is the importance scoring function of the k-th modality.

[0075] S4. Build a multi-objective reward function to drive the hierarchical reinforcement learning agent to generate an intelligent reasoning path. Among them, the calculation formula of the multi-objective reward function is:

[0076] R total = R c + R u + R s

[0077] Wherein, R total is the total reward score, R c is the path complexity penalty score, R u is the user feedback reward score, R s is the semantic consistency metric score;

[0078] The calculation formula for the path complexity penalty term is:

[0079] R c = -λ·log(1 + n)

[0080] Wherein, n is the number of path nodes, λ is the penalty weight coefficient, preferably 0.3;

[0081] The calculation formula for the user feedback reward term is:

[0082] R u = β·G

[0083] Wherein, G is the manual score, G ∈ [0, 1], β is the user feedback weight coefficient, preferably 0.5;

[0084] The calculation formula for the semantic consistency metric is:

[0085] R s = γ·g

[0086] Wherein, g is the intelligent inference path score, γ is the semantic weight coefficient, preferably 0.2.

[0087] S5. Construct an adversarial training mechanism to verify the rationality of the intelligent inference path. The adversarial training mechanism includes:

[0088] Construct a discriminator network to distinguish the intelligent inference path from the expert-annotated path, and define an adversarial loss function:

[0089]

[0090] Wherein, L is the adversarial training loss value, is the discriminator network, is the parameter, p r is the expert-annotated path, p f is the intelligent inference path;

[0091] Generate a counterfactual inference-based differential explanation report in the following ways:

[0092] Calculate the contribution of each triple to the prediction through SHAP values.

[0093] Delete the triple with the highest contribution and observe the change value of the prediction probability.

[0094] If the change value of the prediction probability exceeds the threshold, preferably 0.4, then mark this triple as a decisive reasoning basis and generate a natural language explanation;

[0095] Perform incremental knowledge graph updates, including:

[0096] When a new entity in unstructured data is detected, calculate the semantic similarity between the new entity and existing entities, screen the three candidate associated entities with the highest similarity, use a graph convolutional network to predict potential relationships, generate multiple candidate insertion paths based on a reinforcement learning agent, select the path with the highest comprehensive confidence, and when its comprehensive confidence exceeds the preset reference threshold, preferably 0.8, perform knowledge graph expansion, where the calculation method of the comprehensive confidence is:

[0097]

[0098] In the formula, the graph structure score is the rationality score of predicting potential relationships through a graph convolutional network.

[0099] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A decision optimization method that integrates enhanced multi-modal learning and knowledge graphs, characterized in that, Specifically, it includes the following steps: S1. Construct a multi-modal embedding space including text, images, and temporal data, and dynamically align the feature vectors of each modality through a meta-learning framework; S2. Build a hierarchical reinforcement learning agent, whose state space includes the topological structure of the entity relationships in the knowledge graph and the joint encoding vector of multi-modal features; S3. Define a dual action space: the main action space is used to control the selection of the relationship reasoning path in the knowledge graph, and the auxiliary action space is used to adjust the multi-modal data fusion weight; S4. Construct a multi-objective reward function to drive the hierarchical reinforcement learning agent to generate an intelligent reasoning path; S5. Construct an adversarial training mechanism to verify the rationality of the intelligent reasoning path, generate a differential explanation report through counterfactual reasoning, and perform incremental knowledge graph updates.

2. The decision optimization method integrating and enhancing multi-modal learning and knowledge graph according to claim 1, characterized in that In S1, a gated attention mechanism is used to construct a cross-modal feature interaction channel to calculate the semantic correlation degree of text-image features. The calculation formula of the gated attention mechanism is: α t，i = σ(W t ·h t + W i ·h i + b) where α t，i is the attention weight of the text-image modality, α t，i ∈ [0, 1]. When α t，i > 0.7, the cross-modal attention mechanism is activated. σ is the sigmoid function. W t is the text trainable weight matrix for the linear transformation of text features. h t is the text feature vector. W i is the image trainable weight matrix for the linear transformation of image features. h i is the image feature vector, and b is the bias term parameter.

3. A decision optimization method integrating enhanced multi-modal learning and knowledge graph according to claim 1, characterized in that, In S3, the ways in which the auxiliary action space is used to adjust the multi-modal data fusion weight include: where w m is the fusion weight of the m-th modality, exp is the exponential function for Softmax normalization, f m is the importance scoring function of the m-th modality feature, s is the current state vector, M is the total number of modalities, k is the modality traversal index, k ∈ [1, M], f k is the importance scoring function of the k-th modality.

4. A decision optimization method integrating and strengthening multi-modal learning and knowledge graph according to claim 1, characterized in that The calculation formula of the multi-objective reward function in S4 is: R total = R c + R u + R s Wherein, R total is the total reward score, R c is the path complexity penalty score, R u is the user feedback reward score, R s is the semantic consistency metric score; The calculation formula of the path complexity penalty term is: R c = -λ·log(1 + n) In the formula, n is the number of path nodes, and λ is the penalty weight coefficient; The calculation formula of the user feedback reward term is: R u = β · G In the formula, G is the manual score, G ∈ [0, 1], and β is the user feedback weight coefficient; The calculation formula of the semantic consistency metric is: R s = γ·g In the formula, g is the intelligent reasoning path score, and γ is the semantic weight coefficient.

5. A decision optimization method integrating enhanced multi-modal learning and knowledge graph according to claim 1, characterized in that The adversarial training mechanism in S5 includes: Construct a discriminator network to distinguish the intelligent reasoning path from the expert annotation path, and define an adversarial loss function: Where L is the adversarial training loss value, is the discriminator network, is a parameter, p r is the expert annotation path, p f is the intelligent inference path.

6. The decision optimization method integrating and enhancing multi-modal learning and knowledge graph according to claim 1, characterized in that In S5, the ways in which a differential explanation report is generated through counterfactual reasoning include: Calculate the contribution degree of each triple to the prediction through the SHAP value. Delete the triple with the highest contribution degree and observe the change value of the prediction probability. If the change value of the prediction probability exceeds the threshold, mark the triple as a decisive reasoning basis and generate a natural language explanation.

7. A decision optimization method that integrates and enhances multi-modal learning and knowledge graphs according to claim 1, characterized in that, In S5, the ways in which incremental knowledge graph updates are performed include: When a new entity in unstructured data is detected, calculate the semantic similarity between the new entity and the existing entities, screen the three candidate associated entities with the highest similarity, use a graph convolutional network to predict potential relationships, generate multiple candidate insertion paths based on the reinforcement learning agent, select the path with the highest comprehensive confidence, and perform knowledge graph expansion when its comprehensive confidence exceeds the preset reference threshold.

8. A decision optimization method integrating enhanced multi-modal learning and knowledge graph according to claim 7, characterized in that, The calculation method of the comprehensive confidence is: In the formula, the graph structure score is the rationality score of predicting potential relationships through the graph convolutional network.

Images