Multi-modal knowledge graph rule reasoning method and device based on large model

Abstract

The invention discloses a multi-modal knowledge graph rule reasoning method and device based on a large model, and the method comprises the steps: carrying out the feature extraction and cross-modal alignment of input text data and image data, and generating a multi-modal feature vector of a unified semantic space; performing knowledge graph storage on the emotion entities and the relationships by adopting an attribute graph model to complete construction of an emotion knowledge graph; generating an interpretable inference rule from the emotion knowledge graph by using a large language model, and eliminating a conflict rule in combination with logic verification; calculating the confidence coefficient of a reasoning path based on an attention mechanism, and carrying out quantitative evaluation on a rule reasoning result; the knowledge graph and the rule base are updated online according to user feedback, and the real-time performance and accuracy of the inference system are optimized through weight adjustment and a forgetting mechanism. According to the method, through innovative technologies such as multi-modal data integration, dynamic knowledge evolution and interpretability reasoning, the limitation of a traditional sentiment analysis method in the aspects of evidence dimension, adaptive capacity, interpretability and the like is broken through.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge reasoning, and particularly relates to a multi-modal knowledge graph rule reasoning method and device based on a large model. Background Art

[0002] In today's era of information explosion, sentiment analysis and reasoning technology has become a research hotspot in the field of artificial intelligence, and its application scenarios cover multiple fields such as public opinion monitoring, intelligent customer service, and user behavior analysis. With the explosion of social media and multi-modal data, traditional sentiment analysis methods have gradually revealed significant limitations. Early models based on statistical learning rely on sentiment lexicons and manual feature engineering, which can achieve basic sentiment classification, but are weak in complex semantic understanding (such as irony, metaphor) and cross-domain migration. Although deep learning models (such as BERT, GPT) have improved the accuracy through context semantic modeling, their end-to-end black-box characteristics make the reasoning process unexplainable and it is difficult to trace the root cause of sentiment generation.

[0003] Existing technical solutions mainly face three challenges: First, rule-based methods rely on manually defined logic, which is costly to maintain and has insufficient coverage in the dynamically changing Internet context; second, data-driven machine learning models rely too much on labeled data and have limited generalization ability in small-sample scenarios (such as emerging fields); third, traditional single-modal analysis methods cannot fully utilize the sentiment clues contained in unstructured data such as images and videos. Although some studies have tried to construct sentiment knowledge graphs to achieve causal reasoning, existing graphs are mostly extracted based on static rules or remote supervision, and there are problems such as high relationship noise and insufficient multi-modal fusion.

[0004] In recent years, large language models (LLMs) have demonstrated powerful few-shot learning capabilities, but the factual biases and rule generalization defects in their generated content still restrict their application in sentiment reasoning. At the same time, the structural advantages of knowledge graphs have not been deeply coordinated with deep learning, resulting in models being difficult to have both interpretability and domain adaptability. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a multi-modal knowledge graph rule reasoning method and device based on a large model. Through multi-modal data preprocessing, dynamic rule generation, and logical verification mechanisms, it breaks through the bottlenecks of traditional technologies in interpretability, domain migration ability, and multi-modal integration, and provides traceable and verifiable sentiment analysis results for intelligent systems.

[0006] To achieve the above objective, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a multi-modal knowledge graph rule reasoning method based on a large model, including the following steps:

[0008] Extract features from the input text data and image data and perform cross-modal alignment to generate multi-modal feature vectors in a unified semantic space;

[0009] During the emotional entity recognition process, comprehensively utilize the aligned text features and image features, assist in disambiguation and completion with the help of the context information in the image, and recognize the emotional entities in the text; further verify and enhance the emotional entities through the knowledge-enhanced pre-trained model ERNIE and the domain dictionary, predict the relationships between the recognized emotional entities to obtain the relationships between entities, and use the attribute graph model to store the knowledge graph of the emotional entities and the relationships between entities to complete the construction of the emotional knowledge graph;

[0010] Use a large language model to generate interpretable inference rules from the emotional knowledge graph, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification;

[0011] Match the emotional entities recognized in the input text with the nodes in the emotional knowledge graph, trigger the matching rules to generate an emotional reason explanation chain, calculate the confidence of the inference path based on the attention mechanism, and quantitatively evaluate the rule inference results;

[0012] Online update the knowledge graph and rule base according to user feedback, and optimize the real-time performance and accuracy of the inference system through weight adjustment and forgetting mechanism.

[0013] As a preferred technical solution, the extraction of features from the input text data and image data and cross-modal alignment to generate multi-modal feature vectors in a unified semantic space is specifically as follows:

[0014] Use the pre-trained language model BERT to extract features from the input text data to obtain text features;

[0015] Use an object detection model to perform object recognition processing on the input image data, generate the bounding box coordinates corresponding to each target object, and input the target object bounding box coordinates into the image description generation model BLIP to extract image features;

[0016] Use the CLIP model to calculate the semantic similarity between the text feature vector and the image feature vector, and optimize the feature alignment through the contrastive learning loss to obtain cross-modal features in a unified semantic space.

[0017] As a preferred technical solution, the construction of the emotional knowledge graph is specifically as follows:

[0018] Use the ERNIE model combined with a domain dictionary to identify sentiment entities; the domain dictionary contains professional terms and common entity names in a specific domain, and can capture specific domain terms that the ERNIE model may miss;

[0019] Concatenate the feature vectors of sentiment entities, and use a preset neural network model to predict the relationships between entities. When training the preset neural network model, optimize the parameters of the neural network model by minimizing the gap between the predicted relationship probability and the actual relationship label to accurately identify the semantic relationships between entities;

[0020] Use an attribute graph model to structurally store sentiment entities and relationships. In the attribute graph model, the node attributes include entity unique identifiers, entity types, sentiment polarities, and entity feature vectors; the edge attributes include the starting node, ending node, relationship type, relationship confidence, and corresponding rule identifiers of the relationship.

[0021] As a preferred technical solution, generate interpretable inference rules from the sentiment knowledge graph using a large language model, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification. Specifically:

[0022] Design a hierarchical prompt template to guide the GPT-4 model to generate rules. The prompt template adopts a hierarchical strategy: first, incorporate domain knowledge to guide the rule generation process to be more in line with the domain reality; second, emphasize multimodal constraints, requiring that the generated rules must comprehensively consider text and image information; finally, adopt interpretability constraints, requiring that the evidence sources be marked in the rules to make the generated rules have clear bases and interpretability;

[0023] Use a hybrid loss function to optimize the generated rules, including cross-entropy loss and KL divergence loss. The cross-entropy loss is used to measure the difference between the rule prediction result and the true label, and to prompt the rules to classify more accurately; the KL divergence loss is used to control the stability during the rule update process and avoid performance degradation caused by drastic changes in the rules;

[0024] Establish a domain common sense library for rule conflict logic verification, and trigger manual review when the rule confidence is lower than the threshold.

[0025] As a preferred technical solution, for the detected rule conflicts, adopt multiple strategies for resolution, including:

[0026] Priority sorting strategy: Sort the rules according to importance. When conflicts occur, give priority to retaining the rules defined by experts, followed by high-frequency rules, and low-frequency rules with the lowest priority;

[0027] Rule merging strategy: Merge semantically similar rules to improve the accuracy and simplicity of the rules;

[0028] For conflict rules that cannot be resolved in the short term, they will be marked and key optimized in subsequent iterations.

[0029] As a preferred technical solution, the emotional entities identified in the input text are matched with the nodes in the emotional knowledge graph, triggering the matching rules to generate an emotional reason explanation chain, calculating the confidence of the reasoning path based on the attention mechanism, and quantitatively evaluating the rule reasoning results. Specifically:

[0030] Based on the matching strategy of cosine similarity, by calculating the similarity between the semantic feature vectors of the input text entities and the embedding vectors of the knowledge graph nodes, an efficient association between the emotional entities and the knowledge graph nodes is achieved;

[0031] Rule matching is performed by determining whether the input feature vectors meet the rule triggering conditions. The specific rule matching is as follows: a parallel matching algorithm based on the attention mechanism decomposes complex rule conditions into multiple independent sub-conditions for distributed parallel computing, thereby improving the matching efficiency;

[0032] Based on the confidence calculation module, the rule reasoning results are quantitatively evaluated. The confidence calculation module comprehensively considers three dimensions: the strength of the rule itself, the matching accuracy of entity linking, and the credibility of multi-modal evidence, and effectively integrates evidence from different sources through a hierarchical weighted fusion strategy.

[0033] As a preferred technical solution, the knowledge graph and rule base are updated online according to user feedback, and the real-time performance and accuracy of the reasoning system are optimized through weight adjustment and forgetting mechanism. Specifically:

[0034] Receive the corrected annotation of the reasoning result from the user, and update the entity relationship and rule confidence in the knowledge graph;

[0035] Dynamically adjust the rule priority through reinforcement learning, and attenuate or eliminate outdated rules.

[0036] In a second aspect, the present invention provides a multi-modal knowledge graph rule reasoning system based on a large model, which is applied to the multi-modal knowledge graph rule reasoning method based on a large model, and includes a multi-modal data preprocessing module, an emotional knowledge graph construction module, a large model-driven rule generation module, a dynamic reasoning engine module, and an incremental learning optimization module;

[0037] The multi-modal data preprocessing module is used to extract features and perform cross-modal alignment on the input text data and image data to generate multi-modal feature vectors in a unified semantic space;

[0038] The emotional knowledge graph construction module is used to comprehensively utilize the aligned text features and image features during the emotional entity recognition process, assist in disambiguation and completion with the help of the situational information in the image, and identify the emotional entities in the text; further verify and enhance the emotional entities through the knowledge-enhanced pre-trained model ERNIE and the domain dictionary, predict the relationships between the identified emotional entities to obtain the relationships between entities, and use the attribute graph model to store the knowledge graph of the emotional entities and the relationships between entities, thus completing the construction of the emotional knowledge graph;

[0039] The large model-driven rule generation module is used to generate interpretable inference rules from the emotional knowledge graph using the large language model, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification;

[0040] The dynamic inference engine module is used to match the emotional entities identified in the input text with the nodes in the emotional knowledge graph, trigger the matching rules to generate an emotional reason explanation chain, calculate the confidence of the inference path based on the attention mechanism, and quantitatively evaluate the rule inference results;

[0041] The incremental learning optimization module is used to online update the knowledge graph and the rule base according to user feedback, and optimize the real-time performance and accuracy of the inference system through weight adjustment and forgetting mechanism.

[0042] In a third aspect, the present invention provides an electronic device, which includes:

[0043] At least one processor; and,

[0044] A memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the multi-modal knowledge graph rule inference method based on the large model.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the multi-modal knowledge graph rule inference method based on the large model.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. Multi-modal evidence fusion ability. The prior art mainly relies on single-modal text analysis and is difficult to process unstructured data such as images and videos. The present invention realizes the joint inference of multi-modal evidence by mapping the text semantic vector and the image visual feature to a unified space through cross-modal alignment technology.

[0049] 2. Dynamic knowledge evolution mechanism. Traditional methods rely on static rule bases and are difficult to adapt to emerging fields and semantic evolution. The present invention realizes the dual-wheel drive update of the knowledge graph and the rule base through an incremental learning framework. The online feedback module can automatically capture entity relationships, and the rule weight adjustment module dynamically optimizes rule parameters through reinforcement learning.

[0050] 3. Explainable reasoning ability. Existing models are mostly "black box" decision-making and are difficult to provide causal explanations. The present invention generates explanatory results containing evidence chains through the structured storage of the knowledge graph and the logical reasoning of the rule engine.

[0051] 4. Improvement of rule generation efficiency. Traditional rule generation relies on manual annotation, with high costs and low coverage. The present invention uses LLM to automatically generate rules covering multi-modal conditions.

[0052] In summary, through innovative technologies such as multi-modal data integration, dynamic knowledge evolution, and explainable reasoning, the present invention breaks through the limitations of traditional sentiment analysis methods in terms of evidence dimension, adaptability, interpretability, etc. Brief Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a flowchart of the multi-modal knowledge graph rule reasoning method based on a large model according to an embodiment of the present invention;

[0055] Figure 2 It is a block diagram of the multi-modal knowledge graph rule reasoning system based on a large model according to an embodiment of the present invention.

[0056] Figure 3 It is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Description of the Embodiments

[0057] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present application.

[0058] References to "embodiments" in this application mean that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0059] As Figure 1 shown, the present method provides a multi-modal knowledge graph rule reasoning method based on a large model, including the following steps:

[0060] S1. Preprocessing of multi-modal data, specifically including:

[0061] S1.1. Text feature extraction;

[0062] Text feature extraction, as the primary step in the sentiment cause regularization reasoning process, aims to convert the input natural language text into a numerical feature vector that can be processed by a computer. This method uses the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) based on the Transformer architecture to implement text feature extraction. The BERT model obtains rich language knowledge and semantic information through pre-training on a large-scale unsupervised text corpus; its bidirectional attention mechanism can make full use of context information, thereby providing an accurate semantic representation for each word in the input text.

[0063] In a specific example, for a given input text s, first use the tokenizer function to tokenize it to obtain a sequence of words or sub-words that are easier for the model to understand. Taking the text "The quick brown fox jumps over the lazy dog." as an example, the tokenized sequence is ["The", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]. Subsequently, the tokenized sequence is input into the BERT model for encoding processing to generate a feature vector h text , that is:

[0064] h text = BERT(Tokenizer(s)),

[0065] The feature vector h textThe output dimension is d×T, where d is the feature dimension, and the specific value is determined by the configuration of the selected BERT model; T is the length of the input text sequence after tokenization, that is, the number of words or subwords obtained by tokenization.

[0066] S1.2. Image feature extraction;

[0067] In multimodal data processing, as an important source of information, image data needs to be converted into visual semantic feature vectors that can be fused with text features in the subsequent process through image feature extraction. This method combines object detection and image description to achieve the above-mentioned purpose of image feature extraction, specifically as follows:

[0068] First, use the object detection model YOLOv5 to perform object recognition on the input image I. YOLOv5 (You Only Look Once version 5) is a real-time object detection model based on the convolutional neural network structure, with high detection accuracy and processing efficiency. Through YOLOv5, multiple target objects in the input image can be quickly and accurately recognized, and the corresponding bounding box coordinates of each target object can be generated.

[0069] Subsequently, use the obtained target object bounding box coordinates as input and send them into the image description generation model BLIP to extract richer visual semantic features. The BLIP (Bootstrapping Language-Image Pre-training) model is a pre-training model that can generate natural language descriptions based on image content. It combines image visual features and language representation knowledge. After focusing on the image information in a specific area, it generates accurate and detailed visual semantic description information. In the specific implementation process, the BLIP model focuses on a specific area of the image according to the aforementioned target bounding box coordinates, so as to generate a visual semantic vector h that describes the content of this area. image .

[0070] The image features extracted by the above method not only contain the basic visual feature information of the target objects in the image, but also can effectively capture the mutual relationship between objects and the overall semantic information of the scene. For example, for a scene image depicting people dining in a restaurant, this feature extraction process can accurately identify specific objects such as people, tables, and food in the image, and generate visual semantic vectors covering the relationship between these objects and the scene. The effective extraction of this comprehensive visual semantic information is of great significance for the task of regularizing reasoning of emotional reasons, mainly because the semantic association between the scene and objects in the image is usually closely related to emotional expression.

[0071] S1.3. Cross-modal alignment;

[0072] After separately completing text feature extraction and image feature extraction, this method obtains the feature representations corresponding to the two-modal data, namely the text semantic vector h text and the image semantic vector h image . However, since text and image belong to different data modalities, there are inherent differences in their feature representations in the semantic space. Therefore, it is necessary to perform cross-modal alignment on these two features to achieve a unified semantic representation.

[0073] Specifically, this method uses the CLIP model to achieve cross-modal alignment of text and image features. CLIP (Contrastive Language-Image Pretraining), a deep neural network model that learns cross-modal semantic association relationships through a contrastive learning mechanism using large-scale image-text data pairs. With the help of the CLIP model, this method can calculate the semantic similarity between the text feature vector and the image feature vector, as shown in the formula:

[0074]

[0075] In the formula, h text and h image represent the text feature and the image feature vector respectively. The cosine similarity metric method used reflects the degree of proximity of the two vectors in the semantic space. The value range of cosine similarity is [-1, 1]. When the similarity value approaches 1, it indicates that the two vectors are very close in the semantic space; on the contrary, when the value approaches -1, it means that the semantic differences between the two vectors are large.

[0076] To further improve the alignment effect of cross-modal features, this method uses a contrastive learning strategy to optimize the feature representation. The core idea of contrastive learning is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs to effectively eliminate the semantic gap between different modalities. In specific implementation, the following contrastive loss function is used for feature optimization:

[0077]

[0078] In the formula, h i and h j represent the positive sample pair feature vectors, h k is the negative sample pair, and τ is the temperature parameter used to adjust the distribution scale of the similarity score. By minimizing this loss function, it can promote the mutual proximity of the text and image feature vectors in the semantic space and achieve more effective cross-modal feature alignment.

[0079] It can be understood that through the above-mentioned cross-modal feature alignment step, the data of both text and image modalities can be effectively mapped into a unified semantic space, thereby generating a unified feature expression. This unified feature expression greatly improves the convenience and accuracy of multimodal data analysis, and provides a richer and more accurate information foundation for subsequent regular reasoning of emotional causes. For example, in a specific sentiment analysis scenario, through this feature alignment process, the emotions expressed in the text can be effectively associated with the visual scene in the image, thereby more accurately inferring the potential causes of the emotions.

[0080] In summary, through preprocessing steps such as text feature extraction, image feature extraction, and cross-modal feature alignment, this method effectively maps multimodal data into a consistent feature representation in a shared semantic space, laying a solid foundation for subsequent regularized reasoning of sentiment. The rational design and effective implementation of this preprocessing approach can significantly improve the performance and accuracy of the overall reasoning system.

[0081] S2, emotional knowledge graph construction;

[0082] In the process of sentiment entity recognition, the aligned text features and image features are comprehensively utilized, and the contextual information in the image is used to assist in disambiguation and completion to identify sentiment entities in the text; the sentiment entities are further verified and enhanced through the knowledge-enhanced pre-training model ERNIE and the domain dictionary, and the relationship prediction of the identified sentiment entities is performed to obtain the relationship between entities. The attribute graph model is used to store the relationship between sentiment entities in the knowledge graph to complete the construction of the sentiment knowledge graph.

[0083] This method uses the knowledge-enhanced pre-trained language model ERNIE to pre-process the text feature vector h text Processing is performed to achieve the recognition of emotional entities. The ERNIE (Enhanced Representation through Knowledge Integration) model is pre-trained on large-scale unsupervised text data, integrating entity and knowledge graph information, and possesses powerful semantic understanding and knowledge reasoning capabilities. Compared with traditional pre-trained models, the ERNIE model demonstrates higher accuracy in semantic recognition of domain-specific text, effectively identifying emotion-related entities such as product names, service types, and people.

[0084] To further improve the accuracy of entity recognition, this method introduces the joint use of a domain dictionary and the ERNIE model. The domain dictionary contains professional terms and common entity names in a specific domain. By comparing its results with those of the ERNIE model, the accuracy of the recognized entities can be effectively confirmed. In addition, the domain dictionary can also capture specific domain terms that the ERNIE model may miss, further improving the recall rate and accuracy of the entity recognition task.

[0085] After entity extraction is completed, it is necessary to further determine the semantic relationships between entities. This method predicts the relationships between entities through a neural network model, and then identifies causal, associative, and other relationships existing between entities. Specifically, this method concatenates the entity feature vectors and predicts the relationships between entities through a neural network model. This model can effectively learn the complex semantic mapping relationships between entities, so as to accurately judge the relationship type and strength between entities. When training this model, it is necessary to rely on the labeled data of entity pairs and relationship types. By minimizing the gap between the predicted relationship probability and the actual relationship label, the parameters of the neural network model are optimized so that it can accurately identify the semantic relationships between entities. In addition, to further improve the performance of relationship extraction, this method can introduce the context information where the entity is located to assist the neural network model in more accurately judging the relationships between entities. At the same time, a multi-task learning strategy can also be adopted to jointly train the relationship extraction and entity classification tasks to further improve the generalization ability of the model.

[0086] After the above entity extraction and relationship extraction steps, this method obtains a series of entities with rich semantic information and the relationships between entities. To facilitate the effective management and utilization of this information, this method uses an attribute graph model to structurally store entities and relationships. The attribute graph model is a flexible and powerful knowledge representation structure, where entities are represented as nodes and the relationships between entities are represented as edges. Each node and edge has rich attribute information. In this method, the node attributes include the entity unique identifier (id), entity type (type, such as product, user, service, etc.), sentiment polarity (polarity, such as positive, negative, neutral), and entity feature vector (embedding), etc.; the attributes of the edge include the starting node (source) of the relationship, the ending node (target), the relationship type (relation, such as causal, associative, etc.), the relationship confidence (confidence), and the corresponding rule identifier (rule_id), etc.

[0087] Storing knowledge graphs using the property graph model has obvious advantages: First, this model can intuitively and efficiently display the complex semantic relationships between entities, facilitating structured knowledge management; Second, the property graph model supports fast query and traversal operations, effectively meeting the needs of knowledge graph analysis and mining. For example, through conditional queries, entities and relationships that meet specific conditions can be quickly retrieved. In addition, the property graph model supports dynamic updates, enabling timely addition of newly discovered entities and relationships, ensuring the timeliness and accuracy of the knowledge graph.

[0088] Through the above steps of entity extraction, relationship extraction, and graph storage, this method finally constructs a structured sentiment knowledge graph. This graph can provide rich and comprehensive semantic information and data support for the reasoning and analysis of sentiment reasons.

[0089] S3. Rule generation driven by large models;

[0090] After completing the construction of the sentiment knowledge graph, this method has obtained a structured knowledge network containing rich entities and entity relationships. However, in order to effectively apply this knowledge to the reasoning task of sentiment reasons, it is necessary to further transform the knowledge graph into executable rules.

[0091] S3.1. Rule generation;

[0092] Rule generation is the initial step in this stage. This method leverages the powerful language understanding and generation capabilities of the LLM large language model to extract structured rules from the sentiment knowledge graph. The core idea of the rules is to combine the multi-modal features of text and images, and perform logical judgments through feature functions, thereby realizing the reasoning of sentiment reasons.

[0093] To guide the GPT-4 model to generate high-quality rules, this method designs a dedicated prompt template. This prompt template adopts a hierarchical strategy: First, incorporate domain knowledge to guide the rule generation process to be more in line with the domain reality; Second, emphasize multi-modal constraints, requiring that the generated rules must comprehensively consider text and image information. For example, when evaluating a product, both the text description and information such as the product appearance in the image need to be considered; Finally, adopt interpretability constraints, requiring that the evidence sources be marked in the rules, making the generated rules have clear bases and interpretability.

[0094] S3.2. Rule optimization;

[0095] The initially generated rules may have problems such as insufficient accuracy or poor generalization ability, so further optimization is needed. This method adopts a contrastive learning strategy, and the loss function is:

[0096]

[0097] Among them, the cross-entropy loss It is used to measure the difference between the rule prediction result and the true label, prompting the rule to classify more accurately. KL divergence It is used to control the stability during the rule update process, avoiding sharp changes in the rules that may lead to performance degradation. During the contrastive learning process, positive and negative samples are constructed to optimize and fine-tune the rule parameters. Positive samples come from instances where the rule is correctly triggered in the labeled data, while negative samples are constructed by adversarial generation or other means to create situations where the rule is wrongly triggered. By learning these positive and negative samples, the model can effectively distinguish between correct and incorrect rule application scenarios, improving the accuracy and generalization ability of the rules.

[0098] Furthermore, this method also expresses the rules in vector form and measures the differences between rules based on cosine similarity to finely adjust and optimize the rules. In practical applications, to cope with the continuous changes in data, this method adopts an incremental learning strategy to continuously update the rule parameters. The rule model is updated in a timely manner through online learning to maintain the real-time performance and adaptability of the rules; knowledge distillation is used to transfer the knowledge of the large LLM model to the lightweight rule engine to improve the efficiency of rule application. At the same time, this method sets a forgetting mechanism to decay the weights of obsolete rules, effectively avoiding the negative impact of expired rules on the inference process.

[0099] S3.3. Rule conflict detection;

[0100] After the rule generation and optimization phase, to ensure the quality of the rule base, rule conflict detection is also required. Among the large number of generated rules, there may be cases that conflict with domain common sense or business rules, such as rules like "high price → positive evaluation" that violate common sense. This method establishes a domain common sense library, which contains general knowledge rules widely recognized in the domain, as the basis for identifying conflicting rules. Through logical verification methods, the generated rules are compared with the domain common sense library. When a rule conflicts with the knowledge in the common sense library and the confidence of the rule is lower than the set threshold, the manual review process is initiated, and domain experts are asked to confirm and correct the conflicting rules.

[0101] To resolve the detected rule conflicts, this method designs multiple strategies for resolution. For example, the priority sorting strategy sorts the rules according to their importance. When conflicts occur, the rules defined by experts are given priority, followed by high-frequency rules, and low-frequency rules have the lowest priority; the rule merging strategy merges semantically similar rules to improve the accuracy and simplicity of the rules. For conflict rules that cannot be resolved in the short term, this method will mark them and focus on optimizing them in subsequent iterations.

[0102] In summary, through rule generation, rule optimization, and rule conflict detection driven by large models, this method has successfully constructed a rule base with high accuracy and low conflicts. This rule base fully combines the advantages of the sentiment knowledge graph and multimodal information, can provide comprehensive and powerful support for the rule-based reasoning of sentiment reasons, and achieves the goal of more efficient, accurate, and interpretable sentiment analysis and reasoning tasks.

[0103] Furthermore, after rule generation and optimization, rule conflict detection is also required. Among the numerous generated rules, there may be some conflicting rules that conflict with domain common sense. For example, "price_high → positive" is inconsistent with our common sense that a high price usually leads to negative evaluations. To eliminate these conflicting rules, this embodiment adopts a logical verification method. First, a domain common sense library is established. This common sense library contains general rules and knowledge in the domain and is an important basis for us to judge whether a rule is reasonable. Then, the generated rules are compared with the common sense library to check for contradictions. When the confidence level of a rule is lower than 0.6, this invention will trigger manual review, and domain experts will make further judgments and corrections on the rules. For the detected conflicting rules, this invention adopts various resolution strategies. Priority ranking is a commonly used method. We stipulate that the rules formulated by domain experts have the highest priority, the frequently generated rules are second, and the low-frequency rules have the lowest priority. When conflicts occur, the rules with higher priority are preferentially retained. Rule merging is also an effective strategy. For some similar rules, such as "poor service → negative" and "bad attitude → negative", we can merge them into "poor service attitude → negative", which can reduce the number of rules and improve the simplicity and accuracy of the rules. For conflicting rules that cannot be resolved, we mark them for subsequent iterative optimization. Through rule conflict detection, we have reduced the rule conflict rate from the initial 18.7% to 3.2%, and successfully intercepted a large number of rules violating industry norms in multiple fields such as healthcare and e-commerce, ensuring the high quality and conflict-free nature of the rule base. Through steps such as large model-driven rule generation, optimization, and conflict detection, this invention finally generates a rule base with high accuracy and no conflicts. This rule base can make full use of the information in the sentiment knowledge graph, combine multimodal data, and provide accurate and reliable rule support for the reasoning of sentiment reasons, thereby achieving more efficient and interpretable sentiment analysis and reasoning.

[0104] S4, Dynamic Inference Engine;

[0105] After completing large model-driven rule generation and conflict detection, this method further designs a dynamic inference engine to achieve the efficient application and dynamic evolution of rules. The dynamic inference engine takes the sentiment knowledge graph as the core and integrates an incremental learning mechanism to form a complete closed-loop inference system. It specifically includes three steps: entity linking, rule matching, and confidence calculation.

[0106] S4.1. Entity Linking;

[0107] Entity linking is an important foundation of the dynamic inference engine. Its role is to match the sentiment entities identified in the input text with the nodes in the sentiment knowledge graph, thereby triggering corresponding rules and generating an explanation chain for the sentiment reason. This method adopts a matching strategy based on cosine similarity and calculates the similarity between the semantic feature vectors of the input text entities and the embedding vectors of the graph nodes to achieve an efficient association between entities and knowledge graph nodes. The formula is:

[0108]

[0109] where, is the feature vector of the text entity after BERT encoding, is the embedding vector of the graph node.

[0110] In addition, this method introduces a multi-modal feature fusion mechanism to jointly perform entity matching on the keywords in the text and the regional features in the image. For example, when analyzing the user comment "The phone gets hot during charging", this method not only focuses on the "phone" entity in the text but also further associates the image feature node of "too high charging temperature" in the graph to improve the accuracy of entity linking by integrating multi-modal evidence. To adapt to the feature differences in different application fields, this method adopts a dynamic threshold mechanism to automatically adjust the threshold of entity matching according to the field features.

[0111] S4.2. Rule Matching;

[0112] After entity linking is completed, the dynamic inference engine enters the rule matching stage. Rule matching is performed by judging whether the input feature vector meets the rule triggering conditions. The formula is as follows:

[0113]

[0114] In the specific implementation of rule matching, this method adopts a parallel matching algorithm based on the attention mechanism, decomposes complex rule conditions into multiple independent sub-conditions, and performs efficient distributed parallel computing, thereby significantly improving the matching efficiency.

[0115] S4.3. Confidence Calculation;

[0116] After rule matching is completed, to further ensure the reliability of the inference results, this method designs a confidence calculation module to quantitatively evaluate the rule inference results. Specifically, the confidence calculation module comprehensively considers three dimensions: the strength of the rule itself, the matching accuracy of entity linking, and the credibility of multi-modal evidence, and effectively integrates evidence from different sources through a hierarchical weighted fusion strategy. The formula is as follows:

[0117]

[0118] Among them, rule c onf(R i ) is the confidence of the rule itself, entity m atch(e i ) reflects the accuracy of entity linking, modal e vidence(m i ) includes the credibility of multi-modal evidences such as text keywords and image region detection.

[0119] In summary, through steps such as entity linking, rule matching, and confidence calculation of the dynamic inference engine, this method effectively realizes efficient emotion cause reasoning based on the emotion knowledge graph and multi-modal features, significantly improves the inference accuracy, efficiency, and reliability, and has good domain generalization ability and adaptability.

[0120] S5. Incremental learning optimization;

[0121] S5.1. Online feedback;

[0122] To ensure that the system continuously adapts to new scenarios, the present invention designs an incremental learning optimization mechanism. The online feedback module collects user annotation data to update the knowledge graph:

[0123]

[0124] Select high-uncertainty samples through an active learning strategy, which combines the entropy method and Bayesian optimization.

[0125] S5.1. Rule weight adjustment;

[0126] Based on the dynamic update of the knowledge graph, the rule base also needs to be optimized according to the feedback of actual applications. This method adopts a reinforcement learning mechanism to adjust the rule weights so that the inference system can more accurately adapt to the situations in different fields. The rule weight adjustment module uses reinforcement learning to optimize the rule parameters:

[0127]

[0128] Among them, θ is the rule parameter, α is the learning rate, and R t is the reward function. The system automatically increases the weight of this rule through reinforcement learning to improve the inference accuracy.

[0129] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0130] Based on the same idea as the large model-based multi-modal knowledge graph rule reasoning method in the above embodiments, the present invention also provides a large model-based multi-modal knowledge graph rule reasoning system, which can be used to execute the above large model-based multi-modal knowledge graph rule reasoning method. For the convenience of description, in the structural schematic diagram of the embodiment of the large model-based multi-modal knowledge graph rule reasoning system, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than those illustrated, or combine certain components, or arrange different components.

[0131] Please refer to Figure 2 , in another embodiment of the present application, a large model-based multi-modal knowledge graph rule reasoning system 100 is provided. The system includes a multi-modal data preprocessing module 101, an emotion knowledge graph construction module 102, a large model-driven rule generation module 103, a dynamic reasoning engine module 104, and an incremental learning optimization module 105;

[0132] The multi-modal data preprocessing module 101 is used to extract features and perform cross-modal alignment on the input text data and image data to generate multi-modal feature vectors in a unified semantic space;

[0133] The emotion knowledge graph construction module 102 is used to identify emotion entities in the text through a knowledge-enhanced pre-trained model ERNIE and a domain dictionary, predict the relationships between the identified emotion entities to obtain the relationships between entities, and use an attribute graph model to store the knowledge graph of the emotion entities and the relationships between entities to complete the construction of the emotion knowledge graph;

[0134] The large model-driven rule generation module 103 is used to generate interpretable inference rules from the emotion knowledge graph using a large language model, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification;

[0135] The dynamic reasoning engine module 104 is used to match the emotion entities identified in the input text with the nodes in the emotion knowledge graph, trigger the matching rules to generate an emotion reason explanation chain, calculate the confidence of the reasoning path based on the attention mechanism, and quantitatively evaluate the rule reasoning results;

[0136] The incremental learning optimization module 105 is used to perform online updates on the knowledge graph and the rule base according to user feedback, and optimize the real-time performance and accuracy of the reasoning system through weight adjustment and forgetting mechanisms.

[0137] It should be noted that the multi-modal knowledge graph rule reasoning system based on large models of the present invention corresponds one-to-one with the multi-modal knowledge graph rule reasoning method based on large models of the present invention. The technical features and their beneficial effects described in the embodiments of the above multi-modal knowledge graph rule reasoning method are applicable to the embodiments of the multi-modal knowledge graph rule reasoning based on large models. For specific content, reference can be made to the description in the method embodiments of the present invention, which will not be elaborated here. This is hereby declared.

[0138] In addition, in the implementation manner of the multi-modal knowledge graph rule reasoning system based on large models in the above embodiments, the logical division of each program module is only an example. In actual applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the multi-modal knowledge graph rule reasoning system based on large models is divided into different program modules to complete all or part of the functions described above.

[0139] Please refer to Figure 3 , in one embodiment, an electronic device for implementing a multi-modal knowledge graph rule reasoning method based on large models is provided. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may further include a computer program stored in the first memory 202 and executable on the first processor 201, such as a multi-modal knowledge graph rule reasoning program 203 based on large models.

[0140] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as: SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as the mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 200. Further, the first memory 202 may also include both the internal storage unit and the external storage device of the electronic device 200. The first memory 202 can be used not only to store application software installed on the electronic device 200 and various types of data, such as the code of the multi-modal knowledge graph rule reasoning program 203 based on large models, but also to temporarily store data that has been output or will be output.

[0141] In some embodiments, the first processor 201 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The first processor 201 is the control core of the electronic device. It uses various interfaces and circuits to connect all components of the entire electronic device, and by running or executing programs or modules stored in the first memory 202, and calling data stored in the first memory 202, it can execute various functions of the electronic device 200 and process data.

[0142] Figure 3 Only the electronic device with components is shown. Those skilled in the art can understand that Figure 3 the shown structure does not constitute a limitation on the electronic device 200. It may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0143] The multi-modal knowledge graph rule inference program 203 stored in the first memory 202 in the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0144] Extract features and perform cross-modal alignment on the input text data and image data to generate multi-modal feature vectors in a unified semantic space;

[0145] Identify sentiment entities in the text through the knowledge-enhanced pre-trained model ERNIE and domain dictionaries, predict the relationships between the identified sentiment entities to obtain the relationships between entities, and use the attribute graph model to store the knowledge graph of sentiment entities and the relationships between entities to complete the construction of the sentiment knowledge graph;

[0146] Generate interpretable inference rules from the sentiment knowledge graph using a large language model, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification;

[0147] Match the sentiment entities identified in the input text with the nodes in the sentiment knowledge graph, trigger the matching rules to generate an explanation chain of sentiment reasons, calculate the confidence of the inference path based on the attention mechanism, and quantitatively evaluate the rule inference results;

[0148] Online update the knowledge graph and rule base according to user feedback, and optimize the real-time performance and accuracy of the inference system through weight adjustment and forgetting mechanism.

[0149] Furthermore, if the modules / units integrated in the electronic device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM).

[0150] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application may include non-volatile and / or volatile memories. Non-volatile memories may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories may include random access memory (RAM) or an external cache. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0151] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not conflict, they should be considered as the scope described in this specification.

[0152] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A multi-modal knowledge graph rule reasoning method based on a large model, characterized in that It includes the following steps: Extract features from the input text data and image data and perform cross-modal alignment to generate multi-modal feature vectors in a unified semantic space; In the process of sentiment entity recognition, comprehensively utilize the aligned text features and image features, and use the situational information in the image to assist in disambiguation and completion to identify the sentiment entities in the text; further verify and enhance the sentiment entities through the knowledge-enhanced pre-trained model ERNIE and the domain dictionary, predict the relationships between the identified sentiment entities to obtain the relationships between entities, and use the property graph model to store the sentiment entities and the relationships between entities in a knowledge graph to complete the construction of the sentiment knowledge graph; Use a large language model to generate interpretable inference rules from the sentiment knowledge graph, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification; Match the sentiment entities identified in the input text with the nodes in the sentiment knowledge graph, trigger the matching rules to generate an explanation chain of sentiment reasons, calculate the confidence of the inference path based on the attention mechanism, and quantitatively evaluate the rule inference results; Online update the knowledge graph and rule base according to user feedback, and optimize the real-time performance and accuracy of the inference system through weight adjustment and forgetting mechanism.

2. The multimodal knowledge graph rule reasoning method based on a large model according to claim 1, wherein The extraction of features from the input text data and image data and cross-modal alignment to generate multi-modal feature vectors in a unified semantic space is specifically as follows: Use the pre-trained language model BERT to extract features from the input text data to obtain text features; Use an object detection model to perform object recognition processing on the input image data, generate the bounding box coordinates corresponding to each target object, and input the target object bounding box coordinates into the image description generation model BLIP to extract image features; Use the CLIP model to calculate the semantic similarity between the text feature vector and the image feature vector, and optimize the feature alignment through the contrastive learning loss to obtain cross-modal features in a unified semantic space.

3. The multimodal knowledge graph rule reasoning method based on a large model according to claim 1, wherein The construction of the sentiment knowledge graph is specifically as follows: Use the ERNIE model combined with the domain dictionary to identify sentiment entities; the domain dictionary contains professional terms and common entity names in a specific domain, and can capture specific domain terms that the ERNIE model may miss; Concatenate the feature vectors of the sentiment entities, and predict the relationships between entities through a preset neural network model. When training the preset neural network model, optimize the parameters of the neural network model by minimizing the gap between the predicted relationship probability and the actual relationship label to accurately identify the semantic relationships between entities; Use the property graph model to structurally store the sentiment entities and relationships. The node attributes in the property graph model include entity unique identifiers, entity types, sentiment polarities, and entity feature vectors; the edge attributes include the starting node, ending node, relationship type, relationship confidence, and corresponding rule identifiers of the relationship.

4. The multimodal knowledge graph rule reasoning method based on a large model according to claim 1, wherein The use of a large language model to generate interpretable inference rules from the sentiment knowledge graph, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification is specifically as follows: Design a hierarchical prompt template to guide the GPT-4 model to generate rules. The prompt template adopts a hierarchical strategy: First, incorporate domain knowledge to guide the rule generation process to better conform to the domain reality. Second, emphasize multimodal constraints, requiring that the generated rules must comprehensively consider text and image information. Finally, adopt interpretability constraints, requiring that the evidence sources be annotated in the rules, so that the generated rules have clear bases and interpretability; Optimize the generated rules using a hybrid loss function, including cross-entropy loss and KL divergence loss. The cross-entropy loss is used to measure the difference between the rule prediction result and the true label, prompting the rules to classify more accurately; the KL divergence loss is used to control the stability during the rule update process, avoiding sharp changes in the rules that may lead to performance degradation; Build a domain common sense library to perform logical verification of rule conflicts, and trigger manual review when the rule confidence is lower than the threshold.

5. The multimodal knowledge graph rule reasoning method based on a large model according to claim 4, wherein, For the detected rule conflicts, adopt multiple strategies for resolution, including: Priority sorting strategy, sort the rules according to importance. When conflicts occur, give priority to retaining the rules defined by experts, followed by high-frequency rules, and low-frequency rules have the lowest priority; Rule merging strategy, merge semantically similar rules to improve the accuracy and conciseness of the rules; For conflict rules that cannot be resolved in the short term, mark them and focus on optimizing them in subsequent iterations.

6. The multimodal knowledge graph rule reasoning method based on a large model according to claim 1, wherein, Match the emotional entities identified in the input text with the nodes in the emotional knowledge graph, trigger the matching rules to generate an emotional reason explanation chain, calculate the confidence of the reasoning path based on the attention mechanism, and quantitatively evaluate the rule reasoning results. Specifically: A matching strategy based on cosine similarity, calculate the similarity between the semantic feature vectors of the input text entities and the node embedding vectors of the knowledge graph, to achieve efficient association between emotional entities and knowledge graph nodes; Rule matching is performed by judging whether the input feature vector meets the rule trigger conditions. The rule matching is specifically: a parallel matching algorithm based on the attention mechanism, decompose complex rule conditions into multiple independent sub-conditions, and perform distributed parallel computing to improve the matching efficiency; Based on a confidence calculation module, quantitatively evaluate the rule reasoning results. The confidence calculation module comprehensively considers three dimensions: the strength of the rule itself, the matching accuracy of entity linking, and the credibility of multimodal evidence, and effectively integrates evidence from different sources through a hierarchical weighted fusion strategy.

7. The multimodal knowledge graph rule reasoning method based on a large model according to claim 1, wherein, Online update the knowledge graph and rule base according to user feedback, and optimize the real-time performance and accuracy of the reasoning system through weight adjustment and forgetting mechanism. Specifically: Receive the corrected annotations of the user on the reasoning results, and update the entity relationships and rule confidences in the knowledge graph; Dynamically adjust the rule priorities through reinforcement learning, and attenuate or eliminate outdated rules.

8. A multi-modal knowledge graph rule reasoning system based on a large model, characterized in that, Applied to the multimodal knowledge graph rule reasoning method based on a large model described in any one of claims 1-7, including a multimodal data preprocessing module, an emotional knowledge graph construction module, a large model-driven rule generation module, a dynamic reasoning engine module, and an incremental learning optimization module; The multimodal data preprocessing module is used to extract features and perform cross-modal alignment on the input text data and image data, generating multimodal feature vectors in a unified semantic space; The sentiment knowledge graph construction module is used to comprehensively utilize the aligned text features and image features during the sentiment entity recognition process, assist in disambiguation and completion with the help of the situational information in the image, and identify the sentiment entities in the text; further verify and enhance the sentiment entities through the knowledge-enhanced pre-trained model ERNIE and the domain dictionary, predict the relationships between the identified sentiment entities to obtain the relationships between entities, and use the property graph model to store the knowledge graph of the sentiment entities and the relationships between entities, completing the construction of the sentiment knowledge graph; The large model-driven rule generation module is used to generate interpretable inference rules from the sentiment knowledge graph using a large language model, optimize the rule parameters through contrastive learning, and eliminate conflicting rules by combining logical verification; The dynamic inference engine module is used to match the sentiment entities identified in the input text with the nodes in the sentiment knowledge graph, trigger the matching rules to generate an explanation chain of sentiment reasons, calculate the confidence of the inference path based on the attention mechanism, and quantitatively evaluate the rule inference results; The incremental learning optimization module is used to online update the knowledge graph and the rule base according to user feedback, and optimize the real-time performance and accuracy of the inference system through weight adjustment and forgetting mechanism.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the large model-based multimodal knowledge graph rule inference method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the large model-based multimodal knowledge graph rule inference method according to any one of claims 1-7.

Images