A neural-symbolic reasoning method and system for multi-modal information processing

Abstract

This invention relates to the fields of artificial intelligence and multimodal reasoning technology, and provides a neural symbolic reasoning method and system for multimodal information processing. The method includes the following steps: acquiring multimodal input data; extracting semantic attribute representations of image data and text data on a predefined attribute set to obtain multimodal semantic attribute representations; mapping the multimodal semantic attribute representations to predicates in first-order logic form to construct a symbolic multimodal attribute fact set; inputting the multimodal attribute fact set and logical rules from a pre-constructed first-order logic knowledge base into a Markov logic network for reasoning to obtain a reasoning result; integrating the reasoning result to generate an interpretable decision result and a reasoning logic chain. This invention uses shared attribute symbols across multiple modalities as a bridge to achieve information fusion, conflict resolution, and high-level consistency constraints among modalities, resulting in reasoning results with interpretability, high robustness, and strong generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal reasoning technology, and particularly relates to a neural symbolic reasoning method and system for multimodal information processing. Background Technology

[0002] In current multimodal intelligent understanding tasks such as medicine, remote sensing, and industrial inspection, image-text fusion mainly relies on continuous vector alignment methods based on deep neural networks. For example, contrastive learning models, represented by CLIP, achieve multimodal retrieval, classification, and understanding tasks by aligning the embedded representations of images and text within a vector space. However, these methods inherently depend on implicit geometric relationships in a high-dimensional continuous vector space, lacking interpretability and difficulty in utilizing explicit domain knowledge. In continuous space, the model cannot clearly provide logical chains such as "why the image matches the text" or "why a certain category is predicted," thus lacking sufficient transparency and auditability in high-risk scenarios such as medical diagnosis, remote sensing interpretation, and industrial inspection. Specifically, these methods suffer from the following problems:

[0003] (1) The alignment process is implicit and uninterpretable: CLIP-type methods achieve implicit alignment by bringing high-dimensional vectors closer / far away. It is difficult to explain which semantic attributes the model makes judgments based on, and it is difficult to provide auditable reasoning basis in security-sensitive scenarios (such as medicine, remote sensing, and industrial inspection).

[0004] (2) Difficulty in utilizing explicit logical knowledge bases: A large amount of domain knowledge exists in the form of first-order logic rules, knowledge bases, or guidelines. Traditional multimodal deep models mainly operate in continuous vector spaces, making it difficult to directly use these explicit logical rules to constrain and enhance model reasoning.

[0005] (3) Lack of symbolic consistency among multimodal models: In practical applications, semantic inconsistencies often occur between image and text modal models. For example, the text may explicitly indicate a certain attribute, but the image model may not recognize it; or the two models may make opposite judgments on the attributes of the same object. However, due to the lack of a unified symbolic representation and logical constraints, existing models are unable to detect and automatically correct these inconsistencies in a timely manner.

[0006] Furthermore, existing deep multimodal models struggle to fully leverage expert knowledge in the form of first-order logic (FOL). In many practical tasks, domain rules often exist in the form of attribute co-occurrence relationships, mutual exclusion relationships, structural patterns, and category discrimination logic, which are difficult to directly embed into existing continuous vector fusion frameworks. Due to the lack of ability to transform visual and textual outputs into semantic units that can be directly manipulated by logical systems, current technologies cannot truly combine the perceptual capabilities of neural networks with explicit logical knowledge, rendering expert knowledge ineffective in practical decision-making.

[0007] Furthermore, the visual and textual modalities are highly complementary: the visual modality provides a wealth of fine-grained structural information, but it is susceptible to blurring, occlusion, or target similarity in certain scenarios; while the textual modality often contains implicit knowledge that vision cannot acquire, such as background information, domain terminology, contextual semantics, or structural descriptions—knowledge that the visual modality cannot extract. For example, semantic attributes in many scenarios (such as details, emotional information, and technical terms in text descriptions) cannot be directly obtained through traditional visual perception models. In existing continuous vector fusion methods, these two types of complementary information are difficult to accurately fuse at the "attribute-level semantics" level, and it is even more difficult to perform cross-modal consistency constraints and conflict detection. This makes the model prone to making incorrect inferences in cases of modal conflict or incomplete modality, especially in tasks that distinguish between similar classes or fine-grained classes.

[0008] Although some neural symbolic systems attempt to incorporate logical rules into deep models—for example, Markov Logic Networks (MLNs) are a framework that combines first-order logic with probabilistic graphical models, enabling reasoning on logical rules with uncertainty—existing methods typically only operate on a single modality or still rely on the alignment and continuity features of vector spaces, and have not achieved a systematic unification with MLNs at the level of multimodal attributes. Summary of the Invention

[0009] The purpose of this invention is to provide a neural symbolic reasoning method for multimodal information processing, aiming to solve the above-mentioned technical problems.

[0010] This invention is implemented as follows: a neural symbolic reasoning method for multimodal information processing, comprising the following steps:

[0011] Acquire multimodal input data; the multimodal input data includes image data and text data;

[0012] Semantic attribute representations of image data and text data on a predefined attribute set are extracted separately to obtain multimodal semantic attribute representations;

[0013] The semantic attribute representations of multimodalities are mapped to predicates in first-order logical form, thus constructing a symbolic set of multimodal attribute facts.

[0014] The multimodal attribute fact set and the logical rules in the pre-constructed first-order logic knowledge base are input into the Markov logic network. Under the probabilistic logic framework, attribute-level belief updates and category judgments are completed to obtain the reasoning results.

[0015] The reasoning results are integrated to generate interpretable decision results and reasoning logic links.

[0016] Furthermore, the steps of extracting semantic attribute representations of image data and text data on a predefined attribute set to obtain multimodal semantic attribute representations specifically include:

[0017] High-level semantic features of image data are extracted by a visual semantic encoder. Then, based on a visual attribute discriminant subnetwork, these high-level semantic features are mapped to the semantic dimension of a predefined set of attributes. The activation values ​​of each attribute are output, and the activation values ​​of each attribute are transformed into interpretable attribute beliefs through thresholding or discretization mechanisms to obtain the semantic attribute representation of the image modality.

[0018] The semantic content shared with the visual attribute space in the text data is parsed by a text semantic encoder. Then, the text attribute extraction subnetwork generates text-side attribute prediction results based on the semantic content. The text-side attribute prediction results are transformed into attribute-level semantic belief representations through thresholding or discretization mechanisms, thus obtaining the semantic attribute representation of the text modality.

[0019] Furthermore, the steps of mapping multimodal semantic attribute representations to predicates of first-order logic form and constructing a symbolic multimodal attribute fact set specifically include:

[0020] Each sample is assigned a logical constant, and then the multimodal semantic attribute representation is mapped to predicate form through an attribute logic mapper to generate a predicate set, which constitutes the symbolic multimodal attribute fact set of the sample. This multimodal attribute fact set enables the multimodal information to achieve structural unification at the symbolic level and provides underlying logical evidence and measurable attribute representation information for subsequent Markov logic network reasoning.

[0021] Furthermore, the steps of inputting the multimodal attribute fact set and the logical rules in the pre-constructed first-order logic knowledge base into the Markov logic network to complete attribute-level belief updates and category judgments within the probabilistic logic framework, and obtaining the inference results, specifically include:

[0022] The multimodal attribute fact set is divided into several attribute groups according to semantic relevance, and reasoning is performed on these attribute groups respectively;

[0023] For each attribute group, calculate the degree of correlation between it and the logical rules, and continuously adjust the confidence of the attributes in multiple iterations to gradually make them conform to the pattern required by the logical rules;

[0024] After multiple rounds of iterative updates, a result of the attribute is obtained after logical rule correction;

[0025] The attribute results after the logical rules are corrected are matched with the category prototype to generate the final category judgment.

[0026] Furthermore, the first-order logic knowledge base encodes domain knowledge in the form of first-order logic, including the implication relationship between attributes, the mutual exclusion relationship between attributes, the co-occurrence rules of attributes, the category discrimination rules, and the structural logical constraints related to the task.

[0027] Furthermore, the steps of integrating the reasoning results and generating interpretable decision outcomes and reasoning logic chains specifically include:

[0028] The reasoning results are applied to downstream tasks to generate interpretable decision results;

[0029] Based on the reasoning logic link generator, the reasoning process of Markov logic network is analyzed to generate interpretable reasoning logic links.

[0030] By outputting the decision results along with the underlying reasoning logic, it provides structured explanatory information and logical decision-making paths.

[0031] Another object of the present invention is to provide a neural symbolic reasoning system for multimodal information processing, for implementing the above-mentioned neural symbolic reasoning method, comprising:

[0032] The first-order logic knowledge base module is used to build and store first-order logic knowledge bases;

[0033] A multimodal input module is used to acquire multimodal input data; the multimodal input data includes image data and text data;

[0034] The multimodal attribute feature extraction module is used to extract the semantic attribute representations of image data and text data on a predefined attribute set, respectively, to obtain the multimodal semantic attribute representation;

[0035] The multimodal attribute fact building module is used to map multimodal semantic attribute representations into predicates of first-order logic form, and build a symbolic multimodal attribute fact set;

[0036] The symbolic logic reasoning module is used to input the multimodal attribute fact set and the logical rules in the pre-built first-order logic knowledge base into the Markov logic network, and complete the attribute-level belief update and category judgment under the probabilistic logic framework to obtain the reasoning result;

[0037] The downstream task execution and interpretation output module is used to integrate the reasoning results and generate interpretable decision results and reasoning logic links.

[0038] The neural symbolic reasoning method for multimodal information processing provided by this invention introduces an attribute-level semantic representation layer to convert continuous image and text features into logically operable semantic units. This layer is then combined with a pre-built first-order logic knowledge base to perform explicit probabilistic logical reasoning within a Markov logic network. This logical reasoning process uses shared attribute symbols across multiple modalities as a bridge to achieve information fusion, conflict resolution, and high-level consistency constraints among modalities. As a result, without relying on any latent space alignment method, it obtains reasoning results with interpretability, high robustness, and strong generalization ability.

[0039] This invention elevates multimodal fusion from a vector alignment paradigm to a symbolic logic alignment paradigm, enabling multimodal consistency at the logical level. Specifically, this invention fully utilizes knowledge unavailable in the image modality contained within the text modality and complements the perceptual attributes of the image side, transforming textual information into logical entities and combining them with the image modality. This effectively supplements the missing knowledge in the image modality, improving the accuracy and interpretability of the task, and allowing for reasonable decisions even in cases of image blurring, insufficient text, or modal conflicts. Furthermore, relying on probabilistic logical reasoning based on Markov logic networks, this invention performs particularly well in tasks involving similar classes and difficult-to-distinguish categories. It can strengthen the combination relationships of key attributes through logical rules, thereby significantly improving the robustness and reliability of classification. Simultaneously, this invention can output a complete reasoning logic chain, including attribute sources, logical rule triggering conditions, and conflict resolution methods, giving the multimodal system high interpretability and meeting the requirements for transparent decision-making in high-security scenarios. Attached Figure Description

[0040] Figure 1 This is an overall structural block diagram of a neural symbolic reasoning system for multimodal information processing provided in an embodiment of the present invention.

[0041] Figure 2 The training flowchart of the neural symbolic reasoning system for multimodal information processing provided in this embodiment of the invention is shown.

[0042] Figure 3 The inference flowchart of the neural symbolic inference system for multimodal information processing provided in this embodiment of the invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0044] The embodiments of this invention aim to solve the core technical bottlenecks in the prior art, such as the lack of interpretability of multimodal fusion, difficulty in utilizing explicit knowledge, difficulty in effectively leveraging modal complementarity, and insufficient ability to distinguish similar classes. By discretizing the semantic attribute layer and using Markov logic network reasoning, it achieves deep fusion of image, text, and logical knowledge, and provides an interpretable, controllable, and reliable neural symbolic reasoning method and system for multimodal information processing.

[0045] Specifically, in one embodiment of the present invention, a neural symbolic reasoning method for multimodal information processing is provided. This method is based on a first-order logic knowledge base and a Markov logic network (MLN), and is used for interpretable joint reasoning on multimodal data such as images and text. It can be extended to various downstream tasks, including but not limited to classification and visual question answering (VQA). The method specifically includes the following steps:

[0046] S1. Acquire multimodal input data; the multimodal input data includes image data and text data;

[0047] S2. Extract the semantic attribute representations of image data and text data on a predefined attribute set to obtain multimodal semantic attribute representations;

[0048] S3. Map the multimodal semantic attribute representations to predicates in first-order logical form to construct a symbolic multimodal attribute fact set;

[0049] S4. Input the multimodal attribute fact set and the logical rules in the pre-constructed first-order logic knowledge base into the Markov logic network, complete the attribute-level belief update and category judgment under the probabilistic logic framework, and obtain the reasoning result;

[0050] S5. Integrate the reasoning results and generate interpretable decision results and reasoning logic links.

[0051] Correspondingly, such as Figure 1 As shown, in another embodiment of the present invention, a neural symbolic reasoning system for multimodal information processing is also provided to implement the above-described neural symbolic reasoning method, comprising:

[0052] The first-order logic knowledge base module is used to build and store first-order logic knowledge bases;

[0053] A multimodal input module is used to acquire multimodal input data; the multimodal input data includes image data and text data;

[0054] The multimodal attribute feature extraction module is used to extract the semantic attribute representations of image data and text data on a predefined attribute set, respectively, to obtain the multimodal semantic attribute representation;

[0055] The multimodal attribute fact building module is used to map multimodal semantic attribute representations into predicates of first-order logic form, and build a symbolic multimodal attribute fact set;

[0056] The symbolic logic reasoning module is used to input the multimodal attribute fact set and the logical rules in the pre-built first-order logic knowledge base into the Markov logic network, and complete the attribute-level belief update and category judgment under the probabilistic logic framework to obtain the reasoning result;

[0057] The downstream task execution and interpretation output module is used to integrate the reasoning results and generate interpretable decision results and reasoning logic links.

[0058] The multimodal input module, the multimodal attribute feature extraction module, and the multimodal attribute fact construction module constitute the neural network, and the output of the neural network can be used as the input of the symbolic logic reasoning module.

[0059] The neural symbolic reasoning method and system provided in this invention aim to establish a unified symbolic representation between image and text modalities through explicit first-order logical reasoning. This addresses the problem that existing technologies cannot effectively combine image and text information in multimodal tasks, and has significant advantages, especially in the following scenarios: when categories are similar or difficult to distinguish, the system can effectively differentiate them through multimodal information. In some tasks, relying solely on image or text modalities often fails to yield clear conclusions. The system can handle these situations, which are difficult to derive through single-modal reasoning, by fusing image and text information and employing symbolic-level logical reasoning.

[0060] In a preferred embodiment of the present invention, a first-order logic knowledge base module is used to construct and store a set of logic rules pre-built by domain experts, encoding domain knowledge in first-order logic form to form a first-order logic knowledge base. This first-order logic knowledge base includes, but is not limited to, implication relationships between attributes, mutual exclusion relationships between attributes, co-occurrence patterns of attributes, category mapping rules, and other task-related structural logical constraints. The first-order logic knowledge base can serve as a static logical reference and also provide rule templates for MLN, providing explicit logical basis for adjusting attribute beliefs and category inference during the reasoning process. Through this module, the system can introduce explicit logical constraints when attributes are uncertain or feature boundaries are ambiguous, making the inference results interpretable, traceable, and auditable, and providing supplementary knowledge support for the decision-making process when image or text information is insufficient.

[0061] In a preferred embodiment of the present invention, the step of extracting semantic attribute representations of image data and text data on a predefined attribute set to obtain multimodal semantic attribute representations specifically includes:

[0062] High-level semantic features of image data are extracted by a visual semantic encoder. Then, based on a visual attribute discriminant subnetwork, these high-level semantic features are mapped to the semantic dimension of a predefined set of attributes. The activation values ​​of each attribute are output, and the activation values ​​of each attribute are transformed into interpretable attribute beliefs through thresholding or discretization mechanisms to obtain the semantic attribute representation of the image modality.

[0063] The semantic content shared with the visual attribute space in the text data is parsed by a text semantic encoder. Then, the text attribute extraction subnetwork generates text-side attribute prediction results based on the semantic content. The text-side attribute prediction results are transformed into attribute-level semantic belief representations through thresholding or discretization mechanisms, thus obtaining the semantic attribute representation of the text modality.

[0064] In practical applications, the multimodal attribute feature extraction module is responsible for transforming continuous image and text features into semantic attribute units that the system can directly manipulate. It serves as a crucial bridge between neural computation and symbolic reasoning in this embodiment of the invention. Structurally, this module includes a visual semantic encoder, a visual attribute discriminant subnetwork, a text semantic encoder, and a text attribute extraction subnetwork. The visual semantic encoder (such as a CNN or Vision Transformer) extracts high-level semantic features from the image. The visual attribute discriminant subnetwork maps these features to the semantic dimension of a predefined set of attributes, outputs the activation values ​​(attribute confidence scores) of each attribute, and transforms them into interpretable attribute beliefs (e.g., presence / absence relationships represented by 0 / 1) through thresholding or other discretization mechanisms. The text semantic encoder (such as models based on Transformer, BERT, LLM, or rule engines) parses semantic content shared with the visual attribute space in the text, such as structural descriptions, functional features, or contextual propositions. The text attribute extraction subnetwork then generates text-side attribute prediction results and corresponding attribute confidence scores. Similarly, the text-side prediction results are discretized into attribute-level semantic belief representations. Through this module, images and text are uniformly mapped to the same shared semantic attribute space, enabling implicit knowledge in the text modality to be explicitly presented and to play a role together with the image modality in subsequent logical reasoning.

[0065] In a preferred embodiment of the present invention, the step of mapping multimodal semantic attribute representations to predicates of first-order logic form and constructing a symbolic multimodal attribute fact set specifically includes:

[0066] Each sample is assigned a logical constant, and then the multimodal semantic attribute representation is mapped to predicate form through the attribute-to-predicate mapper to generate a predicate set, which constitutes the symbolic multimodal attribute fact set of the sample. This multimodal attribute fact set enables the multimodal information to achieve structural unification at the symbolic level and provides underlying logical evidence and measurable attribute representation information for subsequent Markov logic network reasoning.

[0067] In practical applications, the multimodal attribute fact construction module transforms the semantic attribute representations obtained from the two modalities mentioned above into first-order logical atomic predicates (FOL atoms) that the system can process, in order to construct the symbolic multimodal attribute fact set required for MLN inference. Specifically, this module first assigns a logical constant (e.g., 'o' represents the current object) to each sample, and then maps attribute beliefs into predicate forms through an attribute logic mapper: when the image modality predicts that a certain attribute a_k is true, the predicate HasAttr_k(o, image) is generated; when the text modality predicts that the attribute is true, the predicate HasAttr_k(o, text) is generated. Through this process, attribute estimates from different modalities are uniformly represented as a set of logical atoms, constituting the symbolic multimodal attribute fact set of the sample. This multimodal attribute fact set is the intermediate representation connecting the neural network and the symbolic logic inference module, enabling the structural unification of multimodal information at the symbolic level, and providing underlying logical evidence and measurable attribute representation information for subsequent MLN inference.

[0068] In a preferred embodiment of the present invention, the steps of inputting a multimodal attribute fact set and logical rules from a pre-constructed first-order logic knowledge base into a Markov logic network to complete attribute-level belief updates and category judgments within a probabilistic logic framework, and obtaining inference results, specifically include:

[0069] The multimodal attribute fact set is divided into several attribute groups according to semantic relevance, and reasoning is performed on these attribute groups respectively;

[0070] For each attribute group, calculate the degree of correlation between it and the logical rules, and continuously adjust the confidence of the attributes in multiple iterations to gradually make them conform to the pattern required by the logical rules;

[0071] After multiple rounds of iterative updates, a result of the attribute is obtained after logical rule correction;

[0072] The attribute results after the logical rules are corrected are matched with the category prototype to generate the final category judgment.

[0073] In practical applications, the symbolic logic reasoning module is the core of the system's symbolic reasoning process. Its role is to combine the attribute confidence scores provided by the neural network with pre-built logical rules, ensuring that the attribute reasoning process conforms to both data characteristics and logical knowledge constraints, thus achieving a more consistent and interpretable inference process. Before starting reasoning, this module receives a symbolic multimodal attribute fact set generated by the multimodal attribute fact construction module, along with the task-related candidate category range and some known attribute observations, and organizes this information into a structured input unit. Simultaneously, the module invokes a first-order logic knowledge base. Each rule describes the relationships between attributes in a first-order logic manner, such as which attributes typically appear together, which attributes cannot simultaneously be true, or whether certain attributes implicitly contain other higher-level semantic information. Furthermore, each rule has a learnable importance weight, allowing for slight deviations while maintaining overall logical consistency.

[0074] In the inference phase, this embodiment of the invention uses a Markov logic network to construct a joint model, treating the attribute confidence levels generated by the neural network as "preliminary judgments," which are then corrected using logical rules. To make the inference process computationally feasible, this embodiment divides all attributes into several attribute groups based on semantic relevance, and performs inference on these attribute groups separately, allowing the overall inference process to be executed in parallel. Within each attribute group, the degree of association between it and the logical rules is calculated, and the attribute confidence is continuously adjusted in multiple iterations to gradually conform to the pattern required by the logical rules. After several rounds of updates, the system obtains an attribute result corrected by the logical rules. This result is semantically more consistent and structured than the initial output of the neural network, and better reflects the logical constraints of domain knowledge. The corrected attribute result not only serves as a feedback signal, guiding the perception module to learn more consistent and symbolic attribute representations during the training phase, but also provides complete interpretable information, enabling the system to clearly explain which logical rules played a role in the inference process and which attributes were strengthened or weakened by the rules, thereby generating a clear logical inference chain.

[0075] To prevent MLN from becoming overconfident during inference, this embodiment of the invention incorporates an uncertainty adjustment term into the inference mechanism to maintain an appropriate probability distribution range for the inference output. Simultaneously, by adding constraints on known attribute observations, the inference results are made consistent with established information in the data. Finally, the logically corrected attribute results are matched with the category prototype to generate the final category determination. In this way, the symbolic logic inference module of this embodiment not only improves the logical consistency and interpretability of multimodal attribute inference but also enhances its robustness to noise and uncertainty, and exhibits better generalization ability on unseen categories.

[0076] In a preferred embodiment of the present invention, the step of integrating the reasoning results and generating interpretable decision results and reasoning logic links specifically includes:

[0077] The reasoning results are applied to downstream tasks to generate interpretable decision results;

[0078] Based on the reasoning logic link generator, the reasoning process of Markov logic network is analyzed to generate interpretable reasoning logic links.

[0079] By outputting the decision results along with the underlying reasoning logic, it provides structured explanatory information and logical decision-making paths.

[0080] In practical applications, the downstream task execution and interpretation output module includes a task execution unit, which is responsible for applying the MLN inference results to specific tasks and generating interpretable decision results. This task execution unit can perform various downstream tasks such as image or multimodal sample classification, retrieval, cross-modal matching, and risk labeling, depending on application requirements. The inference logic link generator parses the MLN inference process and outputs inference logic links regarding which attributes come from the image modality, which from the text modality, which attributes are strengthened or weakened under rule constraints, and which logical rules play a key role in the decision-making process. By outputting the decision results along with their underlying inference logic links, the system can provide structured explanatory information while maintaining high performance, meeting the auditing and interpretability requirements in high-reliability scenarios and providing users with a clear logical decision-making path.

[0081] like Figure 2 As shown, in a preferred embodiment of the present invention, the goal of the training phase during system operation is to enable the neural network to generate logically operable semantic units. The specific training method is as follows: Image training samples and optional text training samples are input through the training data input unit; the visual semantic encoder training unit performs feature extraction learning on the images, and the text semantic encoder training unit learns the text semantic representation; the attribute feature extraction unit maps multimodal features to attribute-level semantic beliefs, and forms attribute expressions operable by logical modules through a discretization mechanism; the attribute fact construction unit further transforms attribute semantics into symbolic facts. During the training phase, symbolic attributes and first-order logic rules can optionally be input into the MLN rule consistency constraint unit to apply weak logical supervision to attribute prediction; additionally, the entire network is jointly optimized through a loss function and a backpropagation unit, ultimately generating the trained attribute prediction model (i.e., the aforementioned neural network).

[0082] like Figure 3As shown, in a preferred embodiment of the present invention, during the inference stage, image input and text input are first received through a multimodal input unit; the visual semantic encoder and text semantic encoder extract image and text features respectively, and the multimodal attribute feature extraction unit generates attribute-level semantic beliefs for the samples; the multimodal attribute fact construction unit maps attribute-level semantic beliefs to symbolic logical facts, and inputs them together with the logical rules in the first-order logic knowledge base into the MLN inference unit for probabilistic logical inference. The MLN inference unit outputs logically corrected attribute representations, and the logically corrected attribute output unit generates the final attribute-level inference result. Subsequently, the category prototype matching unit performs similarity matching between the attribute-level inference result and the category prototype to complete the category determination. The inference logic link explanation output unit displays the logical rules triggered during the inference process, the attribute sources, and the decision basis, thereby achieving the interpretability of the decision-making process.

[0083] The following embodiments are practical applications of the methods and systems provided in the embodiments of the present invention, and are only illustrative examples and are not limited to these fields.

[0084] Example 1: This example applies the aforementioned neural symbolic reasoning system to an industrial equipment identification scenario. In this system: a visual semantic encoder extracts structural features and key component information from real-time acquired industrial equipment images; a text semantic encoder processes text descriptions from equipment databases, maintenance records, or label descriptions to extract semantic information such as equipment model, functional attributes, or structural features; a multimodal attribute feature extraction module converts visual and textual features into discrete attribute semantics, including whether the equipment possesses a certain structural component or conforms to a typical shape; a first-order logic knowledge base, constructed by engineering experts, contains logical rules such as co-occurrence relationships, mutual exclusion relationships, and functional implication relationships between equipment components; a multimodal attribute fact construction module converts attribute prediction results into logical predicates, forming a symbolic fact set that can be processed by the MLN; and a symbolic logic reasoning module, based on Markov logic networks, performs logical consistency reasoning and conflict correction on attribute predictions, generating logically corrected attribute vectors. The downstream task execution and interpretation output module uses the logically corrected attributes for classification and outputs an interpretable reasoning logic chain containing rule triggering conditions.

[0085] The system's workflow is as follows: The system acquires an image of equipment at the industrial site. The visual semantic encoder extracts the structural appearance information of the equipment surface. If the system stores a model document or structural description of the equipment, the text semantic encoder simultaneously extracts the attribute semantics from the text. Subsequently, the multimodal attribute feature extraction module maps the visual and textual information into a shared attribute space, generating corresponding attribute confidence scores. The multimodal attribute fact construction module transforms these attributes into logical predicates and inputs them, along with a first-order logic knowledge base, into the symbolic logic reasoning module. In this module, the system updates attribute beliefs through multiple rounds of reasoning based on logical rules. For example, if the rule "part A and part B must appear simultaneously" is triggered, the symbolic logic reasoning module automatically completes the missing part attributes; if an attribute prediction contradicts an engineering rule, the symbolic logic reasoning module logically corrects its confidence score. Finally, the system outputs a logically consistent attribute vector and matches it with the equipment category prototype to obtain the final category judgment result. Simultaneously, the system provides explanatory information including rule triggering conditions, attribute correction basis, and reasoning logic links for manual review or system scheduling.

[0086] This embodiment demonstrates that, in industrial equipment identification scenarios, the present invention can establish a logically consistent and semantically clear attribute inference process based on image and text information. Compared with traditional pure neural network methods, the present invention can maintain stable inference even under visual noise, equipment occlusion, or incomplete text descriptions; in fine-grained category differentiation, it can utilize logical rules to compensate for visual deficiencies and improve the accuracy of category determination.

[0087] Furthermore, the reasoning logic generated by the system makes each decision interpretable, clearly demonstrating the source of attributes, rule triggers, and the basis for logical corrections, thus meeting the practical needs of industrial safety audits and quality supervision. The system also boasts excellent scalability; when adding new equipment categories, only attributes and logical rules need to be added to support identification, eliminating the need to retrain the neural network and significantly reducing maintenance costs.

[0088] Example 2: This example applies the aforementioned neural symbolic reasoning system to a scenario involving the joint diagnosis of medical images (such as CT, MRI, X-ray) and clinical text (such as radiology reports and medical record summaries). In this system: A visual semantic encoder (medical image model) is used to extract structural features, abnormal signal regions, and organ morphological attributes from medical images such as CT / MRI / X-ray. A text semantic encoder (clinical text model) is used to process texts such as radiologist reports, electronic medical record summaries, and symptom descriptions, extracting medical attributes such as "enhancement level," "boundary clarity," and "size of suspicious lesions." A multimodal attribute feature extraction module maps the continuous features output by the image and text models to medical attribute-level semantics. A first-order logic knowledge base (medical diagnosis knowledge base) contains medical rules provided by clinical experts, such as disease-feature relationships, image-lesion associations, and symptom-pathology logic, for example: "The presence of lobulation usually indicates a higher probability of malignancy." A multimodal attribute fact construction module maps image attributes and text attributes to logical predicates, which are then input into the symbolic logic reasoning module. The symbolic logic reasoning module (MLN-based) performs attribute consistency reasoning and conflict correction based on medical logic rules. The downstream task execution and interpretation output module (downstream diagnostic assistance module) combines the reasoned attribute vectors to output lesion category predictions (e.g., benign / malignant) and risk scores, and provides a reasoning logic chain for physician review.

[0089] The system operates as follows: First, it receives a set of medical images (such as CT scan slices). The visual semantic encoder automatically identifies the lesion area and extracts image attributes. If clinical text (such as a radiology report or symptom record) is also input, the text semantic encoder extracts semantic attributes, such as "radiology report indicates blurred boundaries" or "suspected solid nodule." The multimodal attribute feature extraction module generates multimodal attributes based on image information and text descriptions, such as the presence of lobulation or whether the text describes "suspected inflammatory changes." Subsequently, the multimodal attribute fact construction module maps "image attributes + text attributes" into logical predicates, which are then input into the symbolic logic reasoning module. The symbolic logic reasoning module performs multiple rounds of reasoning on attribute beliefs based on expert-defined medical rules in the knowledge base. For example, if the image indicates "lobulation," but the text report states "clear boundaries," the system determines the logical priority of the image attributes based on rules and automatically corrects attribute conflicts; if the text states "lesion stable for six months," the system strengthens benign-related attributes; if the image shows "ground-glass opacity + partial solidity," the system completes the inference of "subsolid nodule" based on logical rules. After reasoning is complete, the system generates a logically corrected attribute vector and matches it with the disease category prototype to obtain the final diagnostic assistance result. Simultaneously, the system outputs the rule chains and attribute sources that influence the diagnosis, enabling doctors to clearly understand the basis of the system's inferences.

[0090] This embodiment demonstrates that the present invention can be effectively applied to complex diagnostic scenarios involving the fusion of medical images and clinical text. Compared with traditional deep learning diagnostic systems, the present invention has the following significant technical advantages: It maintains stable inference even under multimodal conflict conditions. In medical scenarios, images and text often differ; the present invention can unify them through medical logic rules, making the inference results more consistent and reasonable. It significantly improves inference performance for complex cases with unclear boundaries and mild feature ambiguity. MLN inference can utilize symptom descriptions or text records to compensate for unclear image features, improving recognition accuracy. It possesses high interpretability, meeting medical safety and auditing requirements. The system outputs a diagnostic link, showing: which attributes come from images, which from text, and which are completed by medical rules. It meets the stringent requirements of the medical industry for traceability and auditability. It has better generalization ability in scenarios with no observed cases or rare diseases. Because the present invention is based on "medical attributes + medical logic rules" rather than category features for reasoning, it can form reasonable inferences for rare diseases or those with no observed lesions. It can adapt to new disease types without extensive labeling, and the system can be expanded by supplementing attribute definitions and medical logic. It does not require retraining neural networks, reducing deployment costs for medical institutions.

[0091] Example 3: This example applies the above-described neural symbolic reasoning system to image classification. In this system: a visual semantic encoder is used to extract high-level semantic features of the image; a text semantic encoder is used to process the input text description or auxiliary labels; a multimodal attribute feature extraction module is used to discretize visual and text features into shared attribute semantics; a first-order logic knowledge base contains logical rules such as co-occurrence, mutual exclusion, and implication; a multimodal attribute fact construction module is used to map attributes to logical predicates; a symbolic logic reasoning module is used to implement attribute consistency reasoning and conflict correction; and a downstream task execution and interpretation output module is used for the final category prediction and reasoning logic link output.

[0092] The system's workflow is as follows: The system takes an image and its corresponding text description as input. An initial attribute confidence score is generated using a visual encoder, while supplementary attribute semantics are extracted using a text encoder. Subsequently, the visual and text attributes are jointly mapped to logical facts and input into the symbolic logic reasoning module. The module iteratively updates the attribute beliefs using logical rules from the knowledge base, outputting a logically corrected attribute vector. Finally, this attribute vector is matched with the category prototype to obtain the classification result, and an interpretable reasoning logic chain involving rule triggering scenarios is generated.

[0093] This embodiment demonstrates that the present invention can successfully complete the classification output with logical consistency from multimodal inputs of images and text. Compared with traditional neural networks, this embodiment can maintain inference stability even under attribute misjudgment conditions, while providing clear explanatory information, improving the reliability and transparency of inference.

[0094] In summary, the embodiments of the present invention significantly improve the performance and interpretability of multimodal reasoning systems by implementing cross-modal logic alignment at the symbolic layer and combining attribute-level semantic expression with symbolic logic reasoning mechanisms. Compared with existing technologies, it has the following technical effects:

[0095] (1) Cross-modal logical consistency constraints significantly enhance inference stability: For the first time, this invention elevates the fusion of image and text modalities from the traditional latent space vector alignment method to logical-level alignment, achieving cross-modal consistency constraints through a unified attribute semantic space and first-order logical rules. Without the need for any embedding alignment or contrastive learning, the system can achieve unified inference between modalities under the guidance of logical rules, effectively avoiding problems such as embedding space drift, modal bias, and representation instability, thereby significantly improving the reliability and controllability of the multimodal fusion process.

[0096] (2) Higher discrimination ability in fine-grained and similar category tasks: For similar categories where image modalities are difficult to distinguish, text modalities are ambiguous, or both are ambiguous, the embodiments of the present invention can use logical rules to supplement attribute-level inference, enabling the system to make consistent and reasonable judgments when the category differences are extremely subtle. At the same time, it can automatically supplement missing semantics when image or text modal information is insufficient, and logically correct unreasonable attributes, thereby forming clearer semantic boundaries between similar categories and significantly improving the discrimination ability in fine-grained recognition.

[0097] (3) The reasoning process is fully explainable and traceable, meeting the requirements of high reliability scenarios: The embodiments of the present invention can generate reasoning logic links at the attribute level, including the source of each attribute (image, text or logical reasoning), the triggered logical rules, the direction of the rule's influence on the attribute, and the basis for the final category judgment. Every decision of the system can provide a complete and transparent logical explanation, which helps to meet the requirements of traceability, auditability and security supervision in high reliability scenarios such as military, transportation, autonomous driving, and medical care. This explainability is difficult to achieve by traditional deep learning models.

[0098] (4) Higher robustness to modality loss and uncertainty: The embodiments of the present invention, through a probabilistic logic reasoning mechanism, enable the system to output stable and reasonable inference results even when faced with complex conditions such as visual noise, image blurring, missing text, modality inconsistency, or partial attribute loss. Logical rules constrain or correct unreasonable attributes, while attribute co-occurrence and semantic hierarchy relationships can complete missing attributes, enabling the system to maintain high inference accuracy and reliability even with incomplete information. Compared with pure neural network models that rely on continuous embedding, the embodiments of the present invention show significantly improved robustness in real-world scenarios.

[0099] (5) The neural network and symbolic reasoning form a closed-loop optimization, resulting in continuous improvement in overall performance: In this embodiment of the invention, the attribute prediction results after logical correction are passed back to the neural network, enabling the neural network to gradually learn more consistent and symbolic attribute representations. This closed-loop structure allows the neural network to benefit from the structural constraints of symbolic rules during training, while symbolic reasoning continuously absorbs the high-dimensional perception capabilities of the neural network, thereby achieving bidirectional enhancement. Ultimately, the system achieves significant performance improvements in attribute prediction, category recognition, and modality fusion.

[0100] It should be noted that each of the above modules or units can be implemented as a computer program, which can run on a computer device. The computer device's memory can store the computer program that makes up each module, enabling the processor to execute each step of the above method.

[0101] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0103] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A neural symbolic reasoning method for multimodal information processing, characterized in that, Includes the following steps: Acquire multimodal input data; the multimodal input data includes image data and text data; Semantic attribute representations of image data and text data on a predefined attribute set are extracted separately to obtain multimodal semantic attribute representations; The semantic attribute representations of multimodalities are mapped to predicates in first-order logical form, thus constructing a symbolic set of multimodal attribute facts. The multimodal attribute fact set and the logical rules in the pre-constructed first-order logic knowledge base are input into the Markov logic network. Under the probabilistic logic framework, attribute-level belief updates and category judgments are completed to obtain the reasoning results. The reasoning results are integrated to generate interpretable decision results and reasoning logic links; The steps for extracting semantic attribute representations of image data and text data on a predefined attribute set to obtain multimodal semantic attribute representations specifically include: High-level semantic features of image data are extracted by a visual semantic encoder. Then, based on a visual attribute discriminant subnetwork, these high-level semantic features are mapped to the semantic dimension of a predefined set of attributes. The activation values ​​of each attribute are output, and the activation values ​​of each attribute are transformed into interpretable attribute beliefs through thresholding or discretization mechanisms to obtain the semantic attribute representation of the image modality. The semantic content shared with the visual attribute space in the text data is parsed by a text semantic encoder. Then, the text attribute extraction subnetwork generates text-side attribute prediction results based on the semantic content. The text-side attribute prediction results are transformed into attribute-level semantic belief representations through thresholding or discretization mechanisms, thus obtaining the semantic attribute representation of the text modality.

2. The neural symbolic reasoning method for multimodal information processing according to claim 1, characterized in that, The steps of mapping multimodal semantic attribute representations to predicates in first-order logical form and constructing a symbolic multimodal attribute fact set specifically include: Each sample is assigned a logical constant, and then the multimodal semantic attribute representation is mapped to predicate form through an attribute logic mapper to generate a predicate set, which constitutes the symbolic multimodal attribute fact set of the sample. This multimodal attribute fact set enables the multimodal information to achieve structural unification at the symbolic level and provides underlying logical evidence and measurable attribute representation information for subsequent Markov logic network reasoning.

3. The neural symbolic reasoning method for multimodal information processing according to claim 1, characterized in that, The steps involved in inputting a multimodal attribute fact set along with logical rules from a pre-built first-order logic knowledge base into a Markov logic network, and then performing attribute-level belief updates and category judgments within a probabilistic logic framework to obtain the inference result, specifically include: The multimodal attribute fact set is divided into several attribute groups according to semantic relevance, and reasoning is performed on these attribute groups respectively; For each attribute group, calculate the degree of correlation between it and the logical rules, and continuously adjust the confidence of the attributes in multiple iterations to gradually make them conform to the pattern required by the logical rules; After multiple rounds of iterative updates, a result of the attribute is obtained after logical rule correction; The attribute results after the logical rules are corrected are matched with the category prototype to generate the final category judgment.

4. The neural symbolic reasoning method for multimodal information processing according to claim 1 or 3, characterized in that, The first-order logic knowledge base encodes domain knowledge in the form of first-order logic, including the implication relationship between attributes, the mutual exclusion relationship between attributes, the co-occurrence rules of attributes, the classification rules, and the structural logical constraints related to the task.

5. The neural symbolic reasoning method for multimodal information processing according to claim 1, characterized in that, The steps for integrating the reasoning results and generating interpretable decision outcomes and reasoning logic chains specifically include: The reasoning results are applied to downstream tasks to generate interpretable decision results; Based on the reasoning logic link generator, the reasoning process of Markov logic network is analyzed to generate interpretable reasoning logic links. By outputting the decision results along with the underlying reasoning logic, it provides structured explanatory information and logical decision-making paths.

6. A neural symbolic reasoning system for multimodal information processing, used to implement the neural symbolic reasoning method according to any one of claims 1-5, characterized in that, include: The first-order logic knowledge base module is used to build and store first-order logic knowledge bases; A multimodal input module is used to acquire multimodal input data; the multimodal input data includes image data and text data; The multimodal attribute feature extraction module is used to extract the semantic attribute representations of image data and text data on a predefined attribute set, respectively, to obtain the multimodal semantic attribute representation; The multimodal attribute fact building module is used to map multimodal semantic attribute representations into predicates of first-order logic form, and build a symbolic multimodal attribute fact set; The symbolic logic reasoning module is used to input the multimodal attribute fact set and the logical rules in the pre-built first-order logic knowledge base into the Markov logic network, and complete the attribute-level belief update and category judgment under the probabilistic logic framework to obtain the reasoning result; The downstream task execution and interpretation output module is used to integrate the reasoning results and generate interpretable decision results and reasoning logic links.

Images