Multi-modal causal reasoning and explaining method, device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a multi-modal causal reasoning and interpretation method, device, equipment and medium, and the method comprises the steps: obtaining original data streams of at least two different modals, and extracting modal features; a cross-modal attention mechanism is utilized to fuse modal features, and causal features are extracted through feature distillation; constructing a dynamic causal graph based on causal features, and updating an edge weight through a causal intensity function; identifying the causal relationship in the dynamic causal graph and performing anti-factual reasoning verification to evaluate the reliability of the causal relationship; and generating a causal interpretation result in combination with the dynamic causal graph and the causal relationship reliability. According to the method, the multi-modal data are fused, the causal features are extracted, and dynamic causal graph updating and anti-factual reasoning verification are combined, so that reliable modeling and explanation of the causal relationship in a complex scene are realized, the defects of single modal or simple fusion in the prior art are overcome, and the accuracy and interpretability of causal reasoning are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal causal reasoning and interpretation method, apparatus, device, and storage medium. Background Technology

[0002] In existing technologies, causal reasoning methods mostly focus on the analysis of single-modal data, such as mining causal relationships using only text data. These methods ignore the potential causal information in other modal data such as images, audio, and sensor signals, resulting in causal relationships that are often one-sided and incomplete. Even in multimodal scenarios, existing methods mainly remain at the level of simple feature fusion to achieve traditional tasks such as classification and recognition, lacking the ability to deeply model and reason about the causal structure between multiple modalities.

[0003] This deficiency is particularly pronounced in the healthcare sector. Clinical diagnosis often involves multiple sources of information, including electronic medical records, medical images, and vital sign monitoring data. Existing technologies typically rely on a single data source for causal analysis, such as making causal inferences based solely on medical record text, without fully utilizing causal clues in medical images or physiological signals. This limitation makes it difficult for models to accurately reveal the interactive causal relationships between different physiological mechanisms when dealing with the causes of complex diseases, thus affecting the reliability of clinical decision support.

[0004] Similar issues exist in risk assessment and event prediction within the fintech sector. Existing models often rely on single-modal data such as transaction records and text announcements, while underutilizing other modalities like market trends, user behavior logs, and voice communication data. Especially when facing complex market environments, traditional causal inference models often rely solely on statistical correlations for inference, struggling to address indirect causal chains and confounding factors present in multimodal data. This makes it easy for financial institutions to make judgment biases and fail to fully grasp potential causal relationships when conducting risk monitoring and anomaly prediction. Summary of the Invention

[0005] The main objective of this invention is to provide a multimodal causal reasoning and interpretation method, apparatus, device, and storage medium, aiming to solve the technical problem that existing technologies mostly remain at the level of single-modal causal analysis or simple multimodal feature fusion, lacking in-depth modeling and reasoning capabilities for complex causal structures between multimodal data, resulting in difficulty in comprehensively and accurately revealing cross-modal causal relationships.

[0006] To achieve the above objectives, the present invention provides a multimodal causal reasoning and explanation method, comprising:

[0007] Acquire multimodal data from raw data streams containing at least two different modalities;

[0008] Feature extraction processing is performed on each of the original data streams in the multimodal data to generate corresponding modal features;

[0009] The modal features are fused across modalities, and the fusion weights are adjusted through a cross-modal attention mechanism to generate fused features. Causal features are then extracted from the fused features through a feature distillation process.

[0010] Based on the causal features, a dynamic causal graph is constructed, which includes nodes representing entities and edges representing causal relationships, and the edges are updated using a causal strength function.

[0011] Counterfactual reasoning is performed on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships;

[0012] A causal explanation is generated based on the dynamic causal graph and the reliability of the causal relationships.

[0013] Furthermore, to achieve the above objectives, the present invention provides a multimodal causal reasoning and explanation apparatus, comprising:

[0014] A multimodal data acquisition module is used to acquire multimodal data from raw data streams that include at least two different modalities.

[0015] The modal feature extraction module is used to perform feature extraction processing on each type of raw data stream in the multimodal data to generate corresponding modal features.

[0016] The cross-modal fusion and distillation module is used to perform cross-modal fusion of the modal features, adjust the fusion weights through a cross-modal attention mechanism to generate fused features, and extract causal features from the fused features through a feature distillation process.

[0017] The dynamic causal graph construction and update module is used to construct a dynamic causal graph containing nodes representing entities and edges representing causal relationships based on the causal features, and to update the edges using a causal strength function;

[0018] The counterfactual reasoning verification module is used to perform counterfactual reasoning verification on the causal relationships identified in the dynamic causal graph in order to analyze the reliability of the causal relationships;

[0019] The causal explanation generation module is used to generate causal explanation results based on the dynamic causal graph and the reliability of the causal relationships.

[0020] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a multimodal causal reasoning and interpretation program stored in the memory and executable on the processor, wherein when the multimodal causal reasoning and interpretation program is executed by the processor, it implements the steps of the multimodal causal reasoning and interpretation method as described above.

[0021] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a multimodal causal reasoning and interpretation program, which, when executed by a processor, implements the steps of the multimodal causal reasoning and interpretation method as described above.

[0022] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a multimodal causal reasoning and explanation method, apparatus, device, and medium, comprising: acquiring raw data streams from at least two different modalities and extracting features to generate modal features; fusing the modal features through a cross-modal attention mechanism and refining causal features through a feature distillation process; constructing a dynamic causal graph containing entity nodes and causal edges based on the causal features, and updating the edges using a causal strength function; identifying causal relationships in the dynamic causal graph and performing counterfactual reasoning verification to analyze the reliability of the causal relationships; and generating causal explanation results based on the dynamic causal graph and the reliability of the causal relationships. This invention introduces a causal feature extraction mechanism during the feature extraction and cross-modal fusion process of multimodal data to model causal associations between different modalities; dynamically updates the causal graph structure through a causal strength function to ensure the temporal accuracy and stability of causal relationships; improves the reliability assessment of causal relationships through counterfactual reasoning verification; and outputs causal explanation results based on this. This invention overcomes the limitations of existing technologies that rely solely on a single modality or simple feature fusion, significantly improving the accuracy and interpretability of causal reasoning in complex multimodal scenarios. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0024] Figure 1 This is a schematic diagram of an application environment for a multimodal causal reasoning and interpretation method according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating an embodiment of the multimodal causal reasoning and explanation method of the present invention;

[0026] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal causal reasoning and explanation device of the present invention;

[0027] Figure 4This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0028] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] The multimodal causal reasoning and explanation method provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain raw data streams from at least two different modalities from the user terminal, extract features to generate modal features, fuse the modal features through a cross-modal attention mechanism, and extract causal features through a feature distillation process. Based on the causal features, a dynamic causal graph containing entity nodes and causal edges is constructed, and the edges are updated using a causal strength function. Causal relationships are identified in the dynamic causal graph, and counterfactual reasoning is performed to verify and analyze the reliability of the causal relationships. Based on the dynamic causal graph and the reliability of the causal relationships, a causal explanation result is generated. This invention introduces a causal feature extraction mechanism into the feature extraction and cross-modal fusion process of multimodal data to model causal associations between different modalities; dynamically updates the causal graph structure through a causal strength function to ensure the temporal accuracy and stability of causal relationships; improves the reliability assessment of causal relationships through counterfactual reasoning verification; and outputs a causal explanation result on this basis. This invention overcomes the limitations of existing technologies that rely solely on a single modality or simple feature fusion, and significantly improves the accuracy and interpretability of causal reasoning in complex multimodal scenarios. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multimodal causal reasoning and explanation method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the multimodal causal reasoning and explanation method proposed in this invention includes the following steps:

[0033] S10, acquire multimodal data including raw data streams of at least two different modalities;

[0034] In this embodiment, acquiring multimodal data comprising at least two different modalities of raw data streams involves establishing a multi-source-oriented input channel at the data receiving end. The raw data streams are unprocessed signals directly from the acquisition device, without any feature processing or compression. For example, visual modal data consists of video frame sequences output by a camera device; text modal data consists of character sequences generated by keyboard input, a natural language processing interface, or a speech-to-text module; speech modal data consists of continuous waveforms acquired by a microphone; and physiological modal data consists of ECG, blood pressure, or blood oxygenation sequences acquired by sensors. The simultaneous presence of at least two modalities ensures that the input data is semantically complementary.

[0035] Each data stream from different modalities needs to be timestamped upon entering the system to mark the data acquisition time. The timestamps are provided by a unified system clock, with accuracy down to the millisecond or microsecond level. To avoid misalignment caused by asynchronous acquisition, the receiving module buffers data from different modalities and aligns the data based on the timestamps. The alignment operation is implemented using a sliding window algorithm or a dynamic time warping algorithm, ensuring that data from different modalities correspond on a unified timeline.

[0036] During data acquisition, data integrity is verified by an anomaly detection module. For video streams, it detects missing frames in the frame sequence; for text streams, it detects truncation or encoding anomalies in character sequences; and for sensor streams, it detects abrupt changes or missing data in the signal waveform. When the integrity level is detected to be below a preset threshold, the data stream is discarded. The integrity detection results are recorded along with the original data for subsequent processing reference.

[0037] All raw data streams that pass integrity checks are ultimately encapsulated into data packets of a uniform format. These packets include a modality identifier field, a timestamp field, a data length field, and a raw content field. The modality identifier distinguishes different modalities, the timestamp is used for subsequent sequential reasoning, the data length is used for storage and retrieval optimization, and the raw content field stores the unprocessed raw data.

[0038] During implementation, visual and speech modal data can be acquired through a combination of cameras and microphones, or text and physiological modal data can be acquired through a combination of text input interfaces and medical sensors. The choice of modal combination varies depending on the scenario. In a healthcare environment, electrocardiogram (ECG) streams and medical image streams can be selected as input modalities, while in a financial business environment, transaction log streams and news text streams can be selected as input modalities.

[0039] Data alignment can employ strict timestamp alignment or dynamic time warping. For data integrity verification, rule-based threshold detection methods can be used, or anomaly detection models can be introduced for automated identification. Data packet encapsulation formats can be optimized based on binary protocols or implemented using structured protocols such as JSON or Protobuf for cross-platform transmission.

[0040] Example: In the field of healthcare, medical imaging data and electrocardiogram (ECG) signal data from patients can be received simultaneously. Medical imaging data is input in the form of video frames, while ECG signals are input as time series. The correspondence between the two modalities in time is achieved through timestamp alignment, ensuring that images and physiological indicators can be combined when analyzing a certain lesion.

[0041] In the fintech business, it can simultaneously receive market transaction data and news text data. Transaction data records prices and trading volumes in the form of time series, while news text data is input as a character sequence. By aligning with timestamps, transaction fluctuations can be correlated with market news, providing a complete data source for subsequent causal relationship analysis.

[0042] This embodiment, by acquiring raw data streams from at least two different modalities, ensures the diversity and complementarity of input information, avoiding information loss and causal inference bias caused by single-modal data. A unified timestamp and alignment method makes cross-modal data comparable in the time dimension, while integrity verification and elimination mechanisms ensure the stability of subsequent analysis. The resulting multimodal data package provides a reliable input foundation for cross-modal fusion and causal modeling.

[0043] S20, Perform feature extraction processing on each type of raw data stream in the multimodal data to generate corresponding modal features;

[0044] In this embodiment, feature extraction processing for each type of raw data stream in multimodal data refers to transforming unprocessed signals from different sensors or acquisition devices into structured representations that can be used for computation and fusion. The raw data streams may be spatial data, time-series data, symbolic sequence data, or composite signals, and different types of data require targeted processing methods.

[0045] Spatial dimension data typically consists of two-dimensional or three-dimensional signals such as video frames, medical images, or remote sensing images. Local spatial features are extracted from this type of data through convolution operations. The size and stride of the convolution kernel determine the resolution and coverage of the local features. After convolution processing, a positional encoding function is introduced to temporally supplement the spatial features, enabling the static features of a single frame to establish a connection along the timeline.

[0046] Time series data typically includes electrocardiogram (ECG) signals, trading time series, and physical quantities collected by sensors. This type of data requires processing using sequence modeling methods, such as recurrent neural networks or one-dimensional convolutional networks to capture temporal dependencies. By introducing a sliding window mechanism, local patterns in the sequence can be extracted and combined with weighting functions to form a temporal feature vector.

[0047] Text streams or symbolic sequence data include natural language input, transaction texts, medical reports, etc. The processing typically involves pre-trained language models that map symbolic sequences to semantic embeddings. A linear transformation is then performed using a weight matrix to adjust the importance of different semantic dimensions. The resulting features incorporate contextual information while preserving symbolic-level distinctiveness.

[0048] For the extracted spatial, time-series, and textual features, a unified standardization process is required. This standardization includes feature dimension unification, data format alignment, and feature range normalization. Subsequently, feature dimension description information is added to each modality feature to label the feature type, number of dimensions, and data source, ensuring correct identification of the feature properties of different modalities during subsequent fusion. The final output modality features correspond one-to-one with the input raw data stream, maintaining traceability and independence.

[0049] In visual modality processing, convolutional neural networks can be used to extract spatial features, which can then be combined with self-attention mechanisms to generate spatial self-attention features. For video frame sequences, joint modeling of spatial and temporal features can be achieved through 3D convolution, or cross-frame dependencies can be captured through 2D convolution combined with temporal position encoding.

[0050] In text modality processing, initial semantic features can be generated directly by calling the embedding layer of a pre-trained language model, or a language model fine-tuned with domain data can be loaded in specific domain scenarios to enhance the specialization of the features. Subsequently, matrix transformations are used to adjust the distribution of semantic vectors to adapt to the input requirements of subsequent fusion structures.

[0051] In time series modality processing, local patterns can be captured through one-dimensional convolution, or long short-term memory networks can be used to model long-term dependencies. For high-frequency sampled financial transaction data, downsampling and window slicing can be used to reduce redundancy; for medical and physiological signals, filtering and noise suppression can be added to improve feature stability.

[0052] For standardization, one can choose an automatic alignment method based on a normalization layer, or one can choose to achieve structural alignment of different modal features through a unified feature dimension mapping. Feature dimension description information can be encoded using additional vectors, or it can be stored directly as labels for the feature matrix.

[0053] Example: In the healthcare field, convolutional processing can be performed on a patient's medical image stream to extract spatial features of lesion areas. Simultaneously, sequence modeling of electrocardiogram (ECG) signals can be performed to extract heart rhythm variation patterns. Furthermore, language modeling can be used to encode the medical record text report to obtain semantic features. These three modal features, after standardization, are uniformly input into the subsequent causal modeling module to achieve a multidimensional interpretation of the disease state.

[0054] In the fintech field, one-dimensional convolutional processing can be applied to transaction log streams to extract fluctuation patterns in trading volume and price; language modeling can be used to process news text streams to extract semantic features related to public opinion; and image processing can be performed on market charts to extract price curve features. After standardization and the addition of descriptive information, transaction features, semantic features, and image features can be processed within a unified framework to explain the causal mechanisms of market events and price fluctuations.

[0055] This embodiment performs targeted feature extraction processing on each type of raw data stream, which can maximize the preservation of key information in different modalities and eliminate redundant noise. Spatial modality features can reflect the local and global structure in an image, textual modality features can reflect semantic content, and time series features can reflect dynamic evolution processes. After standardization and additional descriptive information processing, different modality features can be fused in a unified representation space, providing a reliable input foundation for cross-modal association and causal relationship modeling.

[0056] S30, perform cross-modal fusion of the modal features, adjust the fusion weights through a cross-modal attention mechanism to generate fused features, and extract causal features from the fused features through a feature distillation process;

[0057] In this embodiment, cross-modal feature fusion refers to mapping modal features extracted from different original data streams to a shared representation space, within which information interaction is achieved by calculating the correlation between modalities. Cross-modal attention mechanisms are a key method used in the fusion process. They establish mutual attention relationships between modal features and dynamically allocate fusion weights, thereby highlighting complementary information between modalities.

[0058] Cross-modal attention mechanisms are typically computed based on a similarity function. Features from one modality are used as the query vector, and features from another modality are used as keys and values. Attention weights are derived by calculating the matching degree between the query and the key, and these weights are then used to perform a weighted sum of the values. This allows for feature complementarity between modalities, such as the correspondence between text descriptions and visual targets. To prevent weights from concentrating in a particular area, regularization constraints or normalization operations are usually introduced into the attention computation to ensure that features from different modalities remain relatively balanced during the fusion process.

[0059] After generating the fused features, causal features need to be extracted from them through a feature distillation process. Feature distillation is a knowledge compression method that passes the high-dimensional fused information learned from the upper-layer structure to the lower-layer inference model through a distillation loss function. In this process, the fused features are first input into the upper-layer model (such as a Transformer structure) for encoding, generating an initial fused output. The lower-layer inference model then gradually converges to a more stable and causally interpretable low-dimensional feature representation by comparing the error with the upper-layer output. The causal features that are ultimately retained not only reflect cross-modal semantic alignment but also avoid the interference of redundant information and noise on the modeling of causal relationships.

[0060] In visual-text modal fusion scenarios, visual feature vectors can be used as queries, and text feature vectors as keys and values. The similarity between the two can be calculated to generate weights. This highlights the image regions relevant to the text description, thus forming semantically consistent fused features. Conversely, text features can be used as queries, and visual features as keys and values ​​to capture the corresponding regions of text keywords in visual space.

[0061] In the fusion of medical images and physiological signals, cross-modal attention mechanisms can be used to calculate the mutual information between image modal features and temporal modal features, thereby identifying the correspondence between image regions and specific physiological fluctuations. In this implementation, the weights are not only based on similarity but can also be weighted by combining mutual information values ​​to enhance the robustness of cross-modal dependencies.

[0062] Feature distillation can be implemented in several ways. For example, an initial fusion output can be generated in the upper-layer model using a Transformer, while the lower-layer inference model employs a lightweight neural network structure to extract causal features by minimizing the difference between the initial fusion output and the lower-layer causal features. Alternatively, the sensitivity of the distillation loss can be adjusted by using a temperature parameter, ensuring that the distillation process converges to stable causal features even under different modal imbalances.

[0063] The design of the distillation loss function can vary depending on the application scenario. In scenarios where high prediction accuracy is required, cross-entropy can be chosen as the loss function; in scenarios where stronger causal explanation is required, a composite loss function combining mean squared error and contrastive learning can be chosen to ensure that causal features are stable at both the prediction and explanation levels.

[0064] Example Explanation: In the healthcare field, lesion region features in medical images can be aligned with symptom descriptions in medical case texts using a cross-modal attention mechanism. After fusion, these features are input into an upper-level Transformer to generate fused features, which are then refined through feature distillation to extract causal features. These causal features can reflect the causal relationship between imaging lesions and clinical symptoms, providing doctors with interpretable auxiliary diagnostic evidence.

[0065] In the fintech field, numerical features of market price curves can be fused with semantic features of financial news texts across modalities. By dynamically adjusting weights through an attention mechanism, a correlation can be established between key events mentioned in the news and market fluctuations. Causal features obtained through feature distillation and compression can reveal the causal relationship between market events and price fluctuations, thereby providing support for risk control and investment decisions.

[0066] This embodiment utilizes a cross-modal attention mechanism to establish fine-grained dependencies between features from different modalities, preventing single-modal information from dominating the result and ensuring the comprehensiveness of the fused features in terms of information representation. Furthermore, feature distillation compresses the high-dimensional fused features into causal features, ensuring that the final output retains complementary cross-modal information while removing redundant parts, thereby improving the accuracy and interpretability of subsequent causal inference.

[0067] S40, construct a dynamic causal graph based on the causal features, which includes nodes representing entities and edges representing causal relationships, and update the edges using a causal strength function;

[0068] In this embodiment, constructing a dynamic causal graph based on causal features means using the causal features obtained through feature distillation as input to identify different entities and the causal relationships between them. A causal graph is a directed graph structure where nodes represent entities and edges represent causal relationships. Entities can be concepts, events, or objects extracted from multimodal data, and causal relationships are measured using a causal strength function.

[0069] The construction process begins with parsing causal features to extract entity information related to different modalities. The extracted entity set is mapped to nodes, each containing a specific attribute vector describing its behavior in the original data. Subsequently, based on the statistical dependencies and causal inference mechanisms between nodes, directed edges connecting them are established. The initial edge weights can be estimated using information-theoretic metrics or statistical learning models.

[0070] The process of updating edge weights is driven by a causal strength function. This function typically integrates metrics such as transitive entropy, conditional mutual information, or intervention-based causal effect estimation to quantify the strength of the causal relationship between two nodes. By calculating these metrics between the features of the two nodes connected by each edge and applying a balance coefficient and a nonlinear activation function, updated edge weights can be generated. These updated weights replace the original edge weights in the graph, enabling the dynamic causal graph to continuously reflect changes in causal relationships over time and with data.

[0071] The dynamism is reflected in the continuous iterative updates. Each new data input triggers the recalculation of causal features, leading to changes in node attributes and edge weights. As the update history accumulates, a trajectory of edge weights over time can be formed, providing a basis for subsequent causal stability analysis and counterfactual reasoning.

[0072] In applications that fuse visual and textual modalities, objects in an image (such as image regions) and concepts described in text (such as lesion symptoms) can be used as nodes. Edges are established by calculating the conditional mutual information between the causal features of the two. The causal strength function can combine transitive entropy and mutual information to update the edge weights between image nodes and text nodes, ensuring that the graph structure can dynamically characterize the causal changes between text and visual modalities.

[0073] In healthcare applications, patient imaging modalities and physiological signal modalities can be utilized. First, lesion and signal features at different time points are extracted to form a node set. Then, the causal strength between lesions and signals is calculated based on time-series transfer entropy. The resulting values ​​are processed using an activation function to update the edge weights, thereby reflecting the disease evolution process in a dynamic causal graph.

[0074] In fintech applications, stock price changes and market news events can be used as nodes to calculate the causal relationships between them. By combining conditional mutual information and market impact factors through a causal strength function, the causal strength of an event on price fluctuations can be obtained, and the edges in the dynamic causal graph can be updated in real time. This implementation method can maintain the dynamic accuracy of the graph structure even when the market environment changes rapidly.

[0075] During the update process, different activation functions can be used. For example, the Sigmoid function keeps the edge weights in the [0,1] range, which is suitable for probabilistic modeling; the ReLU function can also be used to preserve the non-negativity of causal strength and highlight strong causal relationships. Different implementations are suitable for different causal modeling needs.

[0076] This embodiment constructs a dynamic causal graph using causal features, which can fully represent entities and causal relationships at the structural level, ensuring that information from different modalities is aligned within a unified graph structure. Updating edge weights using a causal strength function effectively filters out spurious correlations, retaining truly causal connections, allowing the causal graph to evolve over time and consistently reflect the true causal mechanism. This dynamic update mechanism enables causal relationship modeling to possess continuous adaptability and interpretability, providing reliable input for subsequent counterfactual reasoning and causal explanation.

[0077] S50, perform counterfactual reasoning verification on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships;

[0078] In this embodiment, counterfactual reasoning is performed to verify the causal relationships identified in the dynamic causal graph. The goal is to determine the stability and reliability of the causal relationships by simulating different intervention scenarios. Counterfactual reasoning is a method that verifies causal relationships by constructing hypothetical scenarios and comparing them with actual results. It not only relies on existing causal structures but also requires applying specific interventions to existing causal models.

[0079] First, the target causal relationship needs to be identified from the dynamic cause-effect graph. This causal relationship is typically represented by a directed edge between a cause node and an effect node. After identification, appropriate intervention parameters need to be selected. These parameters are used to simulate the control or modification of the characteristics of the cause node, such as numerical changes, state switching, or intensity adjustments. Intervening in the cause node will generate new feature representations, thereby changing the causal chain in the graph.

[0080] To ensure comprehensive verification, features of nodes that are not considered causes need to be extracted to form a non-causal set. These features are considered factors with little or no impact on causality. Simultaneously, nodes representing the external environment or scene context need to be identified and categorized into a context set. This contextual information ensures that counterfactual reasoning is conducted within a reasonable context.

[0081] After preparing the intervention features, non-causal set, and context set, the counterfactual inference function is used to generate simulation results. The counterfactual inference function operates based on the structural equations of the causal graph, outputting the distribution of possible outcomes under the assumed intervention conditions. Furthermore, by calculating the probability distribution of the post-intervention outcomes, statistical indicators such as entropy or kurtosis can be extracted as a reference for system stability.

[0082] To verify the sensitivity of the causal relationship, the intervention parameters need to be perturbed within a set range to obtain the perturbed feature inputs, and the counterfactual inference function is run again. The results under different intervention conditions are compared, and the relative rate of change is calculated to obtain the sensitivity coefficient. Combining the sensitivity coefficient with statistical stability indicators generates a reliability analysis value. Comparing the reliability analysis value with a preset threshold can be used to determine whether the causal relationship has sufficient reliability.

[0083] In medical and health settings, counterfactual verification can be performed to examine the causal relationship between patient lesion characteristics and a specific physiological indicator. First, lesion feature nodes and physiological indicator nodes are extracted from the causal graph as the target relationship. Intervention parameters are set to simulate controlled changes in the physiological indicator, such as simulating an increase in heart rate or a decrease in blood pressure. The remaining uninterrupted nodes are considered the non-causal set, and patient environmental characteristics in the medical setting (such as medication usage) are considered the context set. Simulation results are generated using a counterfactual inference function. The probability distribution of disease development under different intervention values ​​is compared, and the reliability of the causal relationship is determined by combining entropy and rate of change.

[0084] In fintech scenarios, counterfactual verification can be performed by selecting causal relationships between market news events and stock price events. By setting intervention parameters for news events (such as news intensity or sentiment direction), the characteristics of the causal nodes are adjusted in the causal graph. Other market indicators are treated as the non-causal set, and the macroeconomic environment as the context set. A counterfactual inference function is used to generate simulated price fluctuations, and the rate of price change before and after the disturbance is calculated to obtain a sensitivity coefficient. This coefficient is then combined with a stability index to determine the reliability of the causal relationship between news events and price fluctuations.

[0085] In different implementations, counterfactual inference functions can be based on different models. For example, they can use structural equation modeling to generate probability distributions or generative adversarial networks to simulate counterfactual outcomes under intervention. In terms of perturbation design, uniform or Gaussian perturbations can be used to adjust intervention parameters to adapt to the accuracy or robustness requirements of different scenarios.

[0086] This embodiment effectively eliminates spurious causal relationships caused by correlations by performing counterfactual reasoning verification on a dynamic causal graph, ensuring the reliability of the causal relationships retained in the graph. This process not only provides statistical stability of causal relationships but also reflects the robustness of the causal chain under different intervention conditions through a sensitivity coefficient, thereby enhancing the credibility of the causal inference results. This approach allows the causal graph to maintain the scientific validity and interpretability of causal relationships even after dynamic updates in complex environments.

[0087] S60, Generate a causal explanation result based on the dynamic causal graph and the reliability of the causal relationship.

[0088] In this embodiment, causal explanation results are generated based on the dynamic causal graph and the reliability of causal relationships. The goal is to combine the causal structure with the reliability analysis results to output interpretable reasoning conclusions. First, the causal chain of the target causal relationship needs to be extracted from the dynamic causal graph. The causal chain consists of multiple nodes and edges, representing entities and the causal connections between them. The extracted causal chain provides a structured path for explanation, clearly presenting the transmission relationship from cause to effect.

[0089] After obtaining the causal chain, it is necessary to calculate the partial derivatives of each node's features with respect to the prediction results generated by the causal inference engine. This step reveals the direct contribution of each node to the causal inference by quantifying the rate of change of each node in the result. Simultaneously, it is also necessary to perform importance analysis on node features by combining the update history of the dynamic causal graph. The update history records the changes in edge weights over time, reflecting the stability and influence of nodes in different time periods, thereby generating importance scores.

[0090] Combining partial derivatives with importance scores allows us to determine node contribution. Node contribution not only reflects the strength of a single feature's role in the current prediction but also incorporates stability over historical dimensions, providing a more comprehensive quantitative basis for causal explanations. Based on node contribution, high-contribution regions can be labeled in visual modal data, such as highlighting key areas in images using heatmaps; high-contribution phrases can also be extracted from text modal data to highlight important semantic information relevant to the prediction.

[0091] To further enhance the credibility of the explanation results, it is necessary to combine the reliability analysis value of the causal relationship with the accuracy of different modalities in historical inference, and dynamically allocate the weight of each modality. Then, the prediction accuracy of each modality is determined in the current inference task, and a confidence score is generated by weighting the weights with the accuracy. The confidence score is a quantitative indicator of the overall reliability of the causal explanation, reflecting the credibility level of the final explanation.

[0092] During processing, it is also necessary to detect whether there are missing modalities in the input raw data stream. For example, when image modalities are missing, virtual features can be generated through transfer learning using cross-modal fusion features to compensate for the impact of missing modalities. The confidence score is adjusted based on the virtual features to ensure that the explanatory results remain usable even with incomplete data. Finally, high-contribution regions, key phrases, and adjusted confidence scores are integrated to output the final causal explanation.

[0093] In medical and health scenarios, causal chains between disease features and treatment outcomes can be extracted from dynamic causal graphs. The partial derivatives of lesion features with respect to recovery rates are calculated using prediction results, and importance scores for features at different stages are derived by combining update history, ultimately forming the contribution of lesion nodes. High-contribution regions, such as highlighting tumor sites, are annotated in the medical imaging modality, while key phrases, such as core symptoms in medical record descriptions, are extracted in the text modality. The weights of the image and text modalities are dynamically adjusted based on the reliability of causal relationships and historical prediction accuracy, and confidence scores are calculated. When the image modality is missing, virtual image features are generated using the fusion features of the text modality to correct the confidence score, ensuring that the causal explanation results still reliably support clinical diagnosis.

[0094] In fintech scenarios, causal chains between market news, trading indicators, and stock price fluctuations can be extracted. Partial derivatives between news intensity and stock price changes are calculated based on prediction results, and importance scores for different economic indicators are obtained by combining historical data, forming node contribution scores. High-impact phrases, such as "policy tightening" and "interest rate hikes," are extracted from news texts and marked as high-contribution areas on market curves, such as abnormal fluctuations in trading volume. The weights of different modalities are adjusted based on the reliability of causal relationships and the accuracy of historical data, and confidence scores are generated. When trading indicators are missing, virtual trading features are generated using news and macroeconomic characteristics to compensate, ensuring that the final causal explanation remains credible in investment decisions.

[0095] In different implementations, weight allocation can be achieved through dynamic Bayesian updates or through a regression model trained on historical samples. When generating confidence scores, a simple weighting method can be used, or a weighted average and variance correction method can be introduced to enhance stability. The virtual feature generation stage can be implemented using a ransformer-based cross-modal transfer model or by using a variational autoencoder to generate modality-compensated features.

[0096] This embodiment combines dynamic causal graphs, prediction results, and reliability analysis to output causal explanations that include both structured causal chains and numerical confidence levels. This approach maintains the transparency of causal reasoning while enhancing the credibility and operability of the explanation. When modalities are missing, the introduction of virtual features ensures the continuity of the causal explanation under incomplete data conditions, thereby avoiding explanation failure due to missing information.

[0097] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a multimodal causal reasoning and explanation method, apparatus, device, and medium, comprising: acquiring raw data streams from at least two different modalities and extracting features to generate modal features; fusing the modal features through a cross-modal attention mechanism and refining causal features through a feature distillation process; constructing a dynamic causal graph containing entity nodes and causal edges based on the causal features, and updating the edges using a causal strength function; identifying causal relationships in the dynamic causal graph and performing counterfactual reasoning verification to analyze the reliability of the causal relationships; and generating causal explanation results based on the dynamic causal graph and the reliability of the causal relationships. This invention introduces a causal feature extraction mechanism during the feature extraction and cross-modal fusion process of multimodal data to model causal associations between different modalities; dynamically updates the causal graph structure through a causal strength function to ensure the temporal accuracy and stability of causal relationships; improves the reliability assessment of causal relationships through counterfactual reasoning verification; and outputs causal explanation results based on this. This invention overcomes the limitations of existing technologies that rely solely on a single modality or simple feature fusion, significantly improving the accuracy and interpretability of causal reasoning in complex multimodal scenarios.

[0098] In one embodiment, step S10 includes:

[0099] S101, receiving raw data streams from at least two independent data acquisition devices;

[0100] S102, Extract the timestamp information of the original data stream;

[0101] S103, Based on the timestamp information, align the original data streams of different modalities to generate a time-aligned original data stream;

[0102] S104, Detect the data integrity of the time-aligned original data stream;

[0103] S105, discard the original data stream with data integrity below the preset threshold, and obtain the valid original data stream;

[0104] S106, encapsulate the valid raw data stream to generate a multimodal data packet.

[0105] In this embodiment, when receiving raw data streams from at least two independent data acquisition devices, a persistent input channel and a unique device identifier are established for each independent data acquisition device. The channel can be a stateful long connection or a controlled buffered message queue, supporting out-of-order and packet loss retransmission control. The raw data stream refers to a continuous data sequence that has not undergone feature extraction and modality fusion, containing payload and minimized header information. The payload carries content such as visual modality pixel frames, text modality character sequences, or audio modality waveform fragments. The header information includes at least a device identifier, sampling frequency or frame rate, timestamp field, and checksum field. To eliminate access jitter and out-of-order issues caused by network transmission, the input channel is configured with a receive buffer and watermark strategy. Data fragments are stacked and rearranged according to the timestamp field and sequence number, and an early out-of-order fragment that has exceeded the waiting window is discarded using a window discard strategy. Independent data acquisition devices can be cameras and microphones, vital sign monitoring sensors and electronic medical record text acquisition components, or market data matching gateways and news text stream interfaces, all of which meet the requirements of independent acquisition and independent time sources.

[0106] When extracting timestamp information from the raw data stream, a unified time base and time domain semantics are used. If the timestamp is marked by the device hardware, the device-side timestamp is used as the primary time, and the offset and drift with the system standard time are estimated. If the timestamp is generated by the access end, a monotonic clock is used, and the arrival time is recorded as an alignment candidate. Offset estimation is completed through round-trip delay measurement, hardware time synchronization protocol, or cross-correlation alignment based on a sliding window, resulting in a device-level clock mapping function. The mapped standard time is recorded with a unified time zone and unified epoch. The timestamp information includes the time point, precision level, and confidence weight, and is written into the stream metadata for subsequent alignment and quality judgment calls.

[0107] When aligning raw data streams of different modalities based on timestamp information and generating time-aligned raw data streams, a common time grid is first constructed. The granularity of the time grid is determined based on the highest sampling frequency of the participating modalities and the desired fusion delay. For discrete frame modalities, nearest neighbor or maximum similarity mapping within a window is used to map frames to the nearest grid point, and missing frames are marked if the tolerance deviation threshold is exceeded. For continuous waveform or numerical sensing modalities, interpolation or resampling is used to reproject samples onto the common grid, avoiding duplicate counting and energy leakage across grids. Multi-rate alignment is achieved through dual control of modal weights and tolerance deviations. High-speed modes can be downsampled for alignment, while low-speed modes prioritize maintaining the original sampling interval, padding with zeros or retaining the previous valid value when necessary. The time-aligned raw data stream retains the multi-modal synchronization slices and the time consistency markers of the slices at each grid point for subsequent integrity determination.

[0108] When assessing the data integrity of the time-aligned raw data stream, a two-tiered metric system is established, oriented towards slices and segments. At the slice level, coverage, temporal consistency deviation, load validation pass rate, and outlier ratio are calculated. At the segment level, continuous gap length, effective slice ratio, and temporal drift residual are calculated. Coverage is calculated for a single grid window as the ratio of modes with effective loads to the target number of modes. Temporal consistency deviation measures the difference between the actual alignment time of each mode and the grid time. Load validation pass rate verifies load integrity based on validation fields. Outlier ratio is determined based on range constraints, spectral energy distribution, or text character validity checks. Segment-level continuous gap length characterizes long periods of data loss, and temporal drift residual captures systematic deviations that persist after alignment. All metrics are summarized into a data integrity score, accompanied by labels for each mode and the global scope.

[0109] When discarding raw data streams with integrity below a preset threshold and obtaining valid raw data streams, the preset threshold is defined at both modal and global granularities, and is either statically distributed via configuration files or pre-set by historical calibration statistics. The decision logic employs a two-level decision-making process: first modal, then global. If the slice coverage or temporal consistency deviation of a critical modality does not meet the preset threshold, the slice is marked as invalid and not included in the valid raw data stream. If the length of a continuous gap at the segment level exceeds the preset threshold, the entire segment is discarded or split, retaining sub-segments that meet the threshold. The discard operation simultaneously updates the integrity statistics to avoid double counting at the boundary window, ensuring that the obtained valid raw data stream meets the minimum reliable input standard in both the time grid and modal dimensions.

[0110] When encapsulating valid raw data streams to generate multimodal data packets, a container and index structure are defined. The container contains a global header, modal segments, alignment indexes, and checksum tails. The global header records the time grid definition, time base information, version number, and a list of participating modalities. The modal segments are payload areas fragmented by modality, with each fragment carrying the fragment time, fragment length, and payload fingerprint. The alignment index provides a mapping from timestamps to fragment offsets for random location. The checksum tail contains the full packet signature and the cumulative checksum for each modality to support fast consistency verification upon arrival. The multimodal data packet includes a data integrity summary and a drop log, facilitating the downstream reproduction of alignment and filtering trajectories during the loading phase. To reduce subsequent processing latency, the container allows segmented commits and pipelining writes, ensuring that packet segments can be output at the window level while maintaining temporal order consistency.

[0111] This embodiment extracts timestamp information using a unified time base and generates a time-aligned raw data stream on a common time grid. This explicitly eliminates phase and sampling differences between modal signals, and converges the maximum cross-correlation value from non-zero latency to near-zero latency. Subsequent fusion and causal strength estimation avoid spurious associations introduced by time mismatches. By employing a two-layer data integrity metric for slices and segments and adjudicating based on preset thresholds, low-quality segments are isolated before entering downstream computation, preventing the propagation of abnormal payload energy and textual noise to subsequent models and minimizing perturbations to feature distribution and causal structure learning. By encapsulating the effective raw data stream into a multimodal data packet using containers and indexes, time alignment mapping, integrity summaries, and verification information are stored in the same carrier. Subsequent modules can access this data randomly and load it reproducibly, significantly reducing recalculation costs and synchronization overhead, and establishing a stable input baseline from acquisition to analysis.

[0112] In one embodiment, step S20 above includes:

[0113] S201, Perform data structure analysis on each type of raw data stream in the multimodal data;

[0114] S202, For the original data stream with spatial dimension, perform convolution operation to extract spatial features, and apply a position encoding function to process the spatial features to generate spatial features with temporal information;

[0115] S203, use a self-attention mechanism to process the spatial features with temporal information to generate spatial self-attention features;

[0116] S204, combine the spatial features and spatial self-attention features to generate spatial modal features;

[0117] S205, For a raw data stream with a sequence structure, a pre-trained language model is used to process the raw data stream to generate initial semantic features, and a weight matrix is ​​applied to process the initial semantic features to generate sequence modal features;

[0118] S206, Perform format standardization processing on the spatial modal features and sequence modal features to generate standardized modal features;

[0119] S207, add feature dimension description information to the standardized modal features, and use the standardized modal features as modal features corresponding to each original data stream.

[0120] In this embodiment, when performing data structure analysis on each type of raw data stream, a clear data meta-model is established for the input data type and organization. Visual raw data streams use frames as the basic unit, recording attributes such as height, width, number of channels, bit depth, frame rate, and color space. Text raw data streams use symbol sequences as the basic unit, recording encoding methods, word segmentation or character segmentation strategies, maximum sequence length, and special symbol tables. Audio or physiological signal sequence numerical data record sampling rate, quantization precision, channel configuration, and segment length. The analysis process simultaneously generates missing masks and outlier markers for subsequent operator selection and parameter constraints. The results of the data structure analysis are returned as a unified descriptive object, including shape information, temporal granularity, effective location information, and a mask matrix, facilitating direct reference by cross-modal processing units.

[0121] For raw data streams with spatial dimensions, when performing convolution operations to extract spatial features, each frame or slice is first converted into a normalized tensor and the channel order is aligned. Two-dimensional images use two-dimensional convolution operators, while videos use two-dimensional or three-dimensional convolution operators that slide along the temporal dimension. The kernel size, stride, and padding strategy are automatically selected based on the target resolution and computational budget obtained from data structure analysis. Common configurations include a 3x3 convolution kernel and a one-pixel stride to capture local edges and texture patterns, and interlayer residual connections and normalization layers are stacked to stabilize gradients. The obtained spatial features are stored in the form of feature maps, containing the channel dimension and downsampled spatial coordinates.

[0122] When applying positional encoding functions to process spatial features to generate spatial features with temporal information, the time index is mapped to a vector and fused with the spatial features. The time index can be a frame number or a synchronous time grid number, and the positional encoding function can be a learnable embedding or a family of sine and cosine functions. Sine and cosine encoding projects scalar time onto a multi-dimensional vector using trigonometric functions of different frequencies while maintaining cross-scale discriminability; learnable embedding directly maps through a parameter matrix and adaptively adjusts during training. The fusion method can use vector addition for translational equivariance, or feature concatenation followed by a linear transformation to control the number of channels. The processing result carries temporal clues in the channel dimension, forming spatial features with temporal information.

[0123] When using a self-attention mechanism to process spatial features with temporal information to generate spatial self-attention features, the spatial features are first divided into several region tokens or a token sequence is formed using stride sampling. Then, three sets of representations—query, key, and value—are obtained through linear mapping. The attention weights are obtained by scaling the dot product of the query and key and then exponentially normalizing them. The weights are then multiplied by the value and accumulated in the token dimension to form a new region representation. A multi-head configuration is used to capture long-range dependencies and cross-regional interactions in parallel across different subspaces, while a feedforward network and residual normalization are used to stabilize the distribution and improve the representation of nonlinearity. The output spatial self-attention features retain the original spatial and temporal indices, facilitating alignment with earlier convolutional features.

[0124] When combining spatial features and spatial self-attention features to generate spatial modal features, a cross-representation fusion operator is introduced. If the target aims to preserve complementary information between local texture and global structure, channel-dimensional concatenation followed by a one-to-one convolution or linear layer can be used to compress channels and generate a unified dimension. If the target aims to suppress redundancy, point-wise weighted summation can be used, with weights generated adaptively by channel attention gating or statistical measures. The combined result is output as a fixed-dimensional tensor with a mask of spatial coordinates and temporal indices, representing the set of effective locations.

[0125] For raw data streams with sequential structures, when using a pre-trained language model to process the sequence and generate initial semantic features, word segmentation or sub-word splitting and special symbol padding are first performed based on the data structure analysis results. Then, discrete symbols are mapped to embedding vectors and positional embeddings are superimposed. The pre-trained language model can be an autoregressive or autoencoder architecture, with the encoder outputting context-dependent semantic vectors at each position. To control the representation dimension and adapt to the subsequent unified interface, a weight matrix is ​​applied to linearly map or project the initial semantic features. The weight matrix size is consistent with the target dimension and can be shared or set independently layer by layer. The mapped sequence modal features retain the original positional mask for alignment and pooling.

[0126] When standardizing spatial and sequential modal features to generate standardized modal features, the data type, tensor layout, main dimension order, and padding strategy are unified. The layout is standardized to a four-dimensional order of batch, time, and location or token and channel. Missing positions are marked with mask bits and filled with neutral values. The numerical range is robustly scaled to the target interval to mitigate cross-modal dimensional differences, and the time dimension length is pruned or padded to the common time grid length to ensure cross-sample parallelism. The standardized output is registered in the interface layer, supporting modality-based retrieval and random access.

[0127] When adding feature dimension description information to standardized modal features and outputting it as the modal feature for each original data stream, a structured metadata object is created to record the dimension meaning, unit, sampling granularity, coordinate system, position mask, time mask, and source identifier. Spatial modality records the spatial step size and receptive field estimation, while sequence modality records the vocabulary coverage and truncation ratio. Metadata coexists with tensors and is bound by immutable references, allowing any subsequent processing to perform automatic alignment, mask propagation, and dimension verification based on this description. The output interface exposes both iterator and random index access capabilities, facilitating on-demand access by upper-layer fusion and distillation units.

[0128] This embodiment transforms raw data streams from different sources into descriptive objects with consistent semantics through data structure analysis. The combination of convolution operators and positional encoding simultaneously establishes learnable local and temporal representations in both spatial and temporal dimensions. Self-attention supplements cross-regional dependencies and improves the distinguishability of long-distance associations globally. The fusion of spatial features and self-attention features injects global context while preserving detailed textures. Pre-trained language models and weight matrix projections provide semantic representations of sequence signals that can be aligned with spatial signals. Format standardization and dimensional description information ensure that multi-source results maintain a unified shape and clear meta-semantics at the interface layer.

[0129] In one embodiment, step S30 above includes:

[0130] S301, the cross-modal fusion module receives first modal features and second modal features from the feature extraction and processing module;

[0131] S302, determine the information entropy of the first modality feature and the information entropy of the second modality feature;

[0132] S303, determine the joint information entropy of the first modal feature and the second modal feature;

[0133] S304, determine mutual information entropy based on the information entropy and joint information entropy;

[0134] S305, determine the similarity between the first modal feature and the second modal feature;

[0135] S306, Generate adjustment coefficients based on the mutual information entropy and similarity;

[0136] S307, The adjustment coefficient is processed by an exponential function to generate an intermediate weight value, and the intermediate weight value is processed by a normalization function to generate a fusion weight.

[0137] S308, the first modal feature and the second modal feature are weighted and summed using the fusion weights to generate fusion features;

[0138] S309, The fused features are processed by the upper-layer Transformer model of the input feature distillation to generate the initial fused output;

[0139] S310, the initial fusion output is passed to the lower-level causal inference engine, and causal features and prediction results are generated in the lower-level causal inference engine;

[0140] S311, in the upper-layer Transformer model, the difference between the initial fusion output and the causal feature is compared using an error function;

[0141] S312, minimize the error function to optimize the initial fusion output, and use the optimized initial fusion output as the extracted causal feature output.

[0142] In this embodiment, when the cross-modal fusion module receives the first modal features and the second modal features from the feature extraction and processing module, it reads the tensor and its dimensional description using a standardized interface. The two sets of inputs need to be aligned on the time grid and the effective location mask, the channel dimensions are unified to the target dimension, and shape consistency is verified through metadata. To ensure statistical comparability, the two sets of features are first subjected to numerical range standardization and detrending processing, and outliers are robustly truncated and suppressed.

[0143] When determining the information entropy of the first and second modal features, differential entropy estimation is used for continuous representations. The k-nearest neighbor estimator can be used to measure the local volume in the embedding space and summarize the entropy value, or the kernel density can be estimated in the subspace and then calculated by numerical integration. For sequential representations, window aggregation and principal component projection can be performed first to reduce the dimensionality before differential entropy estimation. The information entropy is returned as a scalar, along with a confidence interval as the estimation uncertainty.

[0144] When determining the joint information entropy, the two sets of features are concatenated into a joint variable, and the differenced joint entropy is calculated using an estimator consistent with that of the unimodal model. To reduce the bias caused by dimensionality expansion after concatenation, a stochastic feature map or a learnable low-rank projection is used to compress the joint representation to a controlled dimension while preserving the cross-correlation structure. A shared random seed and bandwidth selection strategy is used during entropy estimation to avoid systematic bias introduced by differences in estimator configuration.

[0145] When determining mutual information entropy based on information entropy and joint information entropy, algebraic synthesis is performed according to the definition of mutual information. To mitigate finite sample bias, a mutual information neural estimator can be introduced as a correction term, and the analytical mutual information and learned mutual information are weighted and fused, with the weights determined by the principle of minimizing validation set bias. Mutual information entropy, as a non-negative scalar, provides a cross-modal statistical dependency measure.

[0146] When determining the similarity between two sets of features, the cosine similarity after mean pooling is calculated for global alignment requirements, and the centered kernel alignment coefficient is calculated for structural alignment requirements. For time-dependent scenarios, time-by-time similarity measurements are performed on a unified time grid, and a weighted average is calculated. The time weights are derived from the effective location mask and data confidence. The similarity output is limited to the interval between zero and one, facilitating its participation in subsequent weight generation along with mutual information entropy.

[0147] When generating adjustment coefficients based on mutual information entropy and similarity, a monotonic mapping function is constructed, combining the two into a fusion strength index. A linear combination with a trainable scale and offset parameter can be used, or a multiplicative combination can be employed to emphasize the case where both values ​​are high. To enhance robustness, a temperature parameter is introduced to control the gradient steepness; the temperature is adaptively updated during training. The adjustment coefficients maintain a scalar or time-step vector form consistent with the batch size.

[0148] When applying exponential functions to adjust coefficients to generate intermediate weight values, an exponential mapping is used to map any real number to the positive domain, avoiding directional reversal caused by negative weights. To prevent numerical overflow, the maximum value of the batch is subtracted before exponentialization. Subsequently, a normalization function is applied to normalize the candidate weight set; a typical implementation is Softmax to obtain the fused weights. Normalization supports temperature adjustment and mask propagation, ensuring that missing positions are not included in the normalization denominator calculation.

[0149] When using fusion weights to weight and sum the two sets of features to generate fused features, a linear combination is performed point-by-point in both the temporal and positional dimensions. The weights can be either global scalars or position-by-position weight maps. To avoid scale drift, a normalization layer and a gating unit are concatenated after the linear combination. The gating parameters are extracted from the pre-fusion features through a lightweight feedforward network. The fused features retain the input temporal index and effective position mask for direct consumption by subsequent networks.

[0150] When inputting the fused features into the upper Transformer model and generating the initial fused output, positional encoding is first injected, and information exchange across positions and time is achieved through multi-head self-attention and stacking of feedforward networks. Residual connections and normalization layers are used to stabilize training, and the dropout rate and hidden dimension are set according to the computational budget and the length of the target sequence. To suppress noise propagation, a learnable gate can be added to the input side to attenuate the contribution of low-confidence positions. The initial fused output serves as an observable intermediate representation for the upper layer, while a classification or regression branch is derived to obtain prediction results for downstream use.

[0151] When the initial fusion output is passed to the lower-level causal inference engine to generate causal features and prediction results, structural equation solving and causal filtering are implemented through fixed or gradually varying structured modules. This engine can contain differentiable causal structure layers, intervention gates, and anti-confounding suppression units. The input is the initial fusion output and necessary context description; the output is causal features that are insensitive to intervention but sensitive to causal paths, while also producing a task-related distribution of prediction results. To prevent gradients from misleading the causal structure backflow, gradient stopping or semi-coupled distillation channels are enabled between the upstream and downstream layers.

[0152] In the upper-level Transformer model, when comparing the differences between the initial fused output and causal features using an error function, a distance metric consistent with the target representation is selected. This can be achieved by using mean squared error to align the continuous representation, or by using distribution-matched KL divergence or contrastive learning loss to maximize the upper bound of mutual information. To prevent representation collapse, a regularization term is introduced to constrain the variance and covariance matrices of the initial fused output, ensuring they maintain sufficient representation rank.

[0153] When minimizing the error function to optimize the initial fusion output and using the optimized initial fusion output as the extracted causal feature output, an adaptive optimization algorithm is used to update the parameters of the upper-layer model. The learning rate is controlled by warm-up and cosine annealing strategies. During distillation, the mask is propagated according to time and location to ensure that the loss is calculated only in the effective region. After training, the optimized output produced by the upper-layer model can be directly used as the extracted causal features during the inference phase, without needing to keep the lower-layer engine connected, thereby reducing latency and energy consumption. If the operating environment allows, the lower-layer engine can still be enabled for online calibration to resist distribution drift.

[0154] This embodiment introduces interpretable statistical and geometric criteria into the weight generation stage by jointly measuring mutual information entropy and similarity. Exponential mapping and normalization ensure that the weights are non-negative and comparable. The fusion calculation achieves fine-grained adaptation in both time and space. The weighted fusion features suppress redundancy while maintaining the integrity of multi-source information. The upper-layer Transformer and the lower-layer causal inference engine form a distillation relationship. The error function drives the upper-layer expression to converge towards causal invariance. The optimized output is more robust to intervention disturbances and more resistant to interference from confounding factors.

[0155] In one embodiment, step S40 above includes:

[0156] S401, Extract multimodal entity information based on the causal features to form a node set;

[0157] S402, Based on the causal relationships between nodes in the node set, construct an edge set connecting the nodes;

[0158] S403, for each edge in the edge set, determine the transfer entropy between the two node features connected by the edge;

[0159] S404, determine the conditional mutual information between the features of the two nodes connected by the edge;

[0160] S405, use a preset balance coefficient to perform a weighted summation of the transit entropy and conditional mutual information to generate a causal strength value;

[0161] S406, Apply an activation function to process the causal strength value and generate updated edge weights;

[0162] S407, Replace the original edge weight of each corresponding edge with the updated edge weight to obtain the updated dynamic causal graph;

[0163] S408, In the subsequent update process, the changes of the edge weights over time are recorded and a time series is formed according to the timestamps to obtain the update history of the dynamic causal graph.

[0164] In this embodiment, when extracting multimodal entity information based on causal features and forming a node set, entity candidate identification and deduplication merging are first performed on the time-aligned causal feature sequence. Visual sources can complete cross-frame association of the same entity through the consistency measure of target trajectory and instance embedding; speech or text sources can complete cross-time merging through referential resolution and named entity linking; sensor sources can complete aggregation through unique device identifiers and measurement similarity. Each node stores entity identifier, valid time interval, source modality list, aggregated causal feature vector, and confidence interval, while maintaining reversible references to the original data stream to support subsequent interpretable output. To avoid duplicate entity inflation, density clustering or minimum cost matching is used to perform entity merging within adjacent time windows, and the threshold is set according to the optimal principle of validation set stability.

[0165] When constructing the edge set connecting nodes based on the causal relationships between nodes in the node set, candidate directed pairs are first enumerated within the lag time window, and pairs that cannot be causally related in time are eliminated. For each candidate pair, fast correlation priors and Granger priors are calculated to narrow the computational scope, followed by a rigorous causality test phase. The edge object includes the start and end nodes, the lag step size, a list of candidate time slices, and the initialized edge weights. To reduce the estimation variance caused by graph density, sparse priors or maximum in-degree constraints can be introduced, prioritizing candidate edges with higher information contributions. The edge set is stored using a dual structure of adjacency list and compressed sparse rows, supporting incremental updates and random access by time slices.

[0166] When determining the transfer entropy between the features of the two nodes connected by each edge in the edge set, a lag embedding is constructed on a unified time grid and quantized using a model-free estimator. The conditional probability density and transfer entropy can be estimated in the joint embedding space using the k-nearest neighbor method, or robustness can be improved using an approximation formula based on mutual information and autoregressive residual correction. The lag step size is determined by scanning the candidate set and maximizing the information gain of the validation set; the time window length is jointly determined by the stationarity test and the autocorrelation cutoff point. To suppress finite sample bias, a resampling confidence band and a one-to-one deletion correction are used, and a mask is introduced for anomalous periods to avoid affecting the global estimation.

[0167] When determining the conditional mutual information between the features of the two nodes connected by an edge, a minimal set of confounding control variables is introduced to eliminate interference from common causes. The control set is automatically selected from the adjacency structure and domain context, and is calculated using a k-nearest neighbor conditional mutual information estimator or a kernel density-based conditional mutual information estimator. If the dimensionality of the control variables is high, they can be compressed first using an invertible dimensionality reduction mapping before estimation. The conditional mutual information is calculated at each candidate lag and cached synchronously with the transitive entropy. The output value includes a variance estimate to facilitate confidence weighting in subsequent weight fusion.

[0168] When using preset balance coefficients to weight and sum transitive entropy and conditional mutual information to generate causal strength values, the two types of measures are first scaled to ensure comparability. The balance coefficients are taken from offline calibration or online adaptive updates. Online updates can use exponential moving averages to track the relative discriminative power of the two measures under the current data distribution. The causal strength values ​​obtained after weighted summation can be exponentially smoothed over time to reduce instantaneous noise while retaining rapidly changing side signals for dynamic response. For periods with large variance, inverse weighting can be applied according to the uncertainty of the measures to reduce the contribution of estimates with low confidence.

[0169] When applying activation functions to process causal strength values ​​to generate updated edge weights, if the target is a probabilistic weight, a sigmoid mapping can be used to map it to the zero-to-one interval; if the target is a relative weight, softmax normalization can be performed on the input edge set. To avoid numerical instability, the input is centered and temperature-scaled, and a mask is used for missing time periods to exclude invalid terms. In non-stationary sequences, a time decay term can be added to gradually reduce the influence of long-term estimates on the current weights. The final updated edge weights are stored in the adjacency list in floating-point format with a timestamp indicating the latest write time.

[0170] When replacing the original edge weights of each corresponding edge with the updated edge weights to obtain the updated dynamic causal graph, batch writes are first performed in the transaction buffer, and then atomic operations are used to commit to the global graph structure, ensuring the consistency of concurrent updates. To ensure topology availability, graph validity checks are performed before commit, including self-loop removal, in-degree pruning, and weak connectivity guarantees. After commit, a topology version number is generated and broadcast to dependent modules, facilitating downstream counterfactual reasoning and interpretation to run on a consistent graph version.

[0171] During subsequent updates, when recording the changes in edge weights over time and forming a time series by timestamp to obtain the update history, a circular buffer or time series database entry is maintained for each edge, recording the timestamp, the updated edge weight, the estimated uncertainty, and the data range involved in the estimation. To save storage and improve retrieval efficiency, a piecewise linear compression and key point retention strategy is introduced, and significant change points and turning points are forcibly retained. The update history supports window retrieval and aggregate queries, providing data basis for credibility calibration in the subsequent importance score calculation and interpretation stages.

[0172] This embodiment characterizes directionality and controlled associations through complementary estimations of transfer entropy and conditional mutual information, while maintaining stable discriminative power under different data distributions and noise levels using balance coefficients and uncertainty weighting. Activation and normalization steps unify heterogeneous metrics into a usable weight space. The graph structure is atomically updated in a transactional manner with version management, ensuring a consistent dynamic causal graph across different downstream stages. Continuous recording of edge weight time series provides longitudinal evidence for subsequent interpretation and reliability analysis, identifying the emergence, strengthening, and decay trends of causal relationships, thereby maintaining the traceability and maintainability of causal modeling in changing environments.

[0173] In one embodiment, step S50 above includes:

[0174] S501, Identify the target causal relationship from the dynamic causal graph;

[0175] S502, using preset intervention processing parameters to intervene in the causal features in the target causal relationship, and generating the intervened features;

[0176] S503, extract node features that are unrelated to the causal features from the node set of the dynamic causal graph to form a non-causal feature set;

[0177] S504, extract scene background node features from the node set of the dynamic cause-effect graph to form a context feature set;

[0178] S505, use the counterfactual inference function to process the set of features, non-causal features, and contextual features after the intervention, and generate counterfactual simulation results;

[0179] S506, Based on the structural equation of the dynamic causal graph, generate the probability distribution after intervention;

[0180] S507, Determine the ratio of kurtosis to entropy of the probability distribution as a statistical stability index;

[0181] S508, Adjust the intervention processing parameters within a preset disturbance range, and generate the disturbance-induced intervention features based on the adjusted intervention processing parameters;

[0182] S509, The counterfactual inference function is used to process the perturbed intervention features, non-causal feature set and context feature set to generate the perturbed counterfactual simulation result;

[0183] S510, determine the relative rate of change between the counterfactual simulation result and the perturbed counterfactual simulation result, and generate a sensitivity coefficient;

[0184] S511, Generate a reliability score based on the statistical stability index and sensitivity coefficient;

[0185] S512, when the reliability score exceeds a preset threshold, the target causal relationship is determined to be reliable.

[0186] In this embodiment, when identifying the target causal relationship from the dynamic causal graph, a directed edge X→Y needs to be selected as the verification object on the latest version of the graph structure. This step typically limits the time window and lag step size, and verifies the persistence of the edge in the update history and the confidence interval of the edge weight, while eliminating self-loops and candidate relationships that do not conform to the causal time sequence. To ensure the reproducibility of the entire verification process, the graph version number, window boundaries, lag set, and sampling strategy should be recorded and used as input metadata for subsequent calculations.

[0187] When intervening in causal features, a common approach is to replace or transform the representation value of node X using preset intervention parameters, thereby generating the intervened feature X'. Intervention methods can be categorized into three types: intensity adjustment, masking replacement, and additive perturbation.

[0188] Intensity adjustment: X' = α·X or X' = x * Where α is the scaling factor, x * This is the fixed value after the replacement.

[0189] Masking replacement: X' = M⊙X; ​​where M is the Bernoulli mask matrix, used to randomly mask some components.

[0190] Additive perturbation: X'=X+ε,ε~N(0,σ) 2I) The parameter range is usually constrained by prior knowledge, for example, α∈[0.5,1.5], the masking ratio range is [0,0.3], and the upper limit of the variance parameter σ is σ max .

[0191] When extracting the non-causal feature set, it is necessary to remove the causal node X and its descendant nodes of the target causal relationship from the node set of the causal graph, retaining only the set of nodes that are not directly related to the causal path of the result Y, and combining this with the filtering rules of conditional mutual information:

[0192] I(X;Z∣Y)<δ

[0193] Where Z represents a candidate node, and δ is a preset threshold. Nodes that meet this condition are assigned to the non-causal feature set.

[0194] When extracting the context feature set, it is necessary to select scene-related background variables from the causal graph node set, such as timestamp encoding, geographic location, device status, or environmental labels, to form the context set F. context When the context dimension is high, it can be compressed into a latent vector using dimensionality reduction methods such as autoencoders to ensure that the input dimension is stable and the main semantics are preserved.

[0195] Counterfactual inference functions in handling X', With F context At that time, based on structural equations:

[0196] Y = g(Pa(Y), U) Y )

[0197] Where Pa(Y) is the set of parent nodes of the result node Y, U Y The noise term is used. During runtime, the original value in Pa(Y) is replaced by the intervened X', while keeping the inputs of other nodes unchanged. The noise term is then sampled to finally generate the counterfactual simulation result Y. cf .

[0198] When calculating the probability distribution after intervention, based on the sample set The distribution can be obtained through kernel density estimation or maximum likelihood fitting:

[0199] P(Y∣do(X=x'))

[0200] The statistical stability of a distribution can be characterized by the ratio of kurtosis to entropy:

[0201]

[0202] When adjusting the intervention parameters, let the original parameter be θ, and generate it within the perturbation range [-∈, ∈].

[0203] θ'=θ+Δθ,Δθ∈[-∈,∈]

[0204] The new intervention feature X'(θ') is obtained. The perturbed simulation result Y is obtained through the counterfactual inference function. cf (θ'), and calculate the sensitivity coefficient:

[0205]

[0206] Finally, the reliability score is obtained by combining stability and sensitivity:

[0207] R = Stability × e -s

[0208] When R > T (threshold), the determination of the target causal relationship is reliable.

[0209] This embodiment, by introducing intervention computation, perturbation analysis, and counterfactual simulation onto the causal graph, enables the causal relationship verification process to possess quantitative standards, reproducibility, and cross-scenario adaptability. The statistical stability index reflects the convergence of the distribution and the level of uncertainty, while the sensitivity coefficient characterizes the model's robustness to intervention perturbations. The reliability score generated by combining these two metrics avoids the one-sidedness of a single indicator, improving the accuracy and credibility of causal relationship verification. Even in scenarios with data noise, distribution drift, or insufficient sampling, this method can still provide stable verification results, thus offering stronger reliability support for causal explanations and decision-making.

[0210] In one embodiment, step S60 above includes:

[0211] S601, Extract the causal chain of the target causal relationship from the dynamic causal graph;

[0212] S602, Based on the prediction results generated by the causal inference engine, determine the partial derivatives of the node features in the causal chain with respect to the prediction results;

[0213] S603, based on the update history of the dynamic cause-effect graph, determines the importance score of node features;

[0214] S604, determine the node contribution by combining the partial derivatives and importance scores;

[0215] S605, on the visual modal data of the original data stream, mark high contribution regions based on the node contribution.

[0216] S606, on the text modal data of the original data stream, extract high-contribution phrases based on the node contribution;

[0217] S607 dynamically assigns weights to each modality based on its inference accuracy and causal reliability score in historical data.

[0218] S608, determine the prediction accuracy of each modality in the current inference task;

[0219] S609, combine the weights and prediction accuracy to generate a confidence score;

[0220] S610, detects whether there is a mode missing in the raw data stream;

[0221] S611, when there is a lack of modality in the original data stream, virtual features are generated by using the fused features after cross-modal fusion through a transfer learning function;

[0222] S612, Adjust the confidence score according to the virtual features;

[0223] S613, integrate the high-contribution region annotations, high-contribution phrases, and adjusted confidence scores to generate causal explanation results.

[0224] In this embodiment, when generating causal explanation results based on the dynamic causal graph and the reliability score of causal relationships, a set of causal chains consistent with the target causal relationship is first extracted from the dynamic causal graph. During extraction, directed acyclic constraints and time consistency constraints are imposed, prioritizing paths with higher edge weights and stability in the update history. Paths are then sorted using a path scoring function, which can be defined as the inverse product of the weighted geometric mean of edge weights and the variance of edge weights, ensuring both strong causal relationships and time stability are satisfied. After obtaining the causal chains, the features of each node in the chain are gradient-attributed based on the prediction results of the causal inference engine, and the partial derivatives of the node features with respect to the prediction results are calculated. in F is the prediction output of the causal reasoning engine for the current task. i Let be the input feature tensor of node i. Gradient attribution is used to reflect the marginal impact of small perturbations on the prediction, avoiding the neglect of directionality by relying solely on statistical correlation.

[0225] Node importance scores are extracted from the update history of the dynamic causal graph. The historical record consists of edge weight time series indexed by timestamps and the number of times a node participates. A weighted moving average strength s is calculated for the set of edges ∈(i) adjacent to node i. i =∑ e∈ε(i) ω e ·EMA(w e ), where ω e For edge-weighted time series, EMA is the exponentially weighted moving average, ω e The ownership of edges to nodes is typically set as a normalized function of the node's in-degree and out-degree. Importance scores characterize long-term structural contributions, avoiding reliance on random fluctuations from single inferences. The node contribution score *c* is obtained by fusing immediate sensitivity with long-term importance.i One implementation is a weighted product c. i =(|g i |) α ·(s i ) 1-α α∈[0,1] is used to adjust the proportion of short-term and long-term signals. A weighted sum c can also be used. i =α|g i |+(1-α)s i This is combined with quantile normalization to eliminate scale differences. To suppress noise, c... i By normalizing within the causal chain using Softmax at temperature τ, the relative strengths become more distinct.

[0226] The interpretation of visual modalities involves region labeling on the visual modal data of the original data stream. This is achieved using convolutional feature maps A... k Construct a gradient-weighted activation mapping with the gradient of the predicted output, weights Heatmap H = ReLU(∑ k α k A k ). Node contribution c i Projecting onto spatial locations, the graph-level contribution is downgraded to the pixel or instance level using a node-to-pixel mapping matrix or object detection bounding box. Then, H is normalized and thresholded to obtain high-contribution regions. The geometry of the region is represented by polygons or a mask matrix, along with a region score s. region Together with the set of source nodes, it ensures that the causal chain can be traced back.

[0227] Text modality interpretation involves phrase extraction from the original text modality data stream. Word-level attribution is performed on the segmented sequences using either integral gradient or perturbation-based locally interpretable models. The integral gradient is defined as:

[0228]

[0229] Where x is the current input and x' is the baseline input. Word-level attribution is weighted and aggregated based on node contribution to obtain phrase-level scores. Dynamic programming is then used to search for the set of non-overlapping phrases with the highest scores within the allowed maximum length, which serves as a list of high-contribution phrases, while preserving the mapping relationship with causal chain nodes.

[0230] Cross-modal confidence representations drive modal weighting with reliability scores of causal relationships. Historical inference accuracy r is given for each modality m. m With reliability coefficient ρ m (As verified by counterfactual reasoning), calculate the weights:

[0231]

[0232] The weights represent a joint measure of availability and reliability. The prediction accuracy (a) is given for the online performance of the current task. m The confidence score can be calculated from the calibration set or cross-validation set of the previous time window.

[0233]

[0234] Logical calibration σ(γC+β) can be used to improve numerical interpretability. To handle missing modes, missing modes are first identified based on data availability markers, completeness test results, and outlier distribution tests. For missing modes m... * Using the transfer learning function T φ Inferring virtual features from fused features after cross-modal fusion Evaluate the quality of virtual features Where D ψ For decoder or discriminator, X proxy Using homologous proxy data, this is used to measure reconstruction consistency or adversarial discrimination scores. The confidence score is adaptively adjusted using the quality coefficient βC+(1-β)q·w. m ＊ β∈[0,1] controls the degree of conservatism, so that the final confidence expression takes into account both causal reliability and data completeness.

[0235] The structured integration of the interpretation results provides three parts within a unified carrier: visual annotations, text phrases, and confidence scores, while maintaining mappings to causal chains, node contributions, and reliability scores. The output format can include image spatial coordinates or a mask matrix, phrase sequences and their scores, global confidence scores and confidence components decomposed to modalities, referenced causal chain identifiers, and timestamps. To facilitate auditing and experimental reproducibility, key parameters used to generate the interpretation are retained, including α, τ, normalization constant, threshold, time window length, and weight update timestamps.

[0236] This paper implements a path-first link extraction and dual-channel attribution integration. Link extraction performs reverse expansion for each target relationship, with the target edge as the endpoint. Based on edge weights and temporal consistency, it generates several candidate chains within a limited depth, selecting the top K chains. The visual channel uses Grad-CAM or its improved version to generate a heatmap on the last convolutional layer. Object detection boxes or instance segmentation masks on the chains are used as masks, and the heatmap is weighted by the mask and linearly fused with node contributions to produce region-level scores and masks. The text channel calculates attribution at the sub-word level using integral gradients, applying hierarchical aggregation from sub-words to words and from words to phrases, and reweighting by node contributions. The products from both channels enter the confidence expression module, where they, along with the reliability score and online accuracy, generate a confidence score, which is then converted into a structured interpretation vector. This implementation has fewer parameters and is suitable for latency-sensitive scenarios.

[0237] Implementation 2 can introduce a graph attention reader, using causal chains as subgraphs and node contributions as prior guidance. A graph attention network is then used to read subgraph features into explanatory embeddings, which are then projected onto visual and textual spaces to generate annotations and phrases. This approach enables chain-level modeling of interactions, enhancing the expressive power of multi-hop causal dependencies. To improve stability, a regularization consistent with reliability scores can be applied to the explanatory embeddings, ensuring that high-reliability relationships dominate the explanations. The transfer learning function can be a cross-modal contrastive pre-trained decoder or a diffusion generator; semantic consistency loss can be added if necessary to stabilize the virtual feature quality assessment.

[0238] To achieve this, a calibration-first confidence fusion approach can be used. In the historical phase, a temperature-scaled or Platt-calibrated model for each modality is constructed. In the online phase, the calibrated score replaces the original accuracy in the weighted fusion, and the confidence expression is rewritten as follows: When a modality is detected as missing, two correction paths are introduced: one by virtual feature quality correction and the other by causal chain redundancy correction. The latter measures the decrease in causal chain score ΔS after removing the modality, and suppresses over-dependence on a single modality with C' = C' - λ·ΔS. This approach is suitable for environments where multi-source devices are susceptible to noise and packet loss.

[0239] Parameter tuning can be performed around four key quantities. The contribution fusion coefficient α affects the balance between gradient sensitivity and structural importance, and is best determined on the validation set through grid search, based on a trade-off between interpretability consistency and stability. The Softmax temperature τ determines the steepness of the contribution distribution; too large a value leads to flatness, while too small a value leads to excessive sharpness. It can be optimized within the interval [0.5, 5] based on interpretability coverage. The fusion of reliability and historical accuracy is achieved through normalized weights. If reliability fluctuates significantly, it can be adjusted using ρ... m Time decay or quantile pruning can be used. The threshold and correction coefficient β for virtual feature quality assessment can be set according to the risk preferences of different businesses. Healthcare tends to favor high β to maintain conservatism, while finance can set additional lower limits and record decision logs according to compliance audit requirements.

[0240] Example: In financial risk control and market analysis scenarios, business systems need to process multimodal data from different sources simultaneously, such as customer transaction records, credit history text records, market charts, news reports, and public opinion information. These data often exhibit complex causal relationships, such as "sudden negative news—stock price fluctuations—margin shortage—forced liquidation." Traditional analysis often relies on a single dimension and cannot capture cross-modal causal logic; therefore, unified modeling and interpretation of multimodal data is a key breakthrough.

[0241] The system first receives real-time trading data streams from stock exchanges, text reports from financial news platforms, market trend curves, and investor credit data. All data carries timestamps, such as transaction time, news release time, and credit report generation time. Alignment is achieved through timestamp comparison, uniformly mapping data from different sources to a time series dimension. When missing market data or incomplete news feeds are detected, the system automatically removes portions that do not meet the threshold based on a completeness index, ensuring that the generated financial data packets are reliable and usable for subsequent calculations.

[0242] At the trading level, the system models sequential data such as trading volume, price volatility, and order book depth, extracting basic numerical features and introducing sequential relationships in the time series through positional encoding. A self-attention mechanism is then used to capture the diffusion impact of abnormal trading in the short term, generating temporal features of the trading modality. At the text level, pre-trained models for financial corpora, such as FinBERT, are used to process news and announcements, extracting sentiment labels and semantic key points, such as "rating downgrade," "liquidity crunch," and "market panic." These results are mapped using a weight matrix and transformed into numerical sequential representations, comparable to trading features in the same space. Finally, the two types of modal features are standardized and appended with indicator descriptions to maintain dimensional alignment in the subsequent fusion stage.

[0243] In the cross-modal fusion process, information entropy is calculated for both transaction features and text features to measure internal uncertainty. The joint information entropy and similarity of these two features are then combined to obtain an adjustment coefficient that quantifies modal correlation. This coefficient undergoes exponential mapping and normalization to generate dynamic weights, which are used to weight and superimpose the two types of features to form the fused representation. The fused features are further input into a multi-layer Transformer structure. At the upper layer, an error function constrains the output to maintain consistency with the causal risk factors generated by the lower-layer causal inference engine. By iteratively minimizing the error, refined causal features are finally obtained, which better reflect potential cross-modal risk factors.

[0244] The dynamic graph constructed based on causal features uses investors, securities, news events, etc., as nodes, and potential causal relationships as edges. For example, the connection between a sharp drop in a stock price and negative news from the previous day can be represented by an edge. The edge weights are calculated by weighting the transitive entropy and conditional mutual information, reflecting the strength of the influence between the features of two nodes. This value is updated to the edge weights after being mapped by an activation function. The dynamic causal graph thus evolves continuously, recording the trajectory of edge weight changes over time, thereby forming a historical evolution sequence of causal relationships.

[0245] In the causal verification phase, the system performs counterfactual reasoning on the relationships identified in the graph, such as assuming "would the stock still fall if a certain negative news item hadn't been published?" Methodologically, it intervenes in the characteristics of the causal nodes in the causal chain, generating modified variables. These modified variables are then input into the counterfactual reasoning function along with irrelevant features and background context to obtain simulation results. By comparing the distribution differences before and after the intervention, and calculating the ratio of kurtosis to entropy as a stability indicator, the intervention parameters are repeatedly adjusted within a preset perturbation range to assess the sensitivity of the results. Finally, a reliability score is output. If the score exceeds a threshold, it indicates that the causal relationship is credible in financial logic.

[0246] In the explanation generation stage, the system extracts the verified causal chain and calculates the partial derivatives of the features of each node in the chain with respect to the final prediction. Combined with the importance score obtained from the historical evolution of the causal graph, the system forms the contribution of each node. On the trading chart, time intervals with high contribution are highlighted, such as intraday trading spikes. In the text data, phrases highly correlated with stock price fluctuations, such as "insolvency," are automatically extracted. Furthermore, the system assigns dynamic weights to different modalities based on historical inference accuracy and causal relationship reliability, and then calculates the overall confidence score by combining this with the current task prediction accuracy. If a modality is missing, such as due to news delays or market gaps, the system uses the fused features to generate virtual features through transfer learning to correct the confidence score. Finally, the visual annotations, text phrases, and adjusted confidence scores are integrated into an explanatory report, allowing financial analysts to clearly see the causal source and credibility level of market anomalies.

[0247] This series of operations enables financial risk control to achieve a more transparent causal chain explanation in a multimodal environment. It can identify the potential causal logic between transactions and news, and still provide robust confidence assessments even when modalities are missing, which helps in the early deployment of investment decisions and risk interventions.

[0248] In the healthcare field, applications can be seen in clinical diagnosis, disease progression prediction, and personalized treatment assessment. Healthcare systems routinely receive multimodal clinical data, including structured numerical information from electronic medical records (such as blood pressure, blood glucose, and blood oxygen levels), medical imaging data (such as CT and MRI scans), pathological slide images, physician-written medical records, and patient follow-up audio or text descriptions. This data, distributed across different modalities, collectively reflects the patient's pathological process. Relying solely on a single modality (such as medical imaging) is often insufficient to capture the underlying complex causal factors; therefore, cross-modal integration is necessary to establish causal relationships.

[0249] The system first receives multimodal raw data streams from medical institution information systems, medical imaging equipment, and wearable health monitoring devices, ensuring that each type of data has a timestamp, such as the generation time of imaging examinations, the sampling time of laboratory tests, and the time point of heart rate data uploaded by monitoring devices. These different modalities of raw data are unified onto the same timeline through timestamp alignment, forming a time-consistent data stream. In the integrity detection stage, if some detection data is missing or the image sequence is incomplete, data that does not meet the threshold is discarded, retaining the valid data stream, and ultimately generating a medical multimodal data package that can be used for causal inference.

[0250] In the feature extraction stage, medical images undergo convolutional operations to extract spatial anatomical features, while temporal information is embedded using a positional encoding function to obtain spatial features that reflect changes over time. A self-attention mechanism is then used to identify the dynamic relationships between potential abnormalities in the image sequence, forming spatial modality features. In the text modality, doctors' diagnostic records and medical history descriptions are processed by a pre-trained language model based on the medical corpus to extract semantic key points, such as "pulmonary infiltration" and "unclear mass boundaries," which are further mapped into numerical vectors to form sequence modality features. After standardization and the addition of dimensional descriptive information, different modality features can be compared and fused within the same representation space.

[0251] In the cross-modal fusion stage, the system calculates the individual and joint information entropies of the image and text modalities, combines them with inter-modal semantic similarity to obtain fusion adjustment coefficients, and obtains cross-modal weights through exponential and normalization operations. Based on these weights, the features of different modalities are weighted and superimposed to obtain the fused representation. The fused features are further input into a hierarchical Transformer network, where error function constraints are maintained with the causal inference engine in the upper layer to ensure that the fusion result can be aligned with the potential causal features. The error minimization process makes the output closer to the causal mechanisms that actually exist in medicine, such as the causal chain of "smoking history—pulmonary nodule formation—decreased lung function".

[0252] In the causal modeling phase, the fused causal features are used to construct a dynamic graph, where nodes can be patient symptoms, biomarkers, or imaging features, and edges represent causal relationships between different nodes. The weight of each edge is obtained by weighting with transitive entropy and conditional mutual information, reflecting the information transmission strength between two medical indicators. By mapping edge weights through activation functions and continuously updating them, the dynamic graph can evolve over time, recording the dynamic trajectory of causal relationships, such as the causal strength between changes in tumor size and changes in patient inflammatory markers fluctuating with the treatment process.

[0253] In the causal verification phase, the system performs counterfactual reasoning on the causal relationships inferred from the dynamic causal graph. For example, it hypothesizes, "If the patient had not received a certain drug treatment, would the inflammatory markers still have decreased?" During operation, new simulated features are generated by intervening in the causal characteristics of the causal chain. These features are then combined with background context features and input into the counterfactual reasoning function to generate comparative results. The kurtosis to entropy ratio of the probability distribution is further calculated as a stability index, and the sensitivity coefficient is calculated under different perturbation conditions to finally obtain a reliability score. If the score is higher than a threshold, the causal relationship is considered clinically credible.

[0254] In the causal explanation generation stage, the system extracts key causal chains and calculates the partial derivatives of each node with respect to the predicted result. Combining this with the importance scores of nodes updated in the dynamic causal graph, the system determines the contribution of each indicator to the diagnostic result. In medical images, the system highlights lesion areas; in medical record text, it annotates key descriptive phrases such as "invasive lesion" or "enhanced image signal." Simultaneously, by combining historical inference accuracy and reliability scores, the system dynamically assigns weights to the image and text modalities and calculates the overall confidence level. When a modality is missing (e.g., pathological images are temporarily unavailable), virtual features are generated through transfer learning using existing fusion features to correct the confidence level, ensuring the explanation remains reasonable even in missing conditions. Ultimately, the explanatory report received by doctors not only includes highly contributing image regions and medical record phrases but also confidence indicators, enabling them to better assess the reliability of diagnostic conclusions.

[0255] This embodiment integrates gradient attribution and structural importance on the causal chain, ensuring a one-to-one correspondence between explanatory elements and causal structures, thus avoiding misleading results from correlation alone. By dynamically adjusting modal weights based on the reliability score of causal relationships and generating confidence scores in conjunction with online accuracy, the explanatory credibility is consistent with data quality, model calibration, and counterfactual verification results. In scenarios with missing modalities, virtual features generated by transfer learning functions are introduced and adaptively corrected with quality scores, explaining the continuity and measurable uncertainty even under incomplete input conditions. The structured output simultaneously includes spatial annotations, key phrases, and confidence expressions, providing a clear audit and reproducibility path for experiments. Combined with the update history, it can be traced back to specific time slices and causal chain versions, thereby achieving cross-modal, traceable, and calibrable causal explanation generation.

[0256] In one embodiment, a multimodal causal reasoning and explanation apparatus is provided, which corresponds one-to-one with the multimodal causal reasoning and explanation methods described in the above embodiments. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal causal reasoning and explanation device of the present invention. The modules include a multimodal data acquisition module 10, a modal feature extraction module 20, a cross-modal fusion and distillation module 30, a dynamic causal graph construction and update module 40, a counterfactual reasoning verification module 50, and a causal explanation generation module 60. Detailed descriptions of each functional module are as follows:

[0257] The multimodal data acquisition module 10 is used to acquire multimodal data from raw data streams including at least two different modalities.

[0258] The modal feature extraction module 20 is used to perform feature extraction processing on each type of raw data stream in the multimodal data to generate corresponding modal features.

[0259] The cross-modal fusion and distillation module 30 is used to perform cross-modal fusion of the modal features, adjust the fusion weights through a cross-modal attention mechanism to generate fused features, and extract causal features from the fused features through a feature distillation process.

[0260] The dynamic causal graph construction and update module 40 is used to construct a dynamic causal graph containing nodes representing entities and edges representing causal relationships based on the causal features, and to update the edges using a causal strength function;

[0261] The counterfactual reasoning verification module 50 is used to perform counterfactual reasoning verification on the causal relationships identified in the dynamic causal graph in order to analyze the reliability of the causal relationships;

[0262] The causal explanation generation module 60 is used to generate causal explanation results based on the dynamic causal graph and the reliability of the causal relationship.

[0263] In one embodiment, the multimodal data acquisition module 10 is specifically used for:

[0264] Receive raw data streams from at least two independent data acquisition devices;

[0265] Extract the timestamp information from the original data stream;

[0266] Based on the timestamp information, the original data streams of different modalities are aligned to generate a time-aligned original data stream;

[0267] Detect the data integrity of the time-aligned raw data stream;

[0268] Discard the raw data stream with a data integrity lower than a preset threshold to obtain a valid raw data stream;

[0269] The valid raw data stream is encapsulated to generate a multimodal data packet.

[0270] In one embodiment, the modal feature extraction module 20 is specifically used for:

[0271] Perform data structure analysis on each type of raw data stream in the multimodal data;

[0272] For a raw data stream with spatial dimensions, perform convolution operations to extract spatial features, and apply a positional encoding function to process the spatial features to generate spatial features with temporal information.

[0273] The spatial features with temporal information are processed using a self-attention mechanism to generate spatial self-attention features;

[0274] The spatial features and spatial self-attention features are combined to generate spatial modal features;

[0275] For a raw data stream with a sequence structure, a pre-trained language model is used to process the raw data stream to generate initial semantic features, and a weight matrix is ​​applied to process the initial semantic features to generate sequence modal features;

[0276] The spatial modal features and sequence modal features are subjected to format standardization processing to generate standardized modal features;

[0277] Add feature dimension description information to the standardized modal features, and use the standardized modal features as the modal features corresponding to each original data stream.

[0278] In one embodiment, the cross-modal fusion and distillation module 30 is specifically used for:

[0279] The cross-modal fusion module receives first modal features and second modal features from the feature extraction and processing module;

[0280] Determine the information entropy of the first modality feature and the information entropy of the second modality feature;

[0281] Determine the joint information entropy of the first modal feature and the second modal feature;

[0282] Mutual information entropy is determined based on the aforementioned information entropy and joint information entropy;

[0283] Determine the similarity between the first modal feature and the second modal feature;

[0284] An adjustment coefficient is generated based on the mutual information entropy and similarity.

[0285] The adjustment coefficients are processed using an exponential function to generate intermediate weight values, and the intermediate weight values ​​are processed using a normalization function to generate fused weights.

[0286] The first modal feature and the second modal feature are weighted and summed using the fusion weights to generate fused features;

[0287] The fused features are processed by the upper-layer Transformer model of the feature distillation to generate the initial fused output.

[0288] The initial fusion output is passed to the lower-level causal inference engine, where causal features and prediction results are generated.

[0289] In the upper-layer Transformer model, the difference between the initial fusion output and the causal features is compared using an error function;

[0290] The error function is minimized to optimize the initial fusion output, and the optimized initial fusion output is used as the extracted causal feature output.

[0291] In one embodiment, the dynamic cause-effect graph construction and update module 40 is specifically used for:

[0292] Multimodal entity information is extracted based on the causal features to form a node set;

[0293] Based on the causal relationships between nodes in the node set, construct a set of edges connecting the nodes;

[0294] For each edge in the edge set, determine the transfer entropy between the two node features connected by the edge;

[0295] Determine the conditional mutual information between the features of the two nodes connected by the edge;

[0296] The transit entropy and conditional mutual information are weighted and summed using a preset balance coefficient to generate a causal strength value.

[0297] The causal strength value is processed by an activation function to generate updated edge weights;

[0298] The updated edge weights are used to replace the original edge weights of each corresponding edge to obtain the updated dynamic causal graph.

[0299] In subsequent updates, the changes in edge weights over time are recorded and time series are formed by timestamps to obtain the update history of the dynamic causal graph.

[0300] In one embodiment, the counterfactual reasoning verification module 50 is specifically used for:

[0301] Identify target causal relationships from the dynamic causal graph;

[0302] The causal features in the target causal relationship are intervened using preset intervention parameters to generate the intervened features;

[0303] Extract node features that are unrelated to the causal features from the node set of the dynamic causal graph to form a non-causal feature set;

[0304] Scene background node features are extracted from the node set of the dynamic cause-effect graph to form a context feature set;

[0305] The counterfactual inference function is used to process the post-intervention feature set, non-causal feature set, and context feature set to generate counterfactual simulation results.

[0306] Based on the structural equation of the dynamic causal graph, the probability distribution after intervention is generated;

[0307] The ratio of kurtosis to entropy of the probability distribution is determined as a statistical stability index.

[0308] The intervention parameters are adjusted within a preset perturbation range, and the perturbation features are generated based on the adjusted intervention parameters.

[0309] The counterfactual inference function is used to process the perturbed intervention features, non-causal feature set, and context feature set to generate the perturbed counterfactual simulation results;

[0310] Determine the relative rate of change between the counterfactual simulation results and the perturbed counterfactual simulation results, and generate a sensitivity coefficient;

[0311] A reliability score is generated based on the statistical stability index and sensitivity coefficient.

[0312] When the reliability score exceeds a preset threshold, the target causal relationship is determined to be reliable.

[0313] In one embodiment, the causal explanation generation module 60 is specifically used for:

[0314] Extract the causal chain of the target causal relationship from the dynamic causal graph;

[0315] Based on the prediction results generated by the causal inference engine, the partial derivatives of the node features in the causal chain with respect to the prediction results are determined.

[0316] Based on the update history of the dynamic cause-effect graph, the importance score of node features is determined;

[0317] The node contribution is determined by combining the partial derivatives and importance scores.

[0318] On the visual modal data of the original data stream, high contribution regions are marked based on the node contribution.

[0319] Based on the node contribution, high-contribution phrases are extracted from the text modal data of the original data stream.

[0320] The weight of each modality is dynamically assigned based on its inference accuracy and causal reliability score in historical data.

[0321] Determine the prediction accuracy of each modality for the current inference task;

[0322] The confidence score is generated by combining the weights and the prediction accuracy.

[0323] Detect whether there are modal missing parameters in the raw data stream;

[0324] When the original data stream has modal missing features, virtual features are generated by using the fused features after cross-modal fusion through a transfer learning function.

[0325] Adjust the confidence score based on the virtual features;

[0326] By integrating the high-contribution region annotations, high-contribution phrases, and adjusted confidence scores, a causal explanation is generated.

[0327] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal causal reasoning and explanation method on the server side.

[0328] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a multimodal causal reasoning and explanation method on the user side.

[0329] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0330] Acquire multimodal data from raw data streams containing at least two different modalities;

[0331] Feature extraction processing is performed on each of the original data streams in the multimodal data to generate corresponding modal features;

[0332] The modal features are fused across modalities, and the fusion weights are adjusted through a cross-modal attention mechanism to generate fused features. Causal features are then extracted from the fused features through a feature distillation process.

[0333] Based on the causal features, a dynamic causal graph is constructed, which includes nodes representing entities and edges representing causal relationships, and the edges are updated using a causal strength function.

[0334] Counterfactual reasoning is performed on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships;

[0335] A causal explanation is generated based on the dynamic causal graph and the reliability of the causal relationships.

[0336] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0337] Acquire multimodal data from raw data streams containing at least two different modalities;

[0338] Feature extraction processing is performed on each of the original data streams in the multimodal data to generate corresponding modal features;

[0339] The modal features are fused across modalities, and the fusion weights are adjusted through a cross-modal attention mechanism to generate fused features. Causal features are then extracted from the fused features through a feature distillation process.

[0340] Based on the causal features, a dynamic causal graph is constructed, which includes nodes representing entities and edges representing causal relationships, and the edges are updated using a causal strength function.

[0341] Counterfactual reasoning is performed on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships;

[0342] A causal explanation is generated based on the dynamic causal graph and the reliability of the causal relationships.

[0343] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0344] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0345] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0346] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal causal reasoning and explanation method, characterized in that, Includes the following steps: Acquire multimodal data from raw data streams containing at least two different modalities; Feature extraction processing is performed on each of the original data streams in the multimodal data to generate corresponding modal features; The modal features are fused across modalities, and the fusion weights are adjusted through a cross-modal attention mechanism to generate fused features. Causal features are then extracted from the fused features through a feature distillation process. Based on the causal features, a dynamic causal graph is constructed, which includes nodes representing entities and edges representing causal relationships, and the edges are updated using a causal strength function. Counterfactual reasoning is performed on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships; A causal explanation is generated based on the dynamic causal graph and the reliability of the causal relationships.

2. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, Acquire multimodal data from raw data streams containing at least two different modalities, including: Receive raw data streams from at least two independent data acquisition devices; Extract the timestamp information from the original data stream; Based on the timestamp information, the original data streams of different modalities are aligned to generate a time-aligned original data stream; Detect the data integrity of the time-aligned raw data stream; Discard the raw data stream with a data integrity lower than a preset threshold to obtain a valid raw data stream; The valid raw data stream is encapsulated to generate a multimodal data packet.

3. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, Feature extraction processing is performed on each of the original data streams in the multimodal data to generate corresponding modal features, including: Perform data structure analysis on each type of raw data stream in the multimodal data; For a raw data stream with spatial dimensions, perform convolution operations to extract spatial features, and apply a positional encoding function to process the spatial features to generate spatial features with temporal information. The spatial features with temporal information are processed using a self-attention mechanism to generate spatial self-attention features; The spatial features and spatial self-attention features are combined to generate spatial modal features; For a raw data stream with a sequence structure, a pre-trained language model is used to process the raw data stream to generate initial semantic features, and a weight matrix is ​​applied to process the initial semantic features to generate sequence modal features; The spatial modal features and sequence modal features are subjected to format standardization processing to generate standardized modal features; Add feature dimension description information to the standardized modal features, and use the standardized modal features as the modal features corresponding to each original data stream.

4. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, The modal features are fused across modalities, and the fusion weights are adjusted using a cross-modal attention mechanism to generate fused features. Causal features are then extracted from the fused features through a feature distillation process, including: The cross-modal fusion module receives first modal features and second modal features from the feature extraction and processing module; Determine the information entropy of the first modality feature and the information entropy of the second modality feature; Determine the joint information entropy of the first modal feature and the second modal feature; Mutual information entropy is determined based on the aforementioned information entropy and joint information entropy; Determine the similarity between the first modal feature and the second modal feature; An adjustment coefficient is generated based on the mutual information entropy and similarity. The adjustment coefficients are processed using an exponential function to generate intermediate weight values, and the intermediate weight values ​​are processed using a normalization function to generate fused weights. The first modal feature and the second modal feature are weighted and summed using the fusion weights to generate fused features; The fused features are processed by the upper-layer Transformer model of the feature distillation to generate the initial fused output. The initial fusion output is passed to the lower-level causal inference engine, where causal features and prediction results are generated. In the upper-layer Transformer model, the difference between the initial fusion output and the causal features is compared using an error function; The error function is minimized to optimize the initial fusion output, and the optimized initial fusion output is used as the extracted causal feature output.

5. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, Based on the causal features, a dynamic causal graph is constructed, comprising nodes representing entities and edges representing causal relationships. The edges are then updated using a causal strength function, including: Multimodal entity information is extracted based on the causal features to form a node set; Based on the causal relationships between nodes in the node set, construct a set of edges connecting the nodes; For each edge in the edge set, determine the transfer entropy between the two node features connected by the edge; Determine the conditional mutual information between the features of the two nodes connected by the edge; The transit entropy and conditional mutual information are weighted and summed using a preset balance coefficient to generate a causal strength value. The causal strength value is processed by an activation function to generate updated edge weights; The updated edge weights are used to replace the original edge weights of each corresponding edge to obtain the updated dynamic causal graph. In subsequent updates, the changes in edge weights over time are recorded and time series are formed by timestamps to obtain the update history of the dynamic causal graph.

6. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, Perform counterfactual reasoning verification on the causal relationships identified in the dynamic causal graph to analyze the reliability of the causal relationships, including: Identify target causal relationships from the dynamic causal graph; The causal features in the target causal relationship are intervened using preset intervention parameters to generate the intervened features; Extract node features that are unrelated to the causal features from the node set of the dynamic causal graph to form a non-causal feature set; Scene background node features are extracted from the node set of the dynamic cause-effect graph to form a context feature set; The counterfactual inference function is used to process the post-intervention feature set, non-causal feature set, and context feature set to generate counterfactual simulation results. Based on the structural equation of the dynamic causal graph, the probability distribution after intervention is generated; The ratio of kurtosis to entropy of the probability distribution is determined as a statistical stability index. The intervention parameters are adjusted within a preset perturbation range, and the perturbation features are generated based on the adjusted intervention parameters. The counterfactual inference function is used to process the perturbed intervention features, non-causal feature set, and context feature set to generate the perturbed counterfactual simulation results; Determine the relative rate of change between the counterfactual simulation results and the perturbed counterfactual simulation results, and generate a sensitivity coefficient; A reliability score is generated based on the statistical stability index and sensitivity coefficient. When the reliability score exceeds a preset threshold, the target causal relationship is determined to be reliable.

7. The multimodal causal reasoning and explanation method as described in claim 1, characterized in that, Based on the dynamic causal graph and the reliability of the causal relationships, a causal explanation is generated, including: Extract the causal chain of the target causal relationship from the dynamic causal graph; Based on the prediction results generated by the causal inference engine, the partial derivatives of the node features in the causal chain with respect to the prediction results are determined. Based on the update history of the dynamic cause-effect graph, the importance score of node features is determined; The node contribution is determined by combining the partial derivatives and importance scores. On the visual modal data of the original data stream, high contribution regions are marked based on the node contribution. Based on the node contribution, high-contribution phrases are extracted from the text modal data of the original data stream. The weight of each modality is dynamically assigned based on its inference accuracy and causal reliability score in historical data. Determine the prediction accuracy of each modality for the current inference task; The confidence score is generated by combining the weights and the prediction accuracy. Detect whether there are modal missing parameters in the raw data stream; When the original data stream has modal missing features, virtual features are generated by using the fused features after cross-modal fusion through a transfer learning function. Adjust the confidence score based on the virtual features; By integrating the high-contribution region annotations, high-contribution phrases, and adjusted confidence scores, a causal explanation is generated.

8. A multimodal causal reasoning and explanation device, characterized in that, The multimodal causal reasoning and explanation device includes: A multimodal data acquisition module is used to acquire multimodal data from raw data streams that include at least two different modalities. The modal feature extraction module is used to perform feature extraction processing on each type of raw data stream in the multimodal data to generate corresponding modal features. The cross-modal fusion and distillation module is used to perform cross-modal fusion of the modal features, adjust the fusion weights through a cross-modal attention mechanism to generate fused features, and extract causal features from the fused features through a feature distillation process. The dynamic causal graph construction and update module is used to construct a dynamic causal graph containing nodes representing entities and edges representing causal relationships based on the causal features, and to update the edges using a causal strength function; The counterfactual reasoning verification module is used to perform counterfactual reasoning verification on the causal relationships identified in the dynamic causal graph in order to analyze the reliability of the causal relationships; The causal explanation generation module is used to generate causal explanation results based on the dynamic causal graph and the reliability of the causal relationships.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multimodal causal reasoning and interpretation program stored in the memory and executable on the processor, wherein the multimodal causal reasoning and interpretation program, when executed by the processor, implements the steps of the multimodal causal reasoning and interpretation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a multimodal causal reasoning and interpretation program, which, when executed by a processor, implements the steps of the multimodal causal reasoning and interpretation method as described in any one of claims 1-7.

Images