Multi-modal time-series causal intelligent decision method and device for complex scene, equipment and medium

Abstract

This application discloses a multimodal temporal causal intelligent decision-making method, apparatus, device, and medium for complex scenarios, relating to the field of artificial intelligence technology. The method includes: constructing an integrated intelligent decision-making framework from multimodal data; achieving shared semantic representation of multi-source data through unified encoding; obtaining reliable fusion causal strength through dual-channel causal discovery; constructing and dynamically updating a causal knowledge graph based on this strength; and generating compliant decisions by combining constraint projection and counterfactual verification. The beneficial effects are: realizing dynamic causal knowledge updating and generating constraint-compliant and reliable decisions based on multimodal data; ensuring that decisions strictly comply with physical and business rules; and improving the reliability, robustness, and real-time performance of decisions in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal temporal causal intelligent decision-making method, apparatus, device, and medium for complex scenarios. Background Technology

[0002] Against the backdrop of the booming digital economy, modern enterprise operations and management are undergoing a profound data-driven transformation. Complex business scenarios in key areas such as energy and power dispatching, supply chain resilience management, financial risk control, and intelligent manufacturing control continuously generate multimodal data. This data encompasses various modalities, including numerical time series (sensor readings, transaction records), topological graph structures (power grid topology, supply chain networks), and domain texts (dispatch instructions, fault reports, compliance documents). Enterprises urgently need to extract reliable causal knowledge from this complex data to achieve real-time intelligent decision-making in response to highly dynamic changes in the business environment.

[0003] Existing technologies mainly adopt the following approaches: First, for multimodal data processing, a post-fusion mode is usually adopted, where different modal data are sent into independent processing pipelines, and the results are spliced ​​only in the later stage of analysis; Second, for causal discovery, either deep neural networks are relied upon for end-to-end learning, or statistical causal inference methods such as frameworks based on conditional independence tests are relied upon; Third, for knowledge updating, an offline knowledge graph model with batch updates is usually adopted; Fourth, for decision generation, large language models usually adopt a "generation-check-correction" post-verification mode, or constraint satisfaction is treated as an independent post-processing step. However, existing practices have the following problems: the use of independent processing pipelines for multimodal data leads to the loss of a large amount of strong correlation information between different modalities, making it difficult to achieve cross-modal semantic alignment; deep neural networks essentially learn correlation rather than causality, which can easily lead to erroneous causal judgments due to spurious correlations, while statistical causal inference methods have limited ability to model nonlinear relationships; the offline batch update mode of knowledge graphs makes the knowledge update cycle measured in days or even weeks, which cannot meet the millisecond-level response requirements; large language models lack strict adherence to physical constraints and business rules when generating decisions, and the post-validation mode is inefficient and cannot guarantee that constraints are met.

[0004] Therefore, how to achieve dynamic causal knowledge updates and generate compliant and reliable decisions based on multimodal data has become an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a multimodal temporal causal intelligent decision-making method, device, equipment and medium for complex scenarios, aiming to solve the technical problem of how to realize dynamic causal knowledge updating and generate constraint-compliant and reliable decisions based on multimodal data.

[0006] To achieve the above objectives, this application proposes a multimodal temporal causal intelligent decision-making method for complex scenarios, including: Acquire multimodal input data, wherein the multimodal input data includes numerical time series data, topological graph structure data, and domain text data; The multimodal input data is uniformly encoded to obtain a shared semantic representation; Perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength; A causal knowledge graph is constructed based on the fused causal strength, and the structural change between the current graph structure and the historical graph structure is calculated based on the Frobenius norm. When the amount of structural change exceeds a preset change threshold, incremental updates are performed on the affected subgraphs to obtain a dynamic causal knowledge graph. In a dynamic causal knowledge graph, vector similarity retrieval and multi-hop causal path traversal are performed to obtain the reasoning context. Based on the inference context, constraint projection and counterfactual verification are performed to obtain a decision scheme that satisfies both physical constraints and business rules.

[0007] In one embodiment, the step of uniformly encoding the multimodal input data to obtain a shared semantic representation includes: The numerical time-series data in the multimodal input data are subjected to time-series reprogramming and structured state-space model encoding to obtain time-series encoded features; Multi-layer graph neural network message passing is performed on the topological graph structure data in the multimodal input data, and attention-weighted aggregation is performed on the node neighbor features to obtain graph structure encoding features; The domain text data in the multimodal input data is input into a pre-trained language model, and the hidden states corresponding to the classification labels are extracted as text encoding features. The classification labels are inserted at the beginning of the domain text data. The pre-trained language model is a bidirectional encoder representation model fine-tuned by the domain text. Based on the temporal coding features, graph structure coding features, and text coding features, a contrastive learning loss function is constructed and cross-modal semantic alignment is performed to obtain a shared semantic representation.

[0008] In one embodiment, the step of performing temporal reprogramming and structured state-space model encoding on the numerical time-series data in the multimodal input data to obtain temporal encoded features includes: The numerical time-series data is processed into time-series blocks to obtain multiple time-series data blocks; The temporal data blocks are mapped and transformed through a learnable reprogrammable layer to obtain semantic token representations; The input-dependent dynamic state transition matrix, input matrix, and output matrix are calculated based on the semantic token representation using a selective scanning mechanism, wherein the selective scanning mechanism is an adaptive selection of state space parameters based on the semantic token representation. Based on the dynamic state transition matrix, input matrix, and output matrix, a structured state-space dual computation is performed on the semantic token representation matrix to obtain the temporal coding features.

[0009] In one embodiment, the step of performing dual-channel causal discovery on the shared semantic representation to obtain fused causal strength includes: Attention calculations are performed on the shared semantic representation through neural network channels to obtain an attention weight matrix; Weighted features are obtained based on the attention weight matrix; The weighted features are concatenated with the variable representations in the shared semantic representation and input into a multilayer perceptron for causal strength estimation to obtain the first causal strength. Apply sparsity regularization constraints to the first causal strength to obtain the sparsified causal strength; The second causal strength is obtained by performing a kernel-based conditional mutual information test on the shared semantic representation through a statistical inference channel. The adaptive gating weights are calculated based on the first causal strength and the second causal strength. The sparsed causal strength and the second causal strength are weighted and fused according to the adaptive gating weights to obtain the fused causal strength.

[0010] In one embodiment, the step of performing vector similarity retrieval and multi-hop causal path traversal in a dynamic causal knowledge graph to obtain the inference context includes: Convert causal nodes in a dynamic causal knowledge graph into fixed-dimensional semantic vectors; Based on reasoning requirements, perform vector similarity retrieval in the semantic vector space to filter target causal nodes; Starting from the target causal node, perform a multi-level, multi-hop causal path traversal along the directed causal edges to obtain the traversal result; Based on the traversal results, the causal path and causal strength information are structurally integrated to obtain the reasoning context.

[0011] In one embodiment, the step of performing constraint projection and counterfactual verification based on the inference context to obtain a decision scheme that satisfies physical constraints and business rules includes: Physical constraints and business rules are formalized into a set of differentiable constraint functions; Based on the reasoning context, an initial decision sequence is generated using a large language model to obtain the initial decision probability distribution; The degree of constraint violation of the initial decision sequence is calculated based on the set of differentiable constraint functions; The constraint penalty term is calculated using a modified linear unit based on the degree of constraint violation and the preset penalty coefficient. Construct a joint optimization objective based on the initial decision probability distribution and the constraint penalty term; Based on the joint optimization objective, the initial decision sequence is subjected to constrained optimization decoding based on the Lagrange multiplier method to obtain candidate decision schemes; The candidate decision schemes are processed based on the structural causal model to obtain decision schemes that satisfy physical constraints and business rules, wherein the structural causal model is a pre-built model that includes fused causal strength.

[0012] In one embodiment, the step of processing the candidate decision schemes based on the structural causal model to obtain decision schemes that satisfy physical constraints and business rules includes: The candidate decision schemes are calculated based on the structural causal model to obtain a first probability distribution under intervention conditions and a second probability distribution under natural observation conditions, wherein the intervention conditions are the values ​​of intervention variables that are forcibly set by executing operators. By comparing the difference between the first probability distribution and the second probability distribution, the causal effect difference results are obtained; When the difference in causal effect meets the preset causal validity condition, the corresponding candidate decision scheme is retained to obtain the filtered scheme. The selected solutions are determined to be the decision-making solutions that meet both physical constraints and business rules.

[0013] Furthermore, to achieve the above objectives, this application also proposes a multimodal temporal causal intelligent decision-making device for complex scenarios, the multimodal temporal causal intelligent decision-making device for complex scenarios comprising: The acquisition module is used to acquire multimodal input data, wherein the multimodal input data includes numerical time series data, topological graph structure data, and domain text data; The encoding module is used to uniformly encode the multimodal input data to obtain a shared semantic representation; The discovery module is used to perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength; The calculation module is used to construct a causal knowledge graph based on the fused causal strength, and to calculate the structural change between the current graph structure and the historical graph structure based on the Frobenius norm; The update module is used to perform incremental updates on the affected subgraphs to obtain a dynamic causal knowledge graph when the amount of structural change exceeds a preset change threshold. The processing module is used to perform vector similarity retrieval and multi-hop causal path traversal in the dynamic causal knowledge graph to obtain the reasoning context; The results module is used to perform constraint projection and counterfactual verification based on the inference context to generate a decision scheme that satisfies physical constraints and business rules.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the multimodal temporal causal intelligent decision-making method for complex scenarios as described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the multimodal temporal causal intelligent decision-making method for complex scenarios as described above.

[0016] This application constructs an integrated intelligent decision-making framework based on multimodal data. It achieves shared semantic representation of multi-source data through unified encoding, obtains reliable causal strength through dual-channel causal discovery, and constructs and dynamically updates a causal knowledge graph based on this strength. Finally, it generates compliant decisions by combining constraint projection and counterfactual verification. The beneficial effects include realizing dynamic causal knowledge updates and generating constraint-compliant and reliable decisions based on multimodal data, ensuring that decisions strictly comply with physical and business rules, and improving the reliability, robustness, and real-time performance of decisions in complex scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the multimodal temporal causal intelligent decision-making method for complex scenarios presented in this application. Figure 2 This is a flowchart illustrating the second embodiment of the multimodal temporal causal intelligent decision-making method for complex scenarios presented in this application. Figure 3 This is a schematic diagram of the module structure of the multimodal temporal causal intelligent decision-making device for complex scenarios according to this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the multimodal temporal causal intelligent decision-making method for complex scenarios in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Existing technologies mainly adopt the following approaches: First, for multimodal data processing, a post-fusion mode is usually adopted, where different modal data are sent into independent processing pipelines, and the results are spliced ​​only in the later stage of analysis; Second, for causal discovery, either deep neural networks are relied upon for end-to-end learning, or statistical causal inference methods such as frameworks based on conditional independence tests are relied upon; Third, for knowledge updating, an offline knowledge graph model with batch updates is usually adopted; Fourth, for decision generation, large language models usually adopt a "generation-check-correction" post-verification mode, or constraint satisfaction is treated as an independent post-processing step. However, existing practices suffer from the following problems: The use of independent processing pipelines for multimodal data leads to the significant loss of strong correlations between different modalities, making cross-modal semantic alignment difficult; deep neural networks essentially learn correlations rather than causality, easily resulting in spurious correlations and erroneous causal judgments, while statistical causal inference methods have limited ability to model nonlinear relationships; the offline batch update mode of knowledge graphs results in knowledge update cycles measured in days or even weeks, failing to meet millisecond-level response requirements; large language models lack strict adherence to physical constraints and business rules during decision generation, and post-validation is inefficient and cannot guarantee constraint satisfaction. Therefore, how to achieve dynamic causal knowledge updates and compliant, reliable decision generation based on multimodal data has become an urgent problem to be solved.

[0023] Based on the above, this application also provides a multimodal temporal causal intelligent decision-making method for complex scenarios, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multimodal temporal causal intelligent decision-making method for complex scenarios proposed in this application.

[0024] In this embodiment, the multimodal temporal causal intelligent decision-making method for complex scenarios includes steps S10 to S70: Step S10: Obtain multimodal input data.

[0025] It should be noted that multimodal input data includes numerical time-series data, topology graph data, and domain text data. Numerical time-series data consists of continuously collected quantitative observation records at fixed time intervals, presenting the changes of business indicators over time in numerical form. Topology graph data is structured data composed of nodes and edges, where nodes represent business entities and edges represent the connections between entities. This data is used to describe the physical or logical connection patterns in scenarios such as power grids and supply chains. Domain text data is natural language information generated within business scenarios, carrying business instructions and event descriptions in text form. This data includes professional content such as scheduling reports and fault descriptions.

[0026] Specifically, firstly, numerical time-series data, such as power grid load curves, equipment temperature sequences, or financial transaction prices, are collected in real time through industrial IoT sensors, business system logs, or transaction interfaces to ensure that continuous dynamic changes in the physical world are captured at a fixed frequency. Secondly, topology graph structure data is extracted from graph databases, network topology management systems, or supply chain platforms to obtain the connection relationships and dependency structures between entities, such as the line connections between power grid buses and transformers, and the material supply networks between enterprises, to characterize the structural constraints and transmission paths of the system. Finally, textual data from fields such as dispatch instructions, operation and maintenance reports, policies and regulations, and expert experience documents are collected, such as power dispatch instruction texts and equipment fault handling procedures, to supplement explicit business rules and implicit expert knowledge. The three types of data are aligned and associated through timestamps or entity identifiers to form a multimodal input covering dynamic monitoring, structural association, and knowledge description, thereby comprehensively characterizing the multidimensional features of complex management scenarios and avoiding information blind spots from a single data source that could lead to subsequent causal discovery and decision-making biases.

[0027] Step S20: Perform unified encoding on the multimodal input data to obtain a shared semantic representation.

[0028] It should be noted that step S20 includes: First, performing time-series reprogramming and structured state-space model encoding on the numerical time-series data in the multimodal input data to obtain time-series encoded features. Time-series reprogramming is a transformation operation that maps the original numerical time-series data to a semantic token sequence. This operation converts irregular or low-dimensional time-series values ​​into a high-dimensional semantic space representation. This operation enables the time-series data to adapt to the input format requirements of subsequent deep models. The structured state-space model is a deep learning model for modeling long time-series sequences. This model characterizes time-series dependencies through a dynamic state matrix. This model can efficiently capture long-distance time-series features and maintain computational stability. Further, the numerical time-series data is processed into time-series blocks to obtain multiple time-series data blocks. Specifically, continuous numerical time-series data (such as power grid load monitoring sequences) is slidably divided into multiple non-overlapping time-series data blocks according to a preset fixed length (such as 128 time points). Each block represents a change pattern within a local time window, thereby reducing the direct computational pressure of long sequences and providing standardized units for subsequent parallel processing. Then, a learnable reprogrammable layer is used to map and transform the time-series data blocks to obtain semantic token representations. Specifically, the learnable reprogrammable layer (usually implemented by linear projection or lightweight neural networks) maps each numerical time-series data block from the original numerical space to a high-dimensional semantic token representation. The specific formula is as follows: Where Z represents the reprogrammed semantic token representation, X represents numerical time series data, X ∈ R^{T×D}, T is the time step, D is the variable dimension, Patch represents the time series segmentation operation, which divides the time series into N patches, Reprogram represents a learnable reprogrammable network, which maps the patches to the semantic space, N represents the number of patches, and d represents the semantic space dimension, which transforms the originally abstract numerical signals (such as voltage fluctuation curves) into semantic vectors that the language model can understand, laying the foundation for multimodal data to share the same mathematical space.

[0029] A selective scanning mechanism is used to compute dynamic state transition matrices, input matrices, and output matrices based on semantic token representations, resulting in input-dependent dynamic state transition matrices. Then, structured state-space dual computation is performed on the semantic token representation matrices based on these matrices to obtain temporal coding features. It's important to understand that the selective scanning mechanism adaptively selects state-space parameters based on semantic token representations, and the dynamic state transition matrix is ​​a parameter matrix describing the evolution of temporal states over time. This matrix is ​​dynamically generated from the input semantic token representations and is used to characterize the state propagation patterns within the temporal data, determining how historical information evolves. The input matrix controls the fusion method of the current input, and the output matrix adjusts the mapping from hidden states to outputs. This allows the model to adaptively adjust information retention strategies for different scenarios such as "load surges" and "stable operation," accurately capturing long-range dependencies. Structured state-space dual computation is a bidirectional temporal feature computation method based on the state-space model. This computation combines forward state propagation and backward feature integration to fully explore the global dependencies of long-term temporal data. Specifically, a selective scanning mechanism is used to adaptively analyze the semantic token representation. Based on the content features of the current input data, the dynamic state transition matrix, input matrix, and output matrix dependent on the input are dynamically calculated. Finally, the Mamba-2 state-space model is used to model the reprogrammed sequence. Based on a structured state-space dual computation framework, the dynamic state transition matrix and input matrix are used to perform recursive state updates on the semantic token representation matrix (the hidden state is determined by the previous state and the current input). The hidden state is then mapped to the output through the output matrix, generating temporal coding features with linear computational complexity. The specific formula is as follows: in This represents the hidden state vector at time t. This represents the hidden state vector at time t-1. The semantic token representation of time t. The temporal coding features at time t are represented. Here is the state transition matrix. For the input matrix, The output matrix is ​​t, which represents the time step index.

[0030] Next, multi-layer graph neural network message passing is performed on the topology graph structure data in the multimodal input data. Attention-weighted aggregation of node neighbor features is then applied to obtain the graph structure encoding features. It's important to understand that multi-layer graph neural network message passing is a computational process that propagates node features layer by layer in a multi-layer network structure. This process allows each layer of nodes to acquire and integrate feature information from a broader neighborhood. Node neighbor features are the feature information corresponding to adjacent nodes that have a direct connection with the current node. These features reflect the current node's surrounding environment and association attributes within the topology. Attention-weighted aggregation is a method of assigning weights and fusing features based on the importance of relationships between nodes. This method assigns higher weight values ​​to neighboring nodes with stronger correlations. Specifically, for the topology graph structure G = (V, E), where V is the set of nodes (e.g., buses and transformers in a power grid) and E is the set of edges (e.g., transmission lines), this embodiment uses a graph neural network (GNN) for encoding. Node representations are updated through a message passing mechanism, specifically using the following formula: in Represents a node In the Layer representation vector, Represents a node In the The layer is represented by a vector, where N(v) represents the set of neighbors of node v, and AGG represents the aggregation function (such as mean aggregation or attention-weighted aggregation). Here, v and u represent the GNN layer indices, and v and u represent the graph node indices. After L layers of message passing, the graph structure encoding features are obtained: in This represents the encoding features of the graph structure. This represents the representation vector of node v in the last layer L. READOUT represents the graph-level readout function (such as global average pooling or attention pooling), V represents the set of all nodes in the graph, and L represents the total number of layers in the GNN. It encodes the global structural constraints of the power grid topology or supply chain network into fixed-dimensional graph structure encoding features, ensuring that physical connections and transmission paths are embedded in subsequent cross-modal causal inference, and avoiding erroneous causal attribution due to ignoring the network topology.

[0031] Then, the domain text data from the multimodal input data is input into a pre-trained language model, and the hidden states corresponding to the classification labels are extracted as text encoding features. It's important to understand that the classification labels are inserted at the beginning of the domain text data, and the pre-trained language model is a bidirectional encoder representation model fine-tuned by the domain text. Specifically, first, a classification label [CLS] is inserted at the beginning of the domain text data (such as the dispatch instruction "transfer the load from station A to station B" or the fault report "the main transformer oil temperature is abnormally high"). Second, the input sequence is input into a bidirectional encoder representation model fine-tuned by the domain text (such as a BERT model based on the transformer encoder structure). The context-aware hidden state at each position is calculated using a bidirectional self-attention mechanism, and the context-aware hidden state is used as the text encoding feature. The specific formula is as follows: in [Text] represents the text encoding features, LLM represents the pre-trained language model (such as BERT, domain-adaptive GPT), text is the input domain text data, and [CLS] represents the classification label, which serves as the overall representation of the text.

[0032] Finally, a contrastive learning loss function is constructed based on temporal coding features, graph structure coding features, and text coding features, and cross-modal semantic alignment is performed to obtain a shared semantic representation. Specifically, firstly, temporal coding features, graph structure coding features, and text coding features are extracted from three coding branches to construct a cross-modal sample pair set. Different modal representations describing the same physical event (such as the alarm text "main transformer oil temperature too high", the historical curve of the transformer's temperature sensor, and its connection position in the power grid topology) are marked as positive sample pairs, while combinations of different physical events or semantically unrelated modalities are marked as negative sample pairs. Secondly, the cosine similarity score of each sample pair in a unified semantic space is calculated and optimized by introducing a contrastive learning loss function based on the temperature coefficient. This forces the feature vectors of positive sample pairs to move closer to each other in the hyperspace while pushing negative sample pairs further apart, thereby forcibly establishing cross-modal semantic associations such as "high temperature alarm text" and "temperature surge curve". Finally, when the model converges, the three heterogeneous features are projected onto a shared representation space with consistent dimensions and semantic alignment, enabling the subsequent causal discovery module to directly calculate the causal dependency between the text description and the temporal pattern, avoiding information fragmentation and loss of association due to differences in data modalities, and obtaining a shared semantic representation that supports cross-modal causal inference.

[0033] Step S30: Perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength.

[0034] It should be noted that step S30 includes: first, performing attention calculation on the shared semantic representation through neural network channels to obtain an attention weight matrix; obtaining weighted features based on the attention weight matrix; concatenating the weighted features with the variable representations in the shared semantic representation and inputting the concatenation into a multilayer perceptron for causal strength estimation to obtain the first causal strength. Specifically, the shared semantic representation... (in Let d be the feature vector of the i-th variable and d be the dimension of the variable. Input these features into the neural network channels and generate query matrices through learnable linear projections. Key matrix Sum matrix The attention weight of variable i on variable j is calculated using the following formula: in The first part represents the attention weight of variable i on variable j, thereby capturing complex nonlinear dependencies between variables (such as identifying non-monotonic association patterns between temperature surges and equipment failures), breaking through the limitations of traditional linear causal models. Secondly, the attention weight is multiplied by the value vector to obtain a weighted feature, which aggregates the influence strength of variable j on variable i. Finally, the weighted feature is concatenated with the shared semantic representations of variables i and j to form a joint feature vector. This vector is then input into a multilayer perceptron for nonlinear transformation and mapped to probability values ​​through a sigmoid activation function, yielding the first causal strength, as shown in the specific formula: in This represents the first causal strength of the neural network channel estimation, with a value range of [0,1]. represents the Sigmoid activation function; MLP stands for Multilayer Perceptron, used to learn causal strength from attention features; Indicates weighted features, This represents the value vector of variable j. and Let i and j represent the shared semantic representations of variables i and j, respectively. This represents a vector concatenation operation, where i and j represent variable indices.

[0035] Next, a sparsity regularization constraint is applied to the first causal strength to obtain the sparsified causal strength. Specifically, the L1 norm regularization term is calculated for all off-diagonal elements (i.e., variable pairs where i ≠ j) in the first causal strength, with the following formula: in This represents the sparsity regularization loss. This represents the regularization coefficient, which controls the strength of sparsity. Indicates the strength of the first causal relationship; This represents the L1 norm; summing the absolute values ​​produces sparsity. This involves iterating through all variable pairs where i is not equal to j, excluding self-loops. Next, the regularization term is added to the total loss function of the neural network and optimized together with the prediction loss. Gradient descent is used to force the model to automatically compress the causal strength parameters of unimportant connections to near-zero values ​​during training. Finally, after multiple rounds of iterative optimization, the sparsified causal strength is obtained. Only truly significant causal relationships are retained (such as retaining strong connections like "high transformer oil temperature → equipment failure" while suppressing weak or indirect correlations like "air humidity → system voltage" to near zero). This ensures that the causal graph structure is simple and conforms to the sparsity assumption of the physical world (in real systems, variables are usually directly causally related to only a few other variables), avoiding the explosion of causal reasoning complexity and spurious correlation interference caused by excessive connections.

[0036] Then, a kernel-based conditional mutual information test is performed on the shared semantic representation through a statistical inference channel to obtain the second causal strength. Specifically, based on the variable features in the shared semantic representation, a conditional selection algorithm (such as a partial correlation-based heuristic search) is used to identify a set of potential confounding factors S for each pair of variables. This set contains common causal variables that may affect the two variables (such as identifying that "ambient temperature" simultaneously affects "equipment load" and "failure rate"), thereby controlling for spurious correlation interference. Secondly, the conditional mutual information test estimates the conditional dependencies between variables and calculates the conditional mutual information. Finally, the conditional mutual information value is converted into a probabilistic form of causal strength score through a normalization mapping to obtain the second causal strength, with a value range of [0,1].

[0037] Subsequently, adaptive gating weights are calculated based on the first and second causal strengths. Specifically, firstly, the first causal feature representation of the neural network channels is extracted based on the first causal strength. The second causal feature representation of the statistical inference channel is extracted based on the second causal strength. First, the two are concatenated into a joint feature vector; second, a learnable gating network (composed of a weight matrix) is used. and bias vector A parameterized single-layer or multi-layer perceptron performs a linear transformation on the joint feature vector and maps it to adaptive gating weights through a sigmoid activation function. The specific formula is expressed as follows: in Indicates adaptive gating weights, This represents the Sigmoid activation function. This represents the weight matrix of the gated network. This represents the first causal characteristic. This represents the second causal characteristic. This represents the bias vector of the gating network, which dynamically determines the fusion ratio of the two channels based on the distribution characteristics of the input data.

[0038] Finally, the sparsed causal strength and the second causal strength are weighted and fused according to the adaptive gating weights to obtain the fused causal strength. The specific formula is as follows: in This indicates the strength of causal integration. It enables the system to automatically increase the weights of neural network channels in complex nonlinear scenarios (such as the propagation of cascaded faults in power grids) to capture nonlinear interaction patterns, and to automatically increase the weights of statistical channels in small-sample, high-noise scenarios (such as sparse monitoring data under extreme weather conditions) to ensure the statistical rigor of causal discovery, thus achieving data-driven adaptive causal modeling.

[0039] Step S40: Construct a causal knowledge graph based on the fusion causal strength, and calculate the structural change between the current graph structure and the historical graph structure based on the Frobenius norm.

[0040] It should be noted that step S40 includes: determining causal nodes and directed causal edges based on the fusion causal strength, and constructing an initial causal knowledge graph; calculating the structural change between the current graph structure and the historical graph structure of the initial causal knowledge graph based on the Frobenius norm.

[0041] It's important to understand that a causal node is the basic unit that carries business entities or state variables. This unit represents the influencing factors or result objects in a causal relationship. This unit is a fundamental component of a causal knowledge graph. A directed causal edge is a directional connection between two causal nodes, indicating a unidirectional influence from one node to the other. This connection clarifies the direction of causal transmission. The Frobenius norm is a calculation criterion used to measure the overall difference in a matrix. This criterion achieves quantitative evaluation by taking the square root of the sum of the squares of the differences between matrix elements.

[0042] Specifically, significant connections with strength exceeding a preset threshold in the causal relationships are selected as directed causal edges. The corresponding variables are mapped to entity nodes in the graph, and the causal strength value is used as the edge weight attribute to construct an initial causal knowledge graph containing a set of nodes and weighted edges. This solidifies the causal discovery results after double verification into a queryable structured knowledge base. Secondly, the Frobenius norm difference between the current graph structure adjacency matrix and the corresponding matrix of the historical version is calculated. That is, the square root of the sum of the squares of the difference values ​​of all corresponding position elements is used to obtain the structural change. The overall difference of this matrix is ​​used to accurately quantify the magnitude of structural changes caused by power grid line switching or supply chain topology adjustments.

[0043] Step S50: When the amount of structural change exceeds the preset change threshold, perform incremental updates on the affected subgraphs to obtain a dynamic causal knowledge graph.

[0044] It should be noted that the affected subgraph is a local graph region in the initial causal knowledge graph whose relationships have changed due to business changes. This subgraph contains the set of causal nodes and directed causal edges that need to be adjusted.

[0045] Specifically, firstly, the current graph structure adjacency matrix is ​​compared element-wise with the corresponding matrix of the historical version. The sum of squares of all positional differences is calculated and then the square root is taken to obtain the structural change. The degree of topological change is accurately quantified by the overall difference measure of this matrix. Secondly, when the structural change exceeds a preset change threshold, the specific node that has undergone topological change and its neighboring nodes are located to form an affected subgraph set. The change boundary is identified to distinguish between the regions that need to be updated and the regions that remain unchanged. Then, only the nodes in the affected subgraph set are re-performed with causal discovery calculation and their causal representations and connections are updated, while keeping the original representations and edge weights of the unchanged subgraphs completely unchanged. Finally, the updated affected subgraphs and the unchanged subgraphs are reconstructed with topological connections and merged to form a complete dynamic causal knowledge graph. This reduces the quadratic complexity of full graph reconstruction to a linear complexity that is only related to the number of changed nodes, achieving millisecond-level incremental updates to adapt to real-time dynamic changes in the business environment.

[0046] Step S60: Perform vector similarity retrieval and multi-hop causal path traversal in the dynamic causal knowledge graph to obtain the reasoning context.

[0047] It should be noted that vector similarity retrieval is a retrieval method that matches feature vectors based on their similarity. This method quickly locates causal knowledge related to the query content. This method ensures the relevance and timeliness of the retrieval results. Multi-hop causal path traversal is a query method that involves multi-level jumps along directed causal edges. This method uncovers long-distance causal relationships that exist indirectly between nodes. This method comprehensively obtains the complete causal transmission chain.

[0048] Further, step S60 includes: converting causal nodes in the dynamic causal knowledge graph into fixed-dimensional semantic vectors; performing vector similarity retrieval in the semantic vector space according to reasoning needs to filter target causal nodes; starting from the target causal node, performing multi-level, multi-hop causal path traversal along directed causal edges to obtain traversal results; and structurally integrating causal path and causal strength information according to the traversal results to obtain the reasoning context.

[0049] Specifically, after the dynamic causal knowledge graph is constructed, inference requirements are generated based on the real-time state and decision-making objectives of the current business scenario. First, key state variables, abnormal event information, and decision-making issues of the current scenario are extracted from the real-time collected multimodal input data. Second, by combining the business objectives and the causal issues to be analyzed, query intents are formed for locating root causes, analyzing the transmission of influences, or evaluating the effectiveness of interventions. Finally, the key state variables, abnormal event information, and query intents are structurally encapsulated to obtain inference requirements for driving knowledge retrieval. Next, each causal node in the dynamic causal knowledge graph (such as event entities like "main transformer oil temperature too high", "cooling system failure", and "line overload trip") is mapped to a fixed-dimensional semantic vector. This vector integrates the node's attribute features with the graph's topological location information, ensuring that semantically similar nodes are close in the vector space, providing a mathematical foundation for efficient subsequent retrieval. Then, based on reasoning needs, query text or conditional variables are encoded into query vectors. The cosine similarity between this query vector and the semantic vectors of all nodes is calculated and sorted. Target causal nodes with similarity exceeding a preset threshold are selected as the starting point for retrieval, thereby quickly locating evidence variables and potential intervention points relevant to the current decision-making scenario. Finally, starting from each target causal node, a multi-level, multi-hop causal path traversal is performed along the directed causal edges, that is, exploring downstream result nodes layer by layer along the outgoing edge direction. The process involves tracing upstream causal nodes layer by layer along the incoming edge direction, recording the node sequence and edge weights on each access path, and calculating the overall confidence score of the entire path by continuously multiplying the causal strength probability values ​​of each edge on the path. This constructs a complete transmission chain from root cause to result (e.g., a three-hop path from "equipment overheating" to "cooling system failure" and then to "power interruption"). Finally, the process collects all traversed multi-hop causal path sequences, node attributes on the path, edge weights, and overall confidence scores of the path. It then structurally integrates the causal path and causal strength information, clearly identifying the causal transmission relationship and quantified confidence scores between evidence variables, intervention variables, and target result variables. This forms a reasoning context containing complete causal logic and probabilistic support, providing interpretable causal basis and precise knowledge support for subsequent constraint projection and decision generation.

[0050] Step S70: Perform constraint projection and counterfactual verification based on the reasoning context to obtain a decision scheme that satisfies physical constraints and business rules.

[0051] It's important to note that constraint projection is the operation of mapping the initial decision result to a feasible domain that satisfies the constraints. This operation adjusts decision parameters to ensure the result conforms to preset limitations, avoiding violations or unenforceable decisions. Counterfactual verification is a verification method that involves hypothesizing an intervention scenario and comparing it with actual observations. This method examines the causal rationality of the decision-making scheme and determines whether the decision can truly bring about the expected changes. Physical constraints are hard limitations determined by equipment characteristics and physical laws. These constraints reflect the inviolable physical boundaries in the scenario and are the fundamental prerequisites that the decision-making scheme must meet. Business rules are normative clauses formed by business processes and management requirements. These rules ensure that decisions conform to business logic and management objectives, ensuring that decisions can be implemented and executed in real-world scenarios.

[0052] Specifically, firstly, physical constraints (such as the capacity limit of power grid transmission lines and equipment temperature safety thresholds) and business rules (such as scheduling operation priority procedures and security isolation requirements) are formalized into a set of differentiable constraint functions. Simultaneously, based on the inference context, an initial sequence of candidate decisions is generated using a language model, thus transforming hard constraints into computable terms in the optimization problem. Secondly, the degree of violation of each constraint function by the candidate decisions is calculated. By modifying linear units, the violation values ​​are converted into penalty terms and combined with the probabilities generated by the language model to construct a joint optimization objective. Constraint satisfaction requirements are directly incorporated into the decoding and generation process rather than through post-hoc checks and corrections, thereby ensuring that each generated candidate decision strictly satisfies both physical feasibility and business requirements. To ensure compliance and avoid repeated retries and efficiency losses due to post-verification failures, the candidate decisions are counterfactually verified based on a pre-built structural causal model. This involves calculating the difference between the conditional probability distribution of the target outcome when a specific variable is forcibly intervened and the probability distribution when that variable is naturally observed. For example, the difference between forcibly reducing the load on a line and actually observing the system stability probability under low load is calculated. Finally, the strength of the causal effect is assessed by comparing the degree of difference between the two probability distributions. Decisions with significant differences and genuine causal necessity are selected as the final output, while pseudo-effective decisions based solely on correlation but lacking causal support are eliminated. This results in decision schemes that meet physical constraints and business rules and have reliable causal basis.

[0053] This embodiment constructs an integrated intelligent decision-making framework from multimodal data, achieves shared semantic representation of multi-source data through unified encoding, obtains reliable causal strength through dual-channel causal discovery, constructs and dynamically updates a causal knowledge graph based on this, and generates compliant decisions by combining constraint projection and counterfactual verification. The beneficial effects include realizing dynamic causal knowledge updates and generating constraint-compliant and reliable decisions based on multimodal data, ensuring that decisions strictly comply with physical and business rules, and improving the reliability, robustness, and real-time performance of decisions in complex scenarios.

[0054] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The multimodal temporal causal intelligent decision-making method for complex scenarios, step S50, further includes steps S201 to S207: Step S201: Formalize the physical constraints and business rules into a set of differentiable constraint functions.

[0055] Specifically, firstly, physical constraints (such as the upper limit of power transmission in power grid transmission lines, the safe threshold of transformer oil temperature, and the stable range of bus voltage) and business rules (such as the priority ranking of dispatch instructions, equipment maintenance window requirements, and load transfer operation procedures) are mapped one by one to mathematical functions of decision variables. Each function outputs a numerical value representing the degree of violation of the corresponding constraint (such as the difference between the actual load of the line and the capacity limit, and the deviation between the planned operation time and the allowable window). Secondly, it ensures that all constraint functions are continuously differentiable within their domain. For absolute value-type non-smooth constraints, a smoothing approximation is used (such as replacing hard truncation with Huber loss) to make the gradient of the constraint function with respect to the decision variables computable, thereby supporting... End-to-end optimization is based on gradient descent. Then, a corresponding upper limit threshold is set for each constraint function (e.g., line capacity usage must not exceed 80%, and operation time deviation must not exceed 10 minutes). The constraint satisfaction requirements are transformed into penalty term calculation (when a decision violates the constraint, a penalty value proportional to the degree of violation is generated by modifying the linear unit; when the constraint is satisfied, the penalty value is zero). Finally, each constraint function and its corresponding threshold parameter are encapsulated into a set of differentiable constraint functions, which can calculate the degree of constraint violation of candidate decisions in real time during the language model decoding process and backpropagate the gradient. This embeds hard constraints into the objective function of generative optimization, realizing the synchronous generation of decisions and the satisfaction of constraints rather than post-event checks and corrections.

[0056] Step S202: Generate an initial decision sequence based on the reasoning context using a large language model to obtain the initial decision probability distribution.

[0057] Specifically, firstly, the reasoning context is encoded into a structured prompt input large language model. This large language model is an architecture combining a bidirectional encoder representation model fine-tuned by domain text and an autoregressive generation model. The bidirectional encoder is responsible for understanding the semantics of the causal context, while the autoregressive generation module is responsible for generating decision sequences word by word. This model is obtained through supervised fine-tuning using specific texts from fields such as power dispatch logs, supply chain decision records, and equipment operation and maintenance procedures after pre-training on massive amounts of general text. Secondly, the model generates candidate decision sequences autoregressively based on conditional probabilities. That is, in each generation step, the conditional probability of each word in the vocabulary is calculated and sampled, gradually constructing a complete decision text. At the same time, the joint probability of the sequence is accumulated from each step as the initial decision probability distribution. Finally, the top-ranked candidate sequences are retained to form an initial decision scheme set, providing a decision basis that is both causal and semantically reasonable for subsequent constraint projection. Thus, the domain-adapted large language model can quickly generate feasible candidate schemes, and the retained probability distribution also provides a differentiable optimization objective for constraint optimization.

[0058] Step S203: Calculate the degree of constraint violation of the initial decision sequence based on the set of differentiable constraint functions.

[0059] Specifically, the initial decision sequence is first parsed into structured decision variables (such as load transfer amount, equipment switching status, and operation time point). The values ​​of each variable are then input into the corresponding differentiable constraint functions (such as the line capacity constraint function to calculate the difference between the actual load and the upper limit, and the time window constraint function to calculate the deviation between the planned operation and the allowed time period). Second, each constraint function outputs a scalar value that quantifies the degree of violation (positive values ​​indicate violation, zero or negative values ​​indicate satisfaction). A larger penalty weight is set for hard constraints (such as the upper limit of equipment temperature safety), and a smaller weight is set for soft constraints (such as economic cost preference), thereby distinguishing the priority of constraints. Finally, the violation degree outputs of all constraint functions are summarized to form a constraint violation vector. Each dimension of this vector corresponds to the violation quantification value of different constraint types, providing a differentiable gradient signal for subsequent penalty term calculations. This enables the optimization process to identify which decision variables need to be adjusted to prioritize the satisfaction of key constraints.

[0060] Step S204: Calculate the constraint penalty term by modifying the linear unit based on the degree of constraint violation and the preset penalty coefficient.

[0061] It should be noted that the degree of constraint violation refers to the deviation between the values ​​of each variable in the initial decision sequence and the range specified by the differentiable constraint function, used to quantify the degree of constraint violation by the decision. The modified linear unit is an activation function used to perform a nonlinear transformation on the degree of constraint violation, avoiding excessively large or small penalty terms and ensuring a stable optimization process.

[0062] Specifically, the degree of constraint violation of each constraint function is first input into a modified linear unit for nonlinear transformation. This unit outputs the original value for positive violations and zero for non-positive violations. The specific formula is expressed as follows: in This indicates the output of the corrected linear unit. Represents the maximum value function. Let represent the quantization function for the i-th constraint, and output the degree of constraint violation in decision Y under that constraint. This represents the preset upper limit of the constraint, so that a penalty is only incurred when the decision actually violates the constraint, and no penalty is incurred when the constraint is satisfied; secondly, the output of the modified linear unit is multiplied by the preset penalty coefficient and summed to construct the constraint penalty term, the specific formula of which is: in This represents summing over all constraints. Indicates a constraint or penalty item. This represents the preset penalty coefficient for the i-th constraint.

[0063] Step S205: Construct a joint optimization objective based on the initial decision probability distribution and constraint penalty terms.

[0064] It should be noted that the initial decision probability distribution generated by the language model is converted into a logarithmic probability form. That is, the conditional probability of each candidate word is taken as logarithm and then summed along the sequence length. This transforms the original probability product operation into a logarithmic probability summation operation, which facilitates a numerically stable linear combination with the penalty term. Secondly, the logarithmic probability is subtracted from the constraint penalty term to construct a joint optimization objective. The constraint penalty term is introduced in the form of a minus sign, so that the larger the penalty value, the smaller the overall objective value. This naturally balances two requirements in maximizing the objective: maintaining a high probability generated by the language model to ensure the semantic rationality and fluency of the decision, and reducing the constraint penalty term to ensure physical feasibility and business compliance. Finally, based on this joint optimization objective, end-to-end constraint optimization decoding is performed on the initial decision sequence. Gradient information is used to guide the adjustment direction of decision variables. Constraint feedback is responded to in real time during the decoding and generation process. Finally, candidate decision schemes that simultaneously satisfy semantic rationality and constraint strictness are output. This realizes a methodological shift from "checking and correcting after generation" to "embedding satisfaction during generation", avoiding repeated retries and efficiency losses caused by constraint conflicts.

[0065] Step S206: Based on the joint optimization objective, the initial decision sequence is subjected to constrained optimization decoding based on the Lagrange multiplier method to obtain candidate decision schemes.

[0066] Specifically, firstly, Lagrange multiplier variables are introduced to transform the constrained optimization problem into an unconstrained Lagrange function form. This involves multiplying each constraint by its corresponding multiplier coefficient and incorporating the result into the joint optimization objective, transforming the original constraints into penalty adjustment terms within the objective function. Secondly, the decision sequence and Lagrange multipliers are iteratively optimized alternately. When the multipliers are fixed, gradient descent is used to adjust the decision variables to reduce constraint violations and increase language model probabilities. When the decision is fixed, the multipliers are updated to amplify the penalty weights for severe constraint violations, thus dynamically balancing constraint satisfaction and generation quality during optimization. Then, at each generation step of decoding, the corrected scores of each candidate lexical are calculated based on the current multiplier weights. Lexicals with high language model probabilities and fewer constraint penalties are prioritized, gradually constructing a complete decision sequence. Finally, when the iteration converges or reaches a preset number of rounds, the optimized decision sequence is output as a candidate decision scheme. This scheme maintains semantic rationality and operational feasibility while satisfying physical constraints and business rules, achieving deep integration of constraints and the generation process rather than post-selection and elimination.

[0067] Step S207: Based on the structural causal model, the candidate decision schemes are processed to obtain decision schemes that satisfy physical constraints and business rules.

[0068] It should be noted that the structural causal model is a pre-constructed model that includes integrated causal relationships. Step S207 includes: calculating candidate decision schemes based on the structural causal model to obtain a first probability distribution under intervention conditions and a second probability distribution under natural observation conditions, wherein the intervention variable value is forcibly set by executing an operator under intervention conditions; comparing the difference between the first probability distribution and the second probability distribution to obtain the causal effect difference result; when the causal effect difference result meets the preset causal validity condition, retaining the corresponding candidate decision scheme to obtain the screened scheme; and determining the decision scheme that meets the physical constraints and business rules based on the screened scheme.

[0069] Specifically, firstly, counterfactual calculations are performed on candidate decision schemes based on a pre-built structural causal model. By executing operators, the intervention variable is forced to take the value specified by the candidate decision, severing its connection with its natural parent node and fixing its value, simulating an external forced intervention scenario. The first probability distribution of the target outcome variable under this intervention condition is then calculated. Secondly, keeping the causal structure unchanged, only the value of the intervention variable is input into the model as the observation condition, calculating the second probability distribution of the target outcome variable under natural observation conditions. This distribution reflects the statistical correlation between variables rather than the causal effect. Finally, the difference between the first and second probability distributions is calculated as the causal effect difference result. By comparing the deviation of the outcome distribution when the decision is enforced and when the same behavior is observed naturally, the greater the difference, the stronger the causal necessity of the intervention variable for the target outcome. Finally, when the difference in causal effect meets the preset causal validity condition (such as the difference value exceeding the preset threshold), the candidate decision scheme is determined to have real causal support rather than just being based on spurious correlation. The scheme is retained in the set of screened schemes. Finally, the scheme with the most significant causal effect is selected from the screened schemes to determine the decision scheme that meets physical constraints and business rules. This eliminates invalid decisions that seem reasonable but lack causal basis, ensuring that the output scheme has reliable causal validity on the basis of physical feasibility and business compliance.

[0070] This embodiment formalizes physical constraints and business rules as differentiable functions. An initial decision is generated using a large language model, and a joint optimization objective is constructed by combining constraint penalty terms. End-to-end constraint optimization decoding is performed using the Lagrange multiplier method to obtain candidate solutions. Then, counterfactual verification is performed based on a structural causal model to screen causally valid decisions. This achieves simultaneous constraint satisfaction and decision generation, ensuring that decisions satisfy both physical constraints and business rules, and improving causal rationality through counterfactual verification.

[0071] Based on the first embodiment of this application, this application also provides a multimodal temporal causal intelligent decision-making device for complex scenarios. Please refer to... Figure 3 The device includes: The acquisition module 10 is used to acquire multimodal input data, which includes numerical time series data, topological graph structure data, and domain text data.

[0072] The encoding module 20 is used to uniformly encode the multimodal input data to obtain a shared semantic representation.

[0073] Discovery module 30 is used to perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength.

[0074] The computation module 40 is used to construct a causal knowledge graph based on the fusion causal strength and to calculate the structural changes between the current graph structure and the historical graph structure based on the Frobenius norm.

[0075] The update module 50 is used to perform incremental updates on the affected subgraphs to obtain a dynamic causal knowledge graph when the amount of structural change exceeds a preset change threshold.

[0076] Processing module 60 is used to perform vector similarity retrieval and multi-hop causal path traversal in the dynamic causal knowledge graph to obtain the reasoning context.

[0077] The results module 70 is used to perform constraint projection and counterfactual verification based on the reasoning context to generate decision schemes that satisfy physical constraints and business rules.

[0078] The multimodal temporal causal intelligent decision-making device for complex scenarios provided in this application employs the multimodal temporal causal intelligent decision-making method for complex scenarios described in the above embodiments, and can solve the technical problem of how to achieve dynamic causal knowledge updates and generate constraint-compliant and reliable decisions based on multimodal data. Compared with the prior art, the beneficial effects of the multimodal temporal causal intelligent decision-making device for complex scenarios provided in this application are the same as those of the multimodal temporal causal intelligent decision-making method for complex scenarios provided in the above embodiments, and other technical features in the multimodal temporal causal intelligent decision-making device for complex scenarios are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0079] This application provides a multimodal temporal causal intelligent decision-making device for complex scenarios. The multimodal temporal causal intelligent decision-making device for complex scenarios includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multimodal temporal causal intelligent decision-making method for complex scenarios in the above embodiment 1.

[0080] The following is for reference. Figure 4 This document illustrates a structural schematic diagram of a multimodal temporal causal intelligent decision-making device suitable for implementing embodiments of this application in complex scenarios. The multimodal temporal causal intelligent decision-making device for complex scenarios in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4The multimodal temporal causal intelligent decision-making device for complex scenarios shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0081] like Figure 4 As shown, a multimodal temporal causal intelligent decision-making device for complex scenarios may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multimodal temporal causal intelligent decision-making device for complex scenarios. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multimodal temporal causal intelligent decision-making device for complex scenarios to communicate wirelessly or wiredly with other devices to exchange data. Although various multimodal temporal causal intelligent decision-making devices for complex scenarios are shown in the figures, it should be understood that it is not required to implement or possess all of them. More or fewer may be implemented alternatively.

[0082] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0083] The multimodal temporal causal intelligent decision-making device for complex scenarios provided in this application, employing the multimodal temporal causal intelligent decision-making method for complex scenarios described in the above embodiments, can solve the technical problem of how to achieve dynamic causal knowledge updates and generate constraint-compliant and reliable decisions based on multimodal data. Compared with the prior art, the beneficial effects of the multimodal temporal causal intelligent decision-making device for complex scenarios provided in this application are the same as those of the multimodal temporal causal intelligent decision-making method for complex scenarios provided in the above embodiments, and other technical features in this multimodal temporal causal intelligent decision-making device for complex scenarios are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0084] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0086] This application provides a computer-readable medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multimodal temporal causal intelligent decision-making method for complex scenarios in the above embodiments.

[0087] The computer-readable medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable medium may be any tangible medium containing or storing a program that can be executed by instructions, used by a device, or used in conjunction with it. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0088] The aforementioned computer-readable medium may be included in a multimodal temporal causal intelligent decision-making device for complex scenarios; or it may exist independently and not be assembled into a multimodal temporal causal intelligent decision-making device for complex scenarios.

[0089] The aforementioned computer-readable medium carries one or more programs that, when executed by a multimodal temporal causal intelligent decision-making device for complex scenarios, enable the device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, all blocks in the flowcharts or block diagrams may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that all blocks in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0091] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0092] The readable medium provided in this application is a computer-readable medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described multimodal temporal causal intelligent decision-making method for complex scenarios. This solves the technical problem of how to achieve dynamic causal knowledge updates and generate constraint-compliant and reliable decisions based on multimodal data. Compared with the prior art, the beneficial effects of the computer-readable medium provided in this application are the same as those of the multimodal temporal causal intelligent decision-making method for complex scenarios provided in the above embodiments, and will not be elaborated upon here.

[0093] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal temporal causal intelligent decision-making method for complex scenarios as described above.

[0094] The computer program product provided in this application can solve the technical problem of how to achieve dynamic causal knowledge updating and generate constraint-compliant and reliable decisions based on multimodal data. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multimodal temporal causal intelligent decision-making method for complex scenarios provided in the above embodiments, and will not be repeated here.

[0095] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A multimodal temporal causal intelligent decision-making method for complex scenarios, characterized in that, The method includes: Acquire multimodal input data, wherein the multimodal input data includes numerical time series data, topological graph structure data, and domain text data; The multimodal input data is uniformly encoded to obtain a shared semantic representation; Perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength; A causal knowledge graph is constructed based on the fused causal strength, and the structural change between the current graph structure and the historical graph structure is calculated based on the Frobenius norm. When the amount of structural change exceeds a preset change threshold, incremental updates are performed on the affected subgraphs to obtain a dynamic causal knowledge graph. In a dynamic causal knowledge graph, vector similarity retrieval and multi-hop causal path traversal are performed to obtain the reasoning context. Based on the inference context, constraint projection and counterfactual verification are performed to obtain a decision scheme that satisfies both physical constraints and business rules.

2. The method as described in claim 1, characterized in that, The step of uniformly encoding the multimodal input data to obtain a shared semantic representation includes: The numerical time-series data in the multimodal input data are subjected to time-series reprogramming and structured state-space model encoding to obtain time-series encoded features; Multi-layer graph neural network message passing is performed on the topological graph structure data in the multimodal input data, and attention-weighted aggregation is performed on the node neighbor features to obtain graph structure encoding features; The domain text data in the multimodal input data is input into a pre-trained language model, and the hidden states corresponding to the classification labels are extracted as text encoding features. The classification labels are inserted at the beginning of the domain text data. The pre-trained language model is a bidirectional encoder representation model fine-tuned by the domain text. Based on the temporal coding features, graph structure coding features, and text coding features, a contrastive learning loss function is constructed and cross-modal semantic alignment is performed to obtain a shared semantic representation.

3. The method as described in claim 2, characterized in that, The step of performing temporal reprogramming and structured state-space model encoding on the numerical time-series data in the multimodal input data to obtain temporal encoded features includes: The numerical time-series data is processed into time-series blocks to obtain multiple time-series data blocks; The temporal data blocks are mapped and transformed through a learnable reprogrammable layer to obtain semantic token representations; The input-dependent dynamic state transition matrix, input matrix, and output matrix are calculated based on the semantic token representation using a selective scanning mechanism, wherein the selective scanning mechanism is an adaptive selection of state space parameters based on the semantic token representation. Based on the dynamic state transition matrix, input matrix, and output matrix, a structured state-space dual computation is performed on the semantic token representation matrix to obtain the temporal coding features.

4. The method as described in claim 1, characterized in that, The step of performing dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength includes: Attention calculations are performed on the shared semantic representation through neural network channels to obtain an attention weight matrix; Weighted features are obtained based on the attention weight matrix; The weighted features are concatenated with the variable representations in the shared semantic representation and input into a multilayer perceptron for causal strength estimation to obtain the first causal strength. Apply sparsity regularization constraints to the first causal strength to obtain the sparsified causal strength; The second causal strength is obtained by performing a kernel-based conditional mutual information test on the shared semantic representation through a statistical inference channel. The adaptive gating weights are calculated based on the first causal strength and the second causal strength. The sparsed causal strength and the second causal strength are weighted and fused according to the adaptive gating weights to obtain the fused causal strength.

5. The method as described in claim 1, characterized in that, The steps of performing vector similarity retrieval and multi-hop causal path traversal in the dynamic causal knowledge graph to obtain the inference context include: Convert causal nodes in a dynamic causal knowledge graph into fixed-dimensional semantic vectors; Based on reasoning requirements, perform vector similarity retrieval in the semantic vector space to filter target causal nodes; Starting from the target causal node, perform a multi-level, multi-hop causal path traversal along the directed causal edges to obtain the traversal result; Based on the traversal results, the causal path and causal strength information are structurally integrated to obtain the reasoning context.

6. The method as described in claim 1, characterized in that, The step of performing constraint projection and counterfactual verification based on the inference context to obtain a decision scheme that satisfies physical constraints and business rules includes: Physical constraints and business rules are formalized into a set of differentiable constraint functions; Based on the reasoning context, an initial decision sequence is generated using a large language model to obtain the initial decision probability distribution; The degree of constraint violation of the initial decision sequence is calculated based on the set of differentiable constraint functions; The constraint penalty term is calculated using a modified linear unit based on the degree of constraint violation and the preset penalty coefficient. Construct a joint optimization objective based on the initial decision probability distribution and the constraint penalty term; Based on the joint optimization objective, the initial decision sequence is subjected to constrained optimization decoding based on the Lagrange multiplier method to obtain candidate decision schemes; The candidate decision schemes are processed based on the structural causal model to obtain decision schemes that satisfy physical constraints and business rules, wherein the structural causal model is a pre-built model that includes fused causal strength.

7. The method as described in claim 6, characterized in that, The step of processing the candidate decision schemes based on the structural causal model to obtain decision schemes that satisfy physical constraints and business rules includes: The candidate decision schemes are calculated based on the structural causal model to obtain a first probability distribution under intervention conditions and a second probability distribution under natural observation conditions, wherein the intervention conditions are the values ​​of intervention variables that are forcibly set by executing operators. By comparing the difference between the first probability distribution and the second probability distribution, the causal effect difference results are obtained; When the difference in causal effect meets the preset causal validity condition, the corresponding candidate decision scheme is retained to obtain the filtered scheme. The selected solutions are determined to be the decision-making solutions that meet both physical constraints and business rules.

8. A multimodal temporal causal intelligent decision-making device for complex scenarios, characterized in that, The device includes: The acquisition module is used to acquire multimodal input data, wherein the multimodal input data includes numerical time series data, topological graph structure data, and domain text data; The encoding module is used to uniformly encode the multimodal input data to obtain a shared semantic representation; The discovery module is used to perform dual-channel causal discovery on the shared semantic representation to obtain the fused causal strength; The calculation module is used to construct a causal knowledge graph based on the fused causal strength, and to calculate the structural change between the current graph structure and the historical graph structure based on the Frobenius norm; The update module is used to perform incremental updates on the affected subgraphs to obtain a dynamic causal knowledge graph when the amount of structural change exceeds a preset change threshold. The processing module is used to perform vector similarity retrieval and multi-hop causal path traversal in the dynamic causal knowledge graph to obtain the reasoning context; The results module is used to perform constraint projection and counterfactual verification based on the inference context to generate a decision scheme that satisfies physical constraints and business rules.

9. A multimodal temporal causal intelligent decision-making device for complex scenarios, characterized in that, The device includes: a memory, a processor, and a multimodal temporal causal intelligent decision-making program for complex scenarios stored in the memory and running on the processor, wherein the multimodal temporal causal intelligent decision-making program for complex scenarios is configured to implement the steps of the multimodal temporal causal intelligent decision-making method for complex scenarios as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a multimodal temporal causal intelligent decision-making program for complex scenarios. When the multimodal temporal causal intelligent decision-making program for complex scenarios is executed by the processor, it implements the steps of the multimodal temporal causal intelligent decision-making method for complex scenarios as described in any one of claims 1-7.

Images