Heterogeneous data integration method and system based on multi-modal AI

Abstract

The application discloses a multi-modal AI-based heterogeneous data integration method and system, relates to the technical field of data integration processing, and first extracts and splices features of infrared image sequences, visible light video sequences and gas concentration sequences based on a two-dimensional convolution network, a three-dimensional convolution network and a long short-term memory network to obtain a multi-modal initial feature tensor, then performs symmetric graph representation and semantic divergence evaluation to obtain a semantic divergence matrix, further performs dynamic gating and topological reconstruction based on a preset threshold and a physical prior causal matrix to generate a causal topological graph, and finally performs asymmetric feature aggregation, and finally outputs a device state label through global pooling and state arbitration. In this way, by embedding the physical causal constraint into the multi-modal fusion process, the problem that key abnormal features are statistically suppressed when there is a semantic conflict between modes is effectively solved, and the decision robustness of device safety monitoring under extreme working conditions is improved.

Description

Technical Field

[0001] This application relates to the field of data integration and processing technology, and more specifically, to a heterogeneous data integration method and system based on multimodal AI. Background Technology

[0002] In the field of safety monitoring of critical power equipment such as high-voltage transformers in new energy power plants, infrared thermal imagers, visible light cameras, and gas sensors deployed on-site continuously perceive the operating status of the equipment from three physical dimensions: thermal distribution, appearance, and gas concentration. However, these heterogeneous data differ significantly in sampling frequency, data dimensionality, and semantic representation. Existing late-stage fusion architectures and graph neural network ensemble methods essentially treat each modality as causally parallel equivalent entities, employing a bidirectional symmetrical equal aggregation strategy. When semantic conflicts occur between modalities, higher-dimensional modalities suppress lower-dimensional modalities carrying key threshold information through statistical advantages, leading to systematic desensitization of early latent fault characteristics in the ensemble model.

[0003] In fact, in transformer safety monitoring scenarios, the three heterogeneous modes follow a strict physical evolution chain: internal thermal accumulation is the physical root cause, continuous overheating induces the degradation of insulation materials and releases volatile gases, ultimately leading to macroscopic morphological changes such as deformation or smoke from the equipment casing. When physical state contradictions arise between modes, the existing symmetrical aggregation architecture cannot adaptively arbitrate conflicts based on this causal logic chain. It cannot identify which mode represents the true signal closer to the physical root cause, nor can it prevent the reverse dilution of the abnormal characteristics of the root mode by the apparent mode. As a result, it faces the structural risk of causal misjudgment under atypical operating conditions such as sensor drift and environmental interference, which seriously restricts the decision robustness and safety reliability of the multimodal integrated system. Summary of the Invention

[0004] This application provides a heterogeneous data integration method and system based on multimodal AI, aiming to solve the technical problem that existing multimodal integration architectures cannot adaptively resolve conflicts based on objective causal logic when physical state contradictions occur between modalities due to the adoption of symmetric aggregation strategies.

[0005] According to a first aspect of this application, a heterogeneous data integration method based on multimodal AI is provided, comprising: Step 1, performing independent modal feature extraction and feature concatenation on infrared image sequences, visible light video sequences, and gas concentration sequences based on two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks to obtain a multimodal initial feature tensor; Step 2, performing symmetric graph representation and semantic divergence evaluation on the multimodal initial feature tensor to obtain a semantic divergence matrix; Step 3, performing dynamic divergence gating and physical topology reconstruction on the semantic divergence matrix based on a preset physical common sense tolerance threshold and a physical prior causal matrix to obtain a causal topology graph; Step 4, performing asymmetric feature aggregation on the multimodal initial feature tensor based on the reconstructed topology based on the causal topology graph to obtain a causal fusion feature tensor; Step 5, performing global feature pooling and final state arbitration on the causal fusion feature tensor to obtain a device status label for device safety monitoring.

[0006] According to a second aspect of this application, a heterogeneous data integration system based on multimodal AI is provided, comprising: a multimodal feature extraction module, used to extract and concatenate independent modal features from infrared image sequences, visible light video sequences, and gas concentration sequences based on two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks, respectively, to obtain a multimodal initial feature tensor; a semantic divergence evaluation module, used to perform symmetric graph representation and semantic divergence evaluation on the multimodal initial feature tensor to obtain a semantic divergence matrix; a causal topology reconstruction module, used to perform dynamic divergence gating and physical topology reconstruction on the semantic divergence matrix based on a preset physical common sense tolerance threshold and a physical prior causal matrix to obtain a causal topology graph; an asymmetric feature aggregation module, used to perform asymmetric feature aggregation on the multimodal initial feature tensor based on the reconstructed topology based on the causal topology graph to obtain a causal fusion feature tensor; and a state arbitration output module, used to perform global feature pooling and final state arbitration on the causal fusion feature tensor to obtain a device status label for device safety monitoring.

[0007] Compared with existing technologies, this application provides a heterogeneous data integration method and system based on multimodal AI. By introducing semantic divergence gating and physical causal topology reconstruction mechanisms into the multimodal graph integration architecture, the integration system can adaptively convert the symmetric graph structure into a causal directed graph structure when semantic conflicts between modalities are detected. This fundamentally blocks noise interference and feature dilution paths in the reverse causal direction, ensuring the dominant position of anomalous signals from the physical root modality in the fusion process. Compared with existing multimodal integration methods based on weighted averaging or symmetric message passing, this application can significantly reduce the probability of systematic misjudgment caused by ignoring physical causal logic under atypical conditions such as sensor drift and sudden changes in ambient temperature. It effectively improves the decision accuracy and safety reliability of the multimodal heterogeneous data integration system under extreme and boundary conditions. At the same time, the introduction of causal topology enables the final state judgment result to have an interpretable capability that can be traced along the physical evolution chain, meeting the stringent requirements of decision transparency and auditability in industrial-grade safety monitoring scenarios. Attached Figure Description

[0008] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 This is a flowchart illustrating the heterogeneous data integration method based on multimodal AI proposed in this application.

[0010] Figure 2 This is a schematic diagram of the data flow for the heterogeneous data integration method based on multimodal AI proposed in this application.

[0011] Figure 3 This is a flowchart illustrating step one of the heterogeneous data integration methods based on multimodal AI proposed in this application.

[0012] Figure 4 This is a flowchart illustrating step two of the heterogeneous data integration method based on multimodal AI proposed in this application.

[0013] Figure 5 This is a flowchart of the heterogeneous data integration system based on multimodal AI of this application. Detailed Implementation

[0014] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0015] like Figure 1 and Figure 2 As shown, this application provides a heterogeneous data integration method based on multimodal AI, including: Step one involves extracting and concatenating modal features independently from infrared image sequences, visible light video sequences, and gas concentration sequences using two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks to obtain multimodal initial feature tensors. It should be understood that in the safety monitoring scenario of high-voltage transformers at new energy power plants, the infrared thermal imager, visible light camera, and gas sensor deployed on-site continuously perceive the equipment's operating status from three physical dimensions: thermal distribution, appearance, and gas concentration. The collected infrared image sequences, visible light video sequences, and gas concentration sequences differ significantly in data dimensions, sampling frequency, and semantic expression. Infrared image sequences are temporal stacks of two-dimensional static frames, with their core information contained in the spatial temperature distribution of each frame. Visible light video sequences are continuous spatiotemporal data volumes with three dimensions: height, width, and time, with their core information encompassing both spatial geometry and dynamic deformation in the temporal dimension. Gas concentration sequences, on the other hand, are continuous sampling of one-dimensional scalar signals along the time axis, with their core information contained in the gradient evolution of concentration values ​​over time. Since the three types of heterogeneous data cannot be directly subjected to unified mathematical operations or feature comparisons in their original form, it is necessary to select a matching deep learning network architecture for each modality's physical characteristics to perform independent feature extraction. After extraction, the features of each modality are uniformly mapped to the same latent space dimension and concatenated along the modality dimension to generate a multimodal initial feature tensor that can carry the full amount of multi-physics information, providing a unified data basis for subsequent symmetric graph representation and semantic divergence evaluation.

[0016] In one embodiment of this application, such as Figure 3 As shown, step one includes: Step 1.1, based on a two-dimensional convolutional network, performing layer-by-layer local receptive field scanning and feature extraction of the spatial temperature distribution of the infrared image sequence to obtain infrared spatial features; Step 1.2, based on a three-dimensional convolutional network, performing spatiotemporal geometric transformation feature extraction along the three dimensions of height, width, and time on the visible light video sequence to obtain visual spatiotemporal features; Step 1.3, based on a long short-term memory network, performing gated hidden state iterative encoding on the gas concentration sequence according to the time step and extracting the hidden layer output of the final time step to obtain gas temporal features; Step 1.4, performing linear mapping of the infrared spatial features, visual spatiotemporal features, and gas temporal features in a unified dimension through independent fully connected projection layers, and concatenating and merging them along the modal dimension to obtain the multimodal initial feature tensor.

[0017] First, step 1.1 is executed. Based on a two-dimensional convolutional network, the infrared image sequence is scanned layer by layer to obtain infrared spatial features by scanning the local receptive field of spatial temperature distribution. Specifically, the infrared image sequence is input into a preset two-dimensional convolutional module based on a residual network. The two-dimensional convolutional kernel is used to perform a sliding scan of the local feature receptive field in the spatial domain. Two-dimensional discrete convolution operations are performed on the infrared image sequence with a bias term added. Then, a nonlinear activation operation is performed using a modified linear unit to extract the infrared spatial features characterizing the thermal distribution pattern. The calculation process can be expressed as follows: in, Infrared spatial characteristics, The input matrix is ​​the result of converting the infrared image sequence into a tensor. This represents the learnable two-dimensional weight matrix in the convolutional layer of a two-dimensional residual network. It is a two-dimensional discrete convolution operator. This represents the bias vector corresponding to a two-dimensional convolutional network layer. To correct the nonlinear activation function of the linear unit. The reason why two-dimensional convolutional networks are suitable for infrared image sequences is that each frame of an infrared thermal imager is essentially a two-dimensional temperature matrix, where the value of each pixel corresponds to the radiation temperature value of the corresponding area on the surface of the device. The two-dimensional convolutional kernel can capture the spatial pattern of temperature gradient distribution in a local receptive field, such as the shape and contour of local hot spots, the density distribution of temperature contour lines, and the spatial features such as the temperature difference gradient between adjacent areas.

[0018] Secondly, step 1.2. Based on a 3D convolutional network, spatiotemporal geometric transformation features along the three dimensions of height, width, and time are extracted from the visible light video sequence to obtain visual spatiotemporal features. Specifically, the visible light video sequence is input into the 3D convolutional neural network module, and the 3D convolutional kernel slides along the three dimensions of height, width, and time to simultaneously capture the geometric structural changes and dynamic displacement motion of the device's appearance. The calculation process can be represented as follows: in, As visual spatiotemporal features, The input matrix is ​​the result of converting the visible light video sequence into a tensor. This refers to the learnable 3D weight tensor in a 3D convolutional network layer. It is a three-dimensional space-time joint discrete convolution operator. This represents the bias vector corresponding to the 3D convolutional network layer. The Sigmoid smoothing nonlinear activation function is used. The reason why 3D convolutional networks are suitable for visible light video sequences is that the video stream captured by visible light cameras naturally possesses a dual spatial and temporal structure. The 3D convolutional kernel can simultaneously extract geometric texture information in the spatial dimension and motion change information in the temporal dimension during a single forward propagation, such as whether the device casing deforms or expands, whether smoke escapes from gaps, and whether external attachments have shifted—all spatiotemporal joint features.

[0019] Next, proceed to step 1.3. Based on the Long Short-Term Memory (LSTM) network, the gas concentration sequence is sequentially gated and iteratively encoded according to time steps, and the hidden layer output of the final time step is extracted to obtain the gas time-series features. Specifically, the gas concentration sequence is input into the LSM network module, and the gradient evolution law of the gas concentration is extracted sequentially according to time steps. For the final time step of the sequence, the gas time-series features that can characterize the historical trend of gas accumulation and release are extracted as the final hidden layer representation using the results of the hidden state iterations of the forget gate, input gate, and output gate. The calculation process can be expressed as follows: in, For gas time series characteristics, The gas concentration sequence from the initial time step 1 to the end time step Full time slice input, This is the final hidden layer state vector calculated by the Long Short-Term Memory network at the last time step. For the Long Short-Term Memory network in the penultimate time step The hidden state vector, This is a standard gating function mapping for a pre-packaged Long Short-Term Memory (LSTM) network. LSM networks are suitable for gas concentration sequences because the concentration data output by gas sensors is essentially a continuous sampling of a one-dimensional scalar along the time axis. Its key information lies not only in the absolute concentration value at the current moment but also in the cumulative trend of concentration change over time. Through a collaborative gating mechanism involving forget gates, input gates, and output gates, LSM networks can selectively memorize the concentration evolution patterns over long time spans, effectively capturing gradual anomalies in gas concentration, from slow increases to sharp spikes.

[0020] Finally, step 1.4 is executed. Infrared spatial features, visual spatiotemporal features, and gas temporal features are linearly mapped to a unified dimension using independent fully connected projection layers, and then concatenated along the modal dimension to obtain the multimodal initial feature tensor. Specifically, since the infrared spatial features, visual spatiotemporal features, and gas temporal features extracted in the above three sub-steps are different in vector dimensions, they cannot be directly used for subsequent graph structure construction and inter-node metric calculation. Therefore, the three features need to be input into their respective fully connected projection layers, and linear transformations are performed using independent parameter matrices to force the feature vectors with different original dimensions to be compressed or expanded to the same latent space dimension. Then, the three features after dimension alignment are concatenated along the modal dimension, and finally combined and output to produce a multimodal initial feature tensor that represents all multiphysics information within the time window. The calculation process can be represented as follows: in, For the initial feature tensor of the multimodal, , , These are the linear fully connected weight matrices used for implementing same-dimensional mapping in the infrared projection layer, visual projection layer, and gas projection layer, respectively. The feature concatenation and union operator reassembles multiple aligned features into a high-dimensional joint tensor along the modal dimension.

[0021] In one specific embodiment of this application, the two-dimensional convolutional network adopts the ResNet-18 architecture with a kernel size of 3×3. After layer-by-layer extraction through four residual blocks, it outputs an infrared spatial feature vector. The three-dimensional convolutional network adopts the C3D architecture with a kernel size of 3×3×3. After spatiotemporal joint extraction through five sets of convolutional layers, it outputs a visual spatiotemporal feature vector. The Long Short-Term Memory network has a hidden layer dimension of 256. After 60 time steps of iterative encoding, it extracts the gas temporal feature vector from the hidden layer at the final time step.

[0022] Step two involves performing symmetric graph representation and semantic divergence evaluation on the multimodal initial feature tensor to obtain the semantic divergence matrix. It should be understood that although the multimodal initial feature tensor has unified the features of the three heterogeneous modes—infrared, visible light, and gas—into a single data structure, no explicit correlation has yet been established between the feature components of each mode. The system cannot determine whether the physical states expressed by different modes are consistent within the current time window. In the scenario of safety monitoring of high-voltage transformers in new energy power plants, when the equipment is in normal operation, the feature expressions of the three modes should exhibit a high degree of semantic consistency: the infrared mode reflects a normal temperature distribution, the visible light mode reflects a normal appearance, and the gas mode reflects a normal concentration level. However, when a latent fault occurs, due to differences in the response speed and sensitivity of different physical quantities, some modes may capture the abnormal signal first, while other modes still output normal features, thus creating semantic contradictions in the physical state determination between modes. Therefore, before performing causal topology reconstruction and asymmetric feature aggregation, it is necessary to conduct a systematic quantitative evaluation of the semantic consistency between modal features in order to accurately identify which modal pairs have semantic conflicts and the severity of the conflicts.

[0023] In one embodiment of this application, such as Figure 4 As shown, step two includes: step 2.1, performing symmetric graph node decomposition and topological initialization on the multimodal initial feature tensor to obtain a symmetric feature graph; step 2.2, performing intermodal geometric and orientation metric calculations on the symmetric feature graph to obtain a joint metric tensor; and step 2.3, performing semantic divergence mapping on the joint metric tensor to obtain a semantic divergence matrix.

[0024] First, execute step 2.1. Perform symmetric graph node decomposition and topological initialization on the multimodal initial feature tensor to obtain a symmetric feature graph. Specifically, decouple and restore the multimodal initial feature tensor output in step one along the splicing dimension, extracting independent node feature vectors representing different modal attributes, namely infrared node feature vectors, visual node feature vectors, and gas node feature vectors, forming the vertex set of the graph. Then, project the extracted modal feature nodes onto a unified graph space and initialize a fully connected symmetric adjacency matrix with a dimension corresponding to the number of nodes, assuming that there are bidirectional, equal information exchange edges between all modes initially, and eliminating self-loop connections. Combine the vertex set with the fully connected symmetric adjacency matrix to construct the initial symmetric feature graph. The mathematical expression of this symmetric feature graph is: in, It is a symmetrical feature map. The graph vertex set decoupled from the multimodal initial feature tensor contains the feature vectors of all independent modalities. For an initialized undirected fully connected symmetric adjacency matrix, and The indexes are used to represent nodes of different modalities. In the 3-modal scenario of this invention, the symmetric adjacency matrix is ​​a 3×3 square matrix, with all diagonal elements being 0 to exclude self-loops, and all off-diagonal elements being 1 to indicate that there is a bidirectional, equally weighted initial connection between any two different modalities. The reason for using a symmetric graph structure as the initial topology is that before semantic conflict detection, the system should not pre-determine any priority or causal direction between modalities, but should start with a completely fair bidirectional connection. Subsequent steps will determine whether asymmetric reconstruction of the topology is necessary based on the actual semantic divergence evaluation results.

[0025] Next, proceed to step 2.2. Perform intermodal geometry and orientation metric calculations on the symmetric feature map to obtain the joint metric tensor. Specifically, extract the feature vectors of each pair of vertices with interactive edges from the input symmetric feature map. With nodes The calculation is performed from two complementary metric dimensions. First, the Euclidean distance between the feature vector pair in the latent space is calculated to quantify the geometric spatial difference in feature magnitude. A larger Euclidean distance indicates a greater deviation in the absolute magnitude of the feature values ​​between the two modalities. Second, the cosine distance (1 minus cosine similarity) is simultaneously calculated to quantify the directional deviation in the semantic evolution trajectory of the two modalities. A larger cosine distance indicates a more inconsistent orientation of the feature vectors in the high-dimensional space, meaning a more divergent semantic evolution direction. Subsequently, a priori weighting coefficient based on scene physical characteristics is introduced to linearly weight and fuse the Euclidean and cosine distances, generating a comprehensive deviation index for each node pair. This comprehensive index is then mapped and combined into a joint metric tensor for output. The calculation process can be represented as follows: in, For nodes in the joint metric tensor With nodes The joint metric of distance and direction between them and The first one extracted from the vertex set respectively The modal eigenvector and the first modality eigenvector Modal feature vectors, The interval between two modal eigenvectors Norm, also known as Euclidean distance, Let cosine similarity be the feature vectors of two modalities in the latent space. This is a hyperparameter used to adjust the weight of the amplitude Euclidean distance in the overall metric. These are hyperparameters used to adjust the weighting of the direction cosine distance in the overall metric, such as weighting coefficients. Set to 0.6, A value of 0.4 is used, indicating a slight emphasis on capturing amplitude differences in this scenario. This is because the temperature characteristic values ​​of the infrared modes exhibit a significant numerical jump during transformer overheating faults, and amplitude-level deviation signals typically appear before direction-level deviation signals. Both Euclidean and cosine distance metrics are employed because a single distance metric cannot fully characterize the semantic differences between modes: Euclidean distance focuses on capturing absolute deviations in feature values ​​but is insensitive to the overall scaling of feature vectors; cosine distance focuses on capturing angular deviations in feature directions but is insensitive to amplitude differences in feature vectors. The linear weighted fusion of these two metrics provides a complementary and comprehensive quantification of semantic deviations between modes from two orthogonal dimensions: amplitude and direction.

[0026] Finally, step 2.3 is executed. Semantic divergence mapping is performed on the joint metric tensor to obtain the semantic divergence matrix. Specifically, the scores of all elements in the joint metric tensor are received, and a nonlinear activation function with a smoothing parameter is used to perform nonlinear normalization calculation on the original scores of each element in the joint tensor. This forces the numerical distribution to be compressed into the probability interval (0,1), making the distribution exhibit a polarized characteristic that highlights extreme conflicts. That is, mode pairs with joint metric values ​​far below the mean threshold are mapped to low divergence scores close to 0, indicating high semantic consistency; mode pairs with joint metric values ​​far above the mean threshold are mapped to high divergence scores close to 1, indicating severe semantic divergence. The normalized values ​​representing the probability of semantic divergence between modes in physical states are filled into the corresponding coordinates of a two-dimensional square matrix, finally generating the semantic divergence matrix. Its calculation process can be represented as follows: in, For nodes in the semantic scatter matrix and nodes Semantic divergence score between them For nodes in the joint metric tensor and nodes The joint metric of distance and direction between them The mean threshold benchmark is used to control the central critical point of divergence surge, i.e., the turning point where physical semantics begin to produce significant contradictions. For example, the mean threshold benchmark... Set to 1.0, this value is determined based on the statistical mean of the joint metric values ​​among the modal eigenvectors under normal operating conditions. This is a steepness adjustment factor; the larger the value, the more the mapping function will exceed the baseline. The more dramatic the jump, the more it is used to reinforce the boundaries between conflicting and non-conflicting states, for example, as a kurtosis adjustment factor. The value is set to 5.0 to ensure that the divergence score has sufficient discriminative power near the threshold. The formula uses a variant of the Sigmoid function, whose core function is to transform continuous joint metrics into divergence scores with probabilistic semantics, enabling the dynamic gating mechanism in step three to perform threshold comparison and binarization discrimination based on a unified probabilistic scale.

[0027] Step 3: Based on a preset physical common sense tolerance threshold and a physical prior causal matrix, dynamic divergence gating and physical topology reconstruction are performed on the semantic divergence matrix to obtain a causal topology graph. It should be understood that in the safety monitoring scenario of high-voltage transformers in new energy power plants, the three heterogeneous modes follow a strict physical evolution chain: internal thermal accumulation is the physical root cause; continuous overheating induces the degradation of insulating materials and the release of volatile organic gases, ultimately leading to macroscopic morphological changes such as deformation or smoke from the equipment casing. When physical state contradictions occur between modes, if the symmetric graph structure initialized in Step 2 is still used for subsequent feature aggregation, the information propagation between all modes remains bidirectional and equal. The high-confidence pseudo-normal features of the downstream manifestation mode (such as visible light) will flow back into the upstream root mode (such as infrared) through the symmetric edge, thereby diluting or even covering the key abnormal signals first captured by the upstream mode. Therefore, the core task of this step is to transform the continuous numerical information in the semantic scatter matrix into discrete topological constraints. That is, between the modal pairs that detect semantic conflicts, the original bidirectional symmetric graph connections are reconstructed into unidirectional causal connections based on objective physical causal prior knowledge, thereby providing a strict topological skeleton for the subsequent asymmetric feature aggregation in step four at the mathematical structure level.

[0028] In one embodiment of this application, step three includes: step 3.1, performing element-by-element gating comparison and binarization discrimination on the scores of each element in the semantic scatter matrix based on a preset physical common sense tolerance threshold to obtain a conflict mask matrix; step 3.2, performing asymmetric connection extraction based on physical priors on the conflict mask matrix to obtain a causal directed mask matrix; and step 3.3, performing topological edge pruning and causal graph structure reconstruction on the causal directed mask matrix to obtain a causal topological graph.

[0029] First, execute step 3.1. Based on a preset physical common sense tolerance threshold, perform element-by-element gating comparison and binarization discrimination on the scores of each element in the semantic divergence matrix to obtain a conflict mask matrix. Specifically, extract each element in the semantic divergence matrix output in step two, i.e., the semantic divergence probability score between each modality node pair, and compare the divergence probability score with the system's preset physical common sense tolerance threshold element-by-element. Use an indicator function for binarization gating: when the divergence score is strictly greater than the threshold, it is determined that a semantic contradiction violating physical laws has occurred between the corresponding modalities, and the output bit is assigned a value of 1; otherwise, if it does not exceed the threshold, it is considered a reasonable deviation within the normal statistical fluctuation range, and the output bit is assigned a value of 0. Recombine all comparison results to generate a binarized conflict mask matrix for accurately identifying the conflict locations in the entire image. Its calculation process can be expressed as: in, Position in the collision mask matrix elements, For nodes in the semantic scatter matrix and nodes Semantic divergence score between them This is an indicator function that returns 1 when the internal conditional expression is true and 0 when it is false. A pre-defined physical common-sense tolerance threshold is used to define the critical point between statistical error and physical inconsistencies. The physical common-sense tolerance threshold is set based on the following: Under normal operating conditions, due to factors such as sensor noise and minor deviations in sampling time, the semantic divergence score between different modes cannot be consistently zero, but will fluctuate within a relatively low range. The function of this threshold is to distinguish this normal statistical fluctuation from actual physical inconsistencies. Only when the divergence score exceeds this threshold is the subsequent causal topology reconstruction process triggered. For example, the physical common-sense tolerance threshold... Set it to 0.75.

[0030] Next, proceed to step 3.2. Perform asymmetric connection extraction based on physical priors on the collision mask matrix to obtain the causal directed mask matrix.

[0031] In the first embodiment, the asymmetric join extraction of the collision mask matrix based on physical priors is implemented using a direct lookup table overwrite method. Specifically, the collision mask matrix generated in the previous sub-step is received, and the collision coordinate pairs marked as 1 are detected. For conflicting coordinate pairs, the system retrieves a predefined physical prior causal matrix from the underlying hard-coded database to extract the true one-way physical evolution law between the corresponding nodes. Matrix fusion is performed using an algebraic routing mechanism: if two nodes are not in conflict (i.e., the conflict mask is 0), the bidirectional interconnection coefficient is retained (i.e., assigned a value of 1); if a conflict occurs (i.e., the conflict mask is 1), the one-way connection coefficient value defined in the physical prior causal matrix is ​​forcibly overwritten. The calculation process can be represented as follows: in, Position in the causal directed mask matrix elements, Position in the physical prior causal matrix elements, Position in the collision mask matrix The element, whose value is a binary constant, is used when physical laws support information from nodes. propagation to nodes The value is 1 if a reverse causal fallacy exists, and 0 if such a fallacy exists. The implementation logic of this first embodiment is as follows: when the conflict mask matrix marks a pair of modal nodes as semantically conflicted, the system retrieves only a static physical prior causal matrix from the underlying hard-coded database, directly looks up the unidirectional connection coefficients of the coordinate pair, and then overwrites them into the causal directed mask matrix in one step. This approach treats all conflicting modal pairs as isolated binary relationships for adjudication, completely ignoring the mediating causal relationships present in multimodal causal chains. That is, the causal source node and the causal end node are often not directly connected, but rather the physical quantities are transmitted step-by-step through one or more mediating nodes.

[0032] Taking the safety monitoring scenario of high-voltage transformers in new energy power plants as an example, this application illustrates the severity of this defect. In this scenario, the three heterogeneous modes follow the following physical evolution chain: the internal thermal accumulation sensed by the infrared thermal imager is the physical root cause. This thermal accumulation first induces the degradation of insulating materials, releasing volatile organic gases (i.e., the gas concentration signal captured by the gas sensor). Continuous overheating and gas corrosion eventually lead to macroscopic morphological changes such as deformation and smoke on the equipment casing (i.e., the visual signal captured by the visible light camera). Therefore, the gas mode plays an indispensable mediating role in the causal chain from infrared to visible light. When a semantic conflict occurs between the infrared node and the visible light node, for example, when the infrared mode outputs overheating characteristics while the visible light mode outputs normal characteristics, the first embodiment will indiscriminately allow unidirectional causal injection from infrared to visible light based on static prior judgment. However, the real-time state of the gas mode (intermediate node) at this time carries crucial evidence for the validity of this decision: if the gas mode also shows an abnormal concentration spike that is highly consistent with the infrared reading, it means that the heat-to-gas conduction link has been confirmed in the current measured data, and the claim that infrared is the physical root cause has been substantially verified by the intermediate node. The causal constraint should be strengthened and reinforced. Conversely, if the concentration reading of the gas mode is completely normal and deviates significantly from the abnormal infrared reading, it means that the heat-to-gas intermediate link is not actually established in the current physical scenario. The overheating signal of the infrared is very likely to originate from external interference factors such as sensor aging and drift or sudden changes in ambient temperature, which is a false alarm. If the unidirectional causal constraint from infrared to visible light is still mechanically established according to static priors, it will cause a systematic misjudgment of the entire integrated system. The first embodiment employs a completely static, hard-coded lookup strategy that is decoupled from the current data situation. Regardless of how the actual state of the intermediary node changes, it will output the same causal direction and the same confidence strength. Under complex and ever-changing industrial field conditions, it faces the structural risk of blind adjudication due to ignoring the evidence transmitted by intermediaries.

[0033] To address the aforementioned deficiencies, a second embodiment is proposed. In the second embodiment of this application, step 3.2 includes: quantizing the mediation path comparison evidence modulation factor of the semantic divergence matrix to obtain the transitive evidence modulation matrix; performing dynamic causal confidence fusion on the transitive evidence modulation matrix to obtain the dynamic causal confidence matrix; and based on the dynamic causal confidence matrix, performing conflict condition determination and causal mask generation on the conflict mask matrix to obtain the causal directed mask matrix.

[0034] First, the semantic divergence matrix is ​​quantized using the mediation path contrast evidence modulation factor to obtain the transmission evidence modulation matrix. It should be understood that in the three-modal scenario of equipment safety monitoring, the causal decision between infrared and visible light should not bypass the gas as an intermediary transmission node and be made independently. Therefore, the primary step in improving the mechanism is to extract real-time contrast evidence from the mediating segment of the causal link.

[0035] Specifically, firstly, from the system's pre-defined physical causal directed link database, for each ordered mode pair... Extract from node To the node The set of all intermediate nodes along a causal directed path constitutes an ordered intermediate path set. Taking infrared to visible light as an example, the extracted intermediate set is {gas}; while for directly causally adjacent ordered pairs such as infrared to gas, the intermediate set is an empty set.

[0036] Subsequently, for ordered pairs of non-empty path sets, for each intermediary node k, a two-sided contrastive modulation calculation is performed: the source node-intermediary node divergence and the intermediary node-target node divergence are extracted from the existing semantic divergence matrix, the difference between the two is calculated and a nonlinear probability domain mapping is performed, and then a chain-like cumulative multiplication is performed along all intermediary nodes on the path; when the path set is empty (i.e., directly adjacent causal pairs), it is assigned a value of 1 according to the empty product convention, indicating that no external modulation is applied.

[0037] The reason for using the difference between target-side divergence and source-side divergence as the modulation signal is that, in the transformer safety monitoring scenario, if the semantic state of the intermediate gas node is closer to the causal source infrared (low source-side divergence) and farther from the causal end visible light (high target-side divergence), it means that the thermal signal has indeed been transmitted along the physical chain to the gas link and triggered a measurable chemical response, providing positive evidence from the intermediate link for the determination that infrared is the true root cause. Conversely, if the intermediate node is closer to visible light and farther from infrared, it constitutes a negative weakening of the claim that infrared is the root cause. Combining the calculation results of all ordered pairs generates the transmission evidence modulation matrix, represented as: In the formula, To transmit the ordered pairs in the modulation matrix of evidence The modulation factor scalar value, with a range of (0,1]; To extract from physical prior causal links, by nodes To the node An ordered set of all intermediary nodes on a directed path; For the specific intermediary node index on the intermediary path; The Sigmoid nonlinear activation function compresses the real-domain input to... interval; To compare the sensitivity hyperparameter, we control the steepness of the effect of the difference in two-sided divergence on the modulation factor. A larger value indicates a more significant magnifying effect of intermediate nodes on the decision result. For example, comparing the sensitivity hyperparameter... Set to 3.0; and These are the source node-intermediate node divergence scores and intermediate node-target node divergence scores extracted from the semantic divergence matrix, respectively, and the divergence scores between the source node infrared and the intermediate node gas extracted from the semantic divergence matrix. This is a chain-product operator used to perform chain-based accumulation of modulation factors for all intermediate nodes along the path. Through the above calculation, the mediation evidence that was originally ignored by static priors is quantified into a continuous dynamic modulation factor, providing real-time data-driven physical link verification basis for subsequent causal confidence fusion, effectively avoiding the risk of blindly establishing causal constraints when evidence of intermediate nodes is missing.

[0038] Then, dynamic causal confidence fusion is performed on the transmitted evidence modulation matrix to obtain a dynamic causal confidence matrix. It should be understood that after obtaining the transmitted evidence modulation matrix, it needs to be organically integrated with the static physical priors at the bottom layer of the system, so that the final causal determination not only follows the basic physical direction law, but also dynamically adjusts the confidence strength according to the actual situation of the field measurement data.

[0039] Specifically, a static physical prior causal matrix (a hard-coded constant matrix with elements of 0 or 1) is retrieved from the system database. The transitive evidence modulation matrix and the physical prior causal matrix are then subjected to an element-wise Hadamard product. The Hadamard product, rather than addition or concatenation, is chosen because this element-wise multiplication structure naturally guarantees a hard boundary: for propagation directions that are not permitted by the physical prior (prior value of 0), no matter how sufficient the evidence on the intermediate link is, the dynamic confidence level will be completely reduced to zero, and a reverse path violating basic physical laws will never be mistakenly opened due to data noise; while for propagation directions that are permitted by the physical prior (prior value of 1), the final effective confidence strength is no longer a constant of 1, but is scaled by the real-time evidence modulation factor of the intermediate link, thus injecting the originally rigid static rules with the activity of the measured data under the current physical scenario. The fusion result output is a continuous dynamic causal confidence matrix, expressed as: In the formula, For ordered pairs in the dynamic causal confidence matrix The dynamic causal confidence scalar value, with a range of [0,1]; For ordered pairs in the physical prior causal matrix A static binary constant, which takes the value 1 when causal direction is allowed and 0 when causal direction is prohibited; This is the modulation factor corresponding to the transmission evidence modulation matrix output from the previous sub-step.

[0040] Through this fusion operation, each candidate causal edge obtains a continuous confidence score that comprehensively considers the prior physical direction and the real-time evidence from the mediator, providing a basis for the final decision that combines physical legitimacy and data authenticity.

[0041] Finally, based on the dynamic causal confidence matrix, conflict condition determination and causal mask generation are performed on the conflict mask matrix to obtain a causal directed mask matrix. Specifically, after receiving the conflict mask matrix and the dynamic causal confidence matrix, for the coordinate pairs marked as 1 in the conflict mask (i.e., modal pairs that have been confirmed to have semantic contradictions), a preset dynamic confidence threshold is used to perform binarization determination on the corresponding elements in the dynamic causal confidence matrix: only when the dynamic confidence is greater than or equal to the threshold is the causal connection in that direction retained and assigned a value of 1; otherwise, it is determined that the evidence of the intermediate link in that causal direction is insufficient to support the establishment of constraints, and it is pruned and assigned a value of 0.

[0042] In transformer monitoring scenarios, this means that when infrared and visible light conflict, but gas-mediated evidence is insufficient to support the infrared root cause claim, the system will not arbitrarily establish a unidirectional injection channel from infrared to visible light based solely on static priors. Instead, it will choose to postpone the decision or refuse to establish the edge, fundamentally preventing blind decision-making. For coordinate pairs marked with a conflict mask of 0 (i.e., conflict-free normal mode pairs), the original bidirectional interoperability coefficient is maintained and assigned a value of 1 to ensure that the normal multimodal information flow is not disturbed. The results of the above two types of conditional branches are uniformly synthesized to generate the final causal directed mask matrix, represented as: In the formula, For ordered pairs in a causal directed mask matrix Binary connectivity coefficients; The binary label comes from the collision mask matrix, where 1 indicates that the mode pair has been determined to be in conflict and 0 indicates that it is not in conflict. This is an indicator function that returns 1 when the condition is true and 0 when the condition is false. These are continuous confidence values ​​derived from the corresponding coordinates in the dynamic causal confidence matrix; A dynamic confidence threshold is used to determine whether the evidence from the mediation link sufficiently supports the establishment of a causal direction when a conflict occurs. For example, a dynamic confidence threshold... Set it to 0.5.

[0043] Through the above mechanism, the rigid adjudication method of static table lookup and one-step overwriting in sub-step 3.2 of the second embodiment is replaced by a three-order flexible arbitration process that first collects intermediate measured evidence, then dynamically modulates causal confidence, and finally makes a fine-grained adjudication based on a threshold. In complex industrial scenarios such as safety monitoring of transformers in new energy power plants, when there is a physical state contradiction between infrared and visible light modes, the system no longer mechanically relies on hard-coded priors to make blind directional decisions. Instead, it automatically traces back along the causal link to intermediate nodes such as gases, extracts real-time physical transmission evidence, and dynamically strengthens or weakens the effective constraint of the causal edge accordingly. This improvement enables the causal topology reconstruction process to have adaptive perception capabilities for the current operating conditions while maintaining the legality of the physical direction. It significantly reduces the probability of causal misjudgment in atypical scenarios such as sensor drift and environmental interference, thereby improving the decision robustness and safety reliability of the multimodal heterogeneous data integration system under extreme and boundary conditions.

[0044] Finally, topological edge pruning and causal graph structure reconstruction are performed. The causal directed mask matrix is ​​pruned and reconstructed to obtain a causal topological graph. Specifically, the causal directed mask matrix output from the previous sub-step is extracted as the mathematical skeleton for graph structure reconstruction. The elements on the diagonal of this matrix are zeroed out to eliminate self-loop interference, and it is directly instantiated as an asymmetric adjacency matrix of the reconstructed graph. At this point, the reverse causal connection edges that originally caused statistical average errors have been forcibly pruned in the matrix (coefficients set to 0), leaving only the forward causal connection edges (coefficients set to 1). The pre-determined set of modal nodes (vertices) and the newly generated asymmetric adjacency matrix (edges) are logically bound in memory, thereby instantiating and outputting the final causal topological graph at the mathematical and data structure level. The calculation process can be represented as follows: in, For a causal directed mask matrix, This section describes the extraction and construction of diagonal matrices, where diagonal elements are subtracted to eliminate self-loops in graph nodes. For a graph-theoretic set representation of a causal topological graph, It is the set of modal node vertices in a multimodal heterogeneous system.

[0045] Step four involves performing asymmetric feature aggregation based on the reconstructed topology graph on the initial multimodal feature tensor to obtain a causal fusion feature tensor. It should be understood that in the safety monitoring scenario of high-voltage transformers in new energy power plants, the feature representation of a single mode often cannot independently support accurate equipment status determination. For example, although the infrared mode can initially capture internal overheating signals, temperature distribution characteristics alone cannot determine whether overheating has already caused subsequent chemical degradation and external damage. Information from the gas mode and visible light mode needs to be integrated into the infrared features to form a complete fault evolution profile. However, this cross-modal information fusion cannot use the traditional symmetric aggregation method because, in the case of semantic conflicts between modes, symmetric aggregation can cause high-confidence pseudo-normal features from the downstream apparent mode to flow back into the upstream root mode through bidirectional edges, thereby diluting or even covering the key abnormal signals initially captured by the upstream mode. Therefore, under the strict constraints of the causal topology graph generated in step three, this step performs asymmetric feature aggregation restricted by topological direction, so that information flows only from the upstream root mode to the downstream representation mode along the forward direction of the physical causal chain, while the information propagation in the reverse direction is completely blocked, thereby achieving deep feature fusion with physical causal consistency while preserving the original features of each mode.

[0046] In one embodiment of this application, step four includes: Step 4.1, based on the directed adjacency relationship of the causal topology graph, causal neighbor constraints are applied to each node in the multimodal initial feature tensor, and masked normalized attention calculation is performed through linear projection and nonlinear activation to obtain a directed attention coefficient matrix; Step 4.2, based on the directed attention coefficient matrix, the features of each upstream neighbor node in the multimodal initial feature tensor are subjected to range projection and weighted summation along the causal directed edges to obtain a neighbor converged feature tensor; Step 4.3, the features of each node in the multimodal initial feature tensor are deeply concatenated with the neighbor converged feature tensor, and nonlinear fusion is performed through an aggregation mapping network to obtain a causal fusion feature tensor.

[0047] First, proceed to step 4.1. Based on the directed adjacency relationships of the causal topology graph, causal neighbor constraints are applied to each node in the multimodal initial feature tensor. Then, masked normalization attention calculation is performed using linear projection and nonlinear activation to obtain the directional attention coefficient matrix. Specifically, the multimodal initial feature tensor output from step one is used as the feature representation basis of the graph nodes, while the asymmetric adjacency relationships in the causal topology graph output from step three are read as addressing constraints. For any central node i in the graph, only its causal predecessor node set in the causal topology graph is detected and extracted, i.e., upstream neighbor nodes pointing forward to the central node. For node pairs without directed edge connections in the causal topology graph, hard computation masking is performed to ensure that node pairs in the reverse causal direction do not participate in any attention weight calculation process. After determining the range of legitimate causal neighbors, a learnable linear transformation matrix is ​​used to perform dimensionality reduction projection on the original features of the central node and each causal neighbor node. The projected feature vectors are then concatenated, and unnormalized correlation coefficients are calculated using a leaky linear rectified activation function. Finally, a normalized exponential function constrained by a physical topology mask is used to calculate the relative weight distribution only within the effective range of causally connected neighbors, generating a directed attention coefficient matrix for precise control of information flow allocation. The calculation process can be represented as follows: in, From the node in the targeted attention coefficient matrix Pass to node Targeted attention coefficient, This is a linear rectified activation function with leakage, used to introduce a small gradient on the negative half-axis to avoid neuron death. Its slope on the negative half-axis can be set as needed, for example, to 0.2. This is the transpose of the weight vector of a learnable single-layer feedforward neural network in the graph attention mechanism. For tensor concatenation operators, This is a shared linear transformation matrix used for dimensionality reduction projection and message extraction of the original node features. , , These are nodes in the multimodal initial feature tensor. ,node and nodes The hidden layer feature vectors For nodes constrained by a causal topological graph The causal predecessor neighbor set contains only nodes that are pointed to by a one-way causal edge. The core mechanism of this formula is that the summation range of the normalized exponential function is strictly limited to the set of causal predecessor neighbors, rather than all neighbor nodes in a traditional graph attention network. This means that only upstream nodes connected by directed edges along the forward direction of the physical causal chain are eligible to participate in the competition and allocation of attention weights, thus realizing the physical constraint of the information flow direction at the attention calculation level.

[0048] Next, proceed to step 4.2. Based on the directed attention coefficient matrix, the features of each upstream neighbor node in the multimodal initial feature tensor are projected along the causal directed edges and weighted summed to obtain the neighbor converged feature tensor. Specifically, the directed attention coefficient matrix generated in the previous sub-step is received, and the cached multimodal initial feature tensor is retrieved as the feature source. Along the directed causal edges in the causal topology graph, the feature vectors of the physical evolution upstream mode, i.e., the neighbor nodes, are mapped and extracted using independent value projection matrices. The role of these value projection matrices is to transform and refine the data distribution of the neighbor features before weighted summing, making the features of different neighbor nodes comparable in the same projection space. Subsequently, a weighted summation operation is performed on the projected features of all forward neighbors according to the directed attention coefficients, i.e., a linear combination is performed using the attention weight as the contribution ratio of each neighbor node feature. This process only accepts feature information flowing in from the causal upstream direction. Node pairs disconnected in the reverse direction have had their attention coefficients set to 0 since they were hard-shielded in the previous sub-step, so no information will flow back from the downstream representation mode to the upstream root mode. The upstream features collected from all nodes are aggregated and packaged to generate a neighbor aggregation feature tensor. The calculation process can be represented as follows: in, Nodes in the feature tensor of neighbor aggregation The collected neighbor convergence feature vectors, From the node in the targeted attention coefficient matrix Pass to node Targeted attention coefficient, This is a range transformation matrix used to transform the data distribution of neighbor features before weighting. Neighbor nodes in the multimodal initial feature tensor The hidden feature vector. It should be noted that for the root node located at the top of the causal topology graph (such as the infrared node), if its causal predecessor neighbor set is empty, then the neighbor convergence feature vector of the node is a zero vector, indicating that the node does not receive information injection from any other modality, and its final fused feature is completely determined by its own original features. This is reasonable in a physical sense, because the state determination of the physical root modality should not be affected by the downstream manifestation modality.

[0049] Finally, step 4.3 is executed. The individual features of each node in the multimodal initial feature tensor are deeply concatenated with the neighbor converged feature tensor, and then nonlinearly fused using an aggregation mapping network to obtain the causal fusion feature tensor. Specifically, for each modal node in the graph, its original feature vector in the multimodal initial feature tensor is extracted. And the upstream convergence information vector corresponding to the neighbor convergence feature tensor. The feature concatenation operator is used to deeply concatenate the self-representation and the received upstream causal representation along the feature dimension, forming a hyperdimensional feature vector with doubled dimensions. This hyperdimensional feature vector is then input into an aggregation mapping network, which consists of feedforward layers with nonlinear activation functions. Through nonlinear mixing, the concatenated hyperdimensional feature is reprojected onto the standard latent space dimension, completing the organic fusion of the self-feature and the upstream causal features. The updated latent state features of all nodes in the entire graph are then re-normalized and merged, outputting a final causal fusion feature tensor with rigorous physical and logical consistency. The computation process can be represented as follows: in, For the updated node The final hidden layer representations, whose set constitute the causal fusion feature tensor, This is a nonlinear activation operator used to endow the aggregation layer with deep nonlinear fitting capabilities. This is the weight matrix of the aggregation mapping layer, used to reproject and map the concatenated superdimensional features (i.e., the original node features and the causal neighbor features) to the standard latent space dimension. For nodes The result is the concatenation of the self-feature vector and the neighbor convergent feature vector along the feature dimension. The reason for using a concatenation followed by nonlinear mapping instead of simple addition or averaging is that the concatenation operation can completely preserve the independent information of the self-feature and the upstream causal features along the feature dimension, avoiding the information confusion and feature collapse that may be caused by the addition operation. The subsequent nonlinear mapping layer gives the network the ability to autonomously learn the optimal fusion ratio and interaction mode between the two types of features.

[0050] Step five involves global feature pooling and final state arbitration of the causal fusion feature tensor to obtain equipment status labels for equipment safety monitoring. It should be understood that the features of each modal node in the fusion feature tensor have incorporated the correlation information from upstream modes in the physical causal chain, and the entire fusion process strictly follows the unidirectional transmission law of the objective physical causal chain. However, at this point, the causal fusion feature tensor still exists as a set of feature vectors of independent nodes in a graph structure. Each node's feature vector represents the local state representation of each mode after causal constraint fusion, and has not yet been integrated into a unified decision-making basis that can represent the global operating state of the entire monitored equipment. In the safety monitoring scenario of high-voltage transformers in new energy power plants, the final equipment status determination needs to comprehensively consider the internal thermal state reflected by the infrared mode, the chemical degradation state reflected by the gas mode, and the external morphological state reflected by the visible light mode, rather than making a one-sided judgment based solely on the local features of a single modal node. Therefore, the core task of this step is to compress the scattered multi-node features in the graph structure into a global comprehensive decision vector, and then map this vector into discrete device state category labels through a state arbitration network, thereby completing a complete inference loop from multimodal heterogeneous data input to device safety state output.

[0051] In one embodiment of this application, step five includes: step 5.1, performing graph-scale global feature mean pooling on the causal fusion feature tensor to obtain a global decision vector; step 5.2, inputting the global decision vector into a state arbitration network composed of multilayer sensing mechanisms to obtain a state probability distribution sequence; step 5.3, performing a maximum value index extraction operation on the state probability distribution sequence to locate the highest probability category, and decoding it into a device status label that can be called by the industrial control system through a dictionary mapping table.

[0052] First, execute step 5.1. Perform graph-scale global feature mean pooling on the causal fusion feature tensor to obtain the global decision vector. Specifically, receive the causal fusion feature tensor output from step four, which contains the feature vectors of all heterogeneous modal nodes updated by physical causal constraints. Perform an arithmetic mean summation (mean pooling) operation on the feature vectors of all modal nodes along the node count dimension, calculating the arithmetic mean of the corresponding dimensions in each node's feature vector, thereby smoothing out the specific differences in node structure and spatial heterogeneity. Through this dimensionality reduction and compression operation, the graph structure data originally composed of multiple independent node feature vectors is transformed into a one-dimensional high-order feature vector, which can represent the global physical evolution state of the entire system at the current moment, thus packaging and generating and outputting the global decision vector. The calculation process can be represented as follows: in, For the global decision vector, This represents the total number of multimodal vertices (nodes) in the entire graph, used for scaling during mean pooling. For the updated node The final hidden layer representation. The reason for using mean pooling instead of max pooling or summation pooling is that in the equipment safety monitoring scenario of this invention, the features of each modal node after causal constraint fusion have internalized the directional information of the physical causal chain. Mean pooling can smoothly integrate the fused features of each node in an equal weighted manner, avoiding the extreme feature dominance effect that may be caused by max pooling, and avoiding the numerical scale instability problem caused by the change in the number of nodes in summation pooling.

[0053] Next, step 5.2 is executed. The global decision vector is input into a state arbitration network composed of multilayer sensing mechanisms to obtain a state probability distribution sequence. Specifically, a one-dimensional global decision vector is input into a state arbitration network composed of multilayer sensing mechanisms pre-defined by the system. This state arbitration network consists of at least one hidden layer and an output classification layer. The global decision vector first undergoes linear transformation and nonlinear activation in the hidden layer, performing deep nonlinear mapping on the input high-dimensional features to extract higher-order decision features. Subsequently, the linear transformation in the output classification layer maps the features to a pre-defined state space dimension, i.e., a dimension space equal to the number of predefined device security state categories, generating the original logical output value. Finally, a normalized exponential function is used to force compression of the probability domain of the original logical output value, ensuring that the output values ​​of each category are non-negative and their sum is strictly 1, thereby calculating and outputting the state probability distribution sequence of the device belonging to each preset security level. The calculation process can be expressed as follows: in, It is a sequence of state probability distributions. The normalized exponential function operator is used to transform unrestricted real-valued outputs into a probability distribution that sums to 1. and These are the learnable weight matrix and bias vector of the hidden layer of the arbitration network, respectively. To modify the activation function of the linear unit, in order to provide the network with deep nonlinear mapping capabilities, and These represent the learnable weight matrix and bias vector of the classification layer output by the arbitration network. This is the global decision vector. The reason why the state arbitration network adopts a multilayer perceptron architecture instead of a single linear classifier is that the decision boundary of the device safety state is often nonlinear in the high-dimensional feature space. For example, the boundary between slight overheating and severe overheating is not a simple linear hyperplane, but a complex surface affected by the interaction of multiple physical factors. The multilayer perceptron can fit this complex decision boundary by introducing a nonlinear activation function.

[0054] Finally, maximum a posteriori probability extraction and final label conversion are performed. A maximum value index extraction operation is performed on the state probability distribution sequence to locate the highest probability category, and then decoded into an equipment status label that can be called by the industrial control system via a dictionary mapping table. Specifically, the input state probability distribution sequence is read, which is a one-dimensional array composed of the probability values ​​of each candidate state category. A maximum value index extraction operation is performed to retrieve and locate the category index corresponding to the probability distribution item with the largest value in the array, i.e., selecting the state category with the highest posterior probability as the final judgment result. The extracted maximum probability category index is input into the system's preset dictionary mapping table, and it is hardware-decoded into a string or enumeration type tag that can be directly called by the industrial control console or on-site maintenance personnel, generating the final equipment status label for output, thus ending the entire algorithm flow of this invention. Its calculation process can be represented as follows: in, For the categorical variables of the device status label, The maximum index extraction operator returns the index of the argument that maximizes the objective function or array. For example, the complete set of predefined device security status categories. It contains four enumeration values: category index 0 corresponds to "normal operation", category index 1 corresponds to "early warning", category index 2 corresponds to "moderate anomaly", and category index 3 corresponds to "critical failure". For a specific enumerated category index in the complete set of categories, For a device to belong to the state probability distribution sequence, it is the first... The scalar value of the corresponding posterior probability of the class.

[0055] It should be noted that the determination methods of the multiple learnable parameters and preset parameters involved in the above embodiments of this application are described below.

[0056] Regarding the training and determination of learnable parameters: The learnable parameters involved in the method of this application include, but are not limited to: the weight matrix and bias vector of the two-dimensional convolutional network in step one, the weight tensor and bias vector of the three-dimensional convolutional network, the internal parameters of the gate function of the long short-term memory network, the weight matrix of the fully connected projection layer, the weight matrix of the aggregation mapping layer, and the hidden layer weight matrix, bias vector, and output classification layer weight matrix and bias vector of the state arbitration network in step five. All of the above learnable parameters are jointly determined through end-to-end supervised training. Specifically, during the training phase, multimodal heterogeneous data samples including infrared image sequences, visible light video sequences, and gas concentration sequences are collected, and domain experts label each set of samples with corresponding device status tags as supervision signals. The labeled samples are input into the complete computational process of this application's method. After forward propagation through steps one to five, a sequence of state probability distributions is obtained. The difference between the predicted probability distribution and the true label is calculated using the cross-entropy loss function as the training loss value. Subsequently, the gradient of the loss value with respect to all the above-mentioned learnable parameters is calculated using the backpropagation algorithm. The values ​​of each parameter are then iteratively updated according to the gradient direction using stochastic gradient descent or its variant optimizer (such as the Adam optimizer) until the training loss converges to a preset threshold or reaches a preset maximum number of iterations. The aforementioned end-to-end supervised training, backpropagation algorithm, and gradient optimizer are all mature techniques in the field of deep learning. Those skilled in the art can select appropriate training configurations such as learning rate, batch size, and number of iterations based on the specific data scale and hardware conditions. The specific implementation details will not be elaborated here.

[0057] Regarding the determination of preset parameters: The preset parameters involved in the method of this application include, but are not limited to: the weighting coefficients for joint metric calculation in step two, the mean threshold benchmark and kurtosis adjustment factor for semantic divergence mapping; the physical common sense tolerance threshold, the physical prior causal matrix in step three, the contrast sensitivity hyperparameter and dynamic confidence threshold in the second embodiment; and the negative half-axis slope of the leakage linear rectified activation function in step four. The above preset parameters are determined according to their properties in the following ways. Among them, the physical prior causal matrix is ​​configured once by domain experts according to the physical evolution mechanism of the monitored equipment. Its element values ​​reflect the objectively existing unidirectional causal propagation direction between each mode. It belongs to the prior setting based on domain knowledge and remains fixed after system deployment. Parameters related to the statistical characteristics of normal operating conditions, such as the physical common sense tolerance threshold and the mean threshold benchmark, are determined by collecting a certain number of calibration data samples under normal operating conditions and statistically analyzing the distribution characteristics (such as mean, standard deviation, percentile, etc.) of the semantic divergence scores or joint metric values ​​between each mode pair. The optimal values ​​for hyperparameters such as weighting coefficients, kurtosis adjustment factors, contrast sensitivity hyperparameters, dynamic confidence thresholds, and negative half-axis slopes are determined through hyperparameter tuning methods such as grid search or cross-validation on the validation dataset. These hyperparameter tuning methods are all standard techniques in the field of machine learning, and those skilled in the art can select them appropriately based on specific application scenarios and performance requirements; therefore, they will not be elaborated upon here.

[0058] In summary, the heterogeneous data integration method based on multimodal AI in this application has been elucidated. By introducing semantic divergence gating and physical causal topology reconstruction mechanisms into the multimodal graph integration architecture, the integration system can adaptively convert the symmetric graph structure into a causal directed graph structure when semantic conflicts between modalities are detected. This fundamentally blocks noise interference and feature dilution paths in the reverse causal direction, ensuring the dominant position of anomalous signals from the physical root modality in the fusion process. Compared with existing multimodal integration methods based on weighted averaging or symmetric message passing, this application can significantly reduce the probability of systematic misjudgment caused by ignoring physical causal logic under atypical conditions such as sensor drift and sudden changes in ambient temperature. It effectively improves the decision accuracy and safety reliability of the multimodal heterogeneous data integration system under extreme and boundary conditions. At the same time, the introduction of causal topology enables the final state determination result to have traceable interpretability along the physical evolution chain, meeting the stringent requirements of decision transparency and auditability in industrial-grade safety monitoring scenarios.

[0059] Figure 5 This is a flowchart of the heterogeneous data integration system based on multimodal AI according to this application. Figure 5As shown, the heterogeneous data integration system 500 based on multimodal AI according to an embodiment of this application includes: a multimodal feature extraction module 510, used to extract and concatenate independent modal features from infrared image sequences, visible light video sequences, and gas concentration sequences based on two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks to obtain a multimodal initial feature tensor; a semantic divergence evaluation module 520, used to perform symmetric graph representation and semantic divergence evaluation on the multimodal initial feature tensor to obtain a semantic divergence matrix; a causal topology reconstruction module 530, used to perform dynamic divergence gating and physical topology reconstruction on the semantic divergence matrix based on a preset physical common sense tolerance threshold and a physical prior causal matrix to obtain a causal topology graph; an asymmetric feature aggregation module 540, used to perform asymmetric feature aggregation on the multimodal initial feature tensor based on the reconstructed topology based on the causal topology graph to obtain a causal fusion feature tensor; and a state arbitration output module 550, used to perform global feature pooling and final state arbitration on the causal fusion feature tensor to obtain a device status label for device safety monitoring.

[0060] It should be noted that the heterogeneous data integration system based on multimodal AI in this application embodiment is similar in principle to the aforementioned heterogeneous data integration method based on multimodal AI. Therefore, the implementation process, implementation principle, and beneficial effects of the heterogeneous data integration system based on multimodal AI can be found in the description of the implementation process, implementation principle, and beneficial effects of the aforementioned method, and will not be repeated.

[0061] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A heterogeneous data integration method based on multimodal AI, characterized in that, include: Step 1: Based on two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks, independent modal feature extraction and feature concatenation are performed on infrared image sequences, visible light video sequences, and gas concentration sequences to obtain multimodal initial feature tensors. Step 2: Perform symmetric graph representation and semantic divergence evaluation on the initial feature tensor of the multimodal mode to obtain the semantic divergence matrix; Step 3: Based on the preset physical common sense tolerance threshold and the physical prior causal matrix, the semantic divergence matrix is ​​dynamically diverged and physically reconstructed to obtain the causal topology graph. Step 4: Based on the causal topology graph, perform asymmetric feature aggregation on the multimodal initial feature tensor based on the reconstructed topology to obtain the causal fusion feature tensor; Step 5: Perform global feature pooling and final state arbitration on the causal fusion feature tensor to obtain the device status label for device safety monitoring.

2. The heterogeneous data integration method based on multimodal AI according to claim 1, characterized in that, Step one includes: Step 1.1: Based on a two-dimensional convolutional network, perform layer-by-layer local receptive field scanning and feature extraction of the spatial temperature distribution in the infrared image sequence to obtain infrared spatial features; Step 1.2: Based on a three-dimensional convolutional network, spatiotemporal geometric transformation features along the three dimensions of height, width, and time are extracted from the visible light video sequence to obtain visual spatiotemporal features; Step 1.3: Based on the Long Short-Term Memory network, the gas concentration sequence is sequentially gated and iteratively encoded according to the time step, and the hidden layer output of the final time step is extracted to obtain the gas time series features. Step 1.4: Perform linear mapping of infrared spatial features, visual spatiotemporal features and gas temporal features in a unified dimension through independent fully connected projection layers, and then stitch and merge them along the modal dimension to obtain the multimodal initial feature tensor.

3. The heterogeneous data integration method based on multimodal AI according to claim 1, characterized in that, Step two includes: Step 2.1: Perform symmetric graph node decomposition and topological initialization on the multimodal initial feature tensor to obtain a symmetric feature graph; Step 2.2: Perform intermodal geometry and orientation metric calculations on the symmetric feature map to obtain the joint metric tensor; Step 2.3: Perform semantic divergence mapping on the joint metric tensor to obtain the semantic divergence matrix.

4. The heterogeneous data integration method based on multimodal AI according to claim 3, characterized in that, Step 2.3 includes: performing a semantic divergence mapping on the joint metric tensor using the following formula: in, For nodes in the semantic scatter matrix and nodes Semantic divergence score between them For nodes in the joint metric tensor and nodes The joint metric of distance and direction between them Using the mean threshold as the benchmark, This is a steepness adjustment factor.

5. The heterogeneous data integration method based on multimodal AI according to claim 1, characterized in that, Step three includes: Step 3.1: Based on the preset physical common sense tolerance threshold, perform element-by-element gating comparison and binarization discrimination on the scores of each element in the semantic scatter matrix to obtain the conflict mask matrix; Step 3.2: Perform asymmetric connection extraction based on physical priors on the collision mask matrix to obtain the causal directed mask matrix; Step 3.3: Perform topological edge pruning and causal graph structure reconstruction on the causal directed mask matrix to obtain the causal topological graph.

6. The heterogeneous data integration method based on multimodal AI according to claim 1, characterized in that, Step four includes: Step 4.1: Based on the directed adjacency relationship of the causal topological graph, causal neighbor constraints are applied to each node in the multimodal initial feature tensor, and masked normalized attention calculation is performed through linear projection and nonlinear activation to obtain the directed attention coefficient matrix. Step 4.2: Based on the directed attention coefficient matrix, the features of each upstream neighbor node in the multimodal initial feature tensor are projected along the causal directed edge and weighted summation is performed to obtain the neighbor converged feature tensor. Step 4.3: Deeply concatenate the features of each node in the multimodal initial feature tensor with the neighbor converged feature tensor, and then perform nonlinear fusion through the aggregation mapping network to obtain the causal fusion feature tensor.

7. The heterogeneous data integration method based on multimodal AI according to claim 1, characterized in that, Step five includes: Step 5.1: Perform graph-scale global feature mean pooling on the causal fusion feature tensor to obtain the global decision vector; Step 5.2: Input the global decision vector into the state arbitration network composed of multilayer sensing mechanisms to obtain the state probability distribution sequence; Step 5.3: Perform a maximum value index extraction operation on the state probability distribution sequence to locate the highest probability category, and decode it into a device state label that can be called by the industrial control system through a dictionary mapping table.

8. The heterogeneous data integration method based on multimodal AI according to claim 5, characterized in that, Step 3.2 includes: performing asymmetric connectivity extraction based on physical priors on the collision mask matrix using the following formula: in, Position in the causal directed mask matrix elements, Position in the physical prior causal matrix elements, Position in the collision mask matrix Element.

9. The heterogeneous data integration method based on multimodal AI according to claim 5, characterized in that, Step 3.2 includes: The semantic divergence matrix is ​​quantized by the mediation path comparison evidence modulation factor to obtain the transmission evidence modulation matrix; Dynamic causal confidence fusion is performed on the transmitted evidence modulation matrix to obtain a dynamic causal confidence matrix; Based on the dynamic causal confidence matrix, conflict condition determination and causal mask generation are performed on the conflict mask matrix to obtain the causal directed mask matrix.

10. A heterogeneous data integration system based on multimodal AI, characterized in that, include: The multimodal feature extraction module is used to extract and concatenate independent modal features from infrared image sequences, visible light video sequences, and gas concentration sequences based on two-dimensional convolutional networks, three-dimensional convolutional networks, and long short-term memory networks to obtain multimodal initial feature tensors. The semantic divergence evaluation module is used to perform symmetric graph representation and semantic divergence evaluation on the initial feature tensor of the multimodal mode to obtain the semantic divergence matrix; The causal topology reconstruction module is used to perform dynamic divergence gating and physical topology reconstruction on the semantic divergence matrix based on a preset physical common sense tolerance threshold and a physical prior causal matrix to obtain a causal topology graph. The asymmetric feature aggregation module is used to perform asymmetric feature aggregation on multimodal initial feature tensors based on reconstructed topology to obtain causal fusion feature tensors; The state arbitration output module is used to perform global feature pooling and final state arbitration on the causal fusion feature tensor to obtain the device state label for device safety monitoring.

Images