A multimodal equipment integrated management system and method based on intelligent AI

Abstract

The present invention belongs to the technical field of equipment integration. The present invention discloses a multimodal equipment integration management system and method based on intelligent AI, including an autonomous perception module, which uses embedded causal analysis to perform real-time causal modeling on multimodal data; combined with a self-supervised anomaly detection mechanism, outputs a device causal graph and a data stream with confidence labels; a liquid feature factory module, based on the device causal graph and the data stream with confidence labels, uses a gated temporal convolutional network to extract time series features, reconstructs feature paths through a dynamic feature melting pool mechanism, and compresses and selects high-dimensional features in combination with a sparse regularization method to generate a feature tensor; it helps to achieve accurate perception, efficient decision-making, precise execution and intelligent optimization management of equipment.

Description

Technical Field

[0001] The present invention relates to the field of equipment integration technology, and more specifically, to an intelligent AI-based multimodal equipment integration management system and method. Background Art

[0002] Patent publication number CN115098156A discloses a network modality management system and management method. The system includes a multimodal network integrated development environment and a multimodal network distributed compilation and deployment environment. The multimodal network integrated development environment also includes a network modality deployment file packaging tool for packaging network modality source files and corresponding configuration files into network modality deployment files. The multimodal network distributed compilation and deployment environment includes a multimodal network modality package manager deployed on a controller server and a network node device network modality package manager deployed on a network node device. The present invention achieves unified management of network modalities in a multimodal network, automatically distributes deployment files to all network node devices within the network, and coordinates the compilation and deployment of source code files on each network node device and different target forwarding modules on the devices, thereby significantly improving the management efficiency of network modalities.

[0003] Existing equipment integration management systems and methods suffer from the following major issues: Cause-and-effect modeling lacks explicitness and interpretability. Traditional equipment integration management systems generally rely on black-box data correlation modeling methods, which can only identify "what is related" but cannot answer "why it is related." Causal relationships are often implicit in complex model parameters and lack a clear structural representation. This lack of explicit causal representation directly leads to the following chain of problems:

[0004] Fault tracing relies on manual experience and lacks automated analysis paths: Because the system cannot clearly express and track the causal path between events, once an anomaly or fault occurs, problem location relies solely on the experience of operations and maintenance personnel. This makes it difficult to achieve automatic fault reasoning and efficient location based on the causal chain, reducing the efficiency and accuracy of system responses.

[0005] Abnormal data is not isolated, interfering with the evolution of causal structures: Without explicit causal relationships, the system also lacks a structural identification and isolation mechanism for "abnormal behavior." When abnormal data enters the system, it often directly participates in model training or graph structure updates, leading to erroneous learning and distorted edge weights. This, in turn, leads to the accumulation of false causal relationships over long-term operations, seriously affecting system stability and judgment accuracy.

[0006] Strategy generation lacks causal chain support, resulting in fragmented control logic: Without a causal structure, the system's decision-making logic often remains solely driven by data correlation, ignoring the actual links of functional collaboration and causal dependency between devices. For example, when the root cause of a motor's speed drop is insufficient gearbox lubrication, the system may only adjust the motor parameters and fail to trigger joint strategies involving the lubrication system, resulting in fragmented strategies and limited execution effectiveness.

[0007] Poor policy generalization capabilities and inability to cope with complex operating conditions or cascading failures: Policy systems lacking causal support often only handle predefined standard scenarios. They lack effective policy coverage for complex operating conditions such as multimodal asynchronous anomalies and chain reactions between devices, requiring extensive manual intervention. Furthermore, the system's inability to conduct causal closed-loop evaluations based on execution feedback results in a lack of dynamic adaptability in policy adjustments, making them susceptible to misleading data or sensory noise.

[0008] In view of this, the present invention proposes a multimodal device integrated management system and method based on intelligent AI to solve the above problems. Summary of the Invention

[0009] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned objectives, the present invention provides the following technical solution: a multimodal device integrated management system based on intelligent AI, comprising:

[0010] The autonomous perception module uses embedded causal analysis to perform real-time causal modeling on multimodal data. Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels.

[0011] The Liquid Feature Factory module uses a gated temporal convolutional network to extract time series features based on the device causal graph and confidence-labeled data streams. It then reconstructs feature paths through a dynamic feature melting pool mechanism and combines sparse regularization methods to compress and select high-dimensional features, ultimately generating feature tensors.

[0012] The path decision module generates multi-strategy execution paths based on feature tensors and device causal graphs. Through a multi-agent course learning mechanism, it constructs a policy behavior tree based on the multi-strategy execution paths. It uses a distributed consensus mechanism to verify the consistency of the execution paths of the policy behavior tree and output a multi-path policy instruction set.

[0013] The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes.

[0014] The cognitive reflection coordination module is used to build a cognitive feedback pathway including multimodal state tracking and execution evaluation based on execution feedback data and execution status data, and to perform integrated optimization management of multimodal devices through an adaptive feedback mechanism.

[0015] Preferably, the real-time causal modeling method includes:

[0016] Heterogeneous data from various sensor terminals is collected and aligned to a unified sampling time axis using interpolation based on local clock synchronization signals. Missing values are then filled using the spatial nearest neighbor interpolation method. The filled heterogeneous data is then subjected to noise filtering to achieve unified spatiotemporal standardization of the heterogeneous data and obtain multimodal data.

[0017] A time window length is preset, and multimodal data is collected through a sliding time window mechanism to form a multimodal dataset. The multimodal dataset consists of multimodal data collected at all time points within the initial time window. A full-variable lag prediction model is constructed, using the multimodal dataset as input, and Granger causality test is performed to obtain a set of directed causal edges.

[0018] The multimodal dataset is input into the structural inference layer of the variational autoencoder, and the distribution characteristics of each multimodal data are encoded. With the goal of minimizing the reconstruction error and the KL divergence of the prior structure, the connection weights of the directed causal edges are calculated to obtain the edge weight set. The data contained in the multimodal dataset is used as nodes, and the directed causal edge set and the edge weight set are combined to output the device causal graph.

[0019] Preferably, the method of the self-supervised anomaly detection mechanism includes:

[0020] Receive multimodal data collected through a sliding window mechanism and perform zero-mean unit variance normalization to eliminate numerical dimension differences, thereby obtaining standardized multimodal data. Introduce two pre-text tasks to learn normal behavior patterns in an unlabeled environment; the two pre-text tasks are pre-text reconstruction and pre-text prediction.

[0021] The standardized multimodal data is reconstructed by reconstructing the pre-text and the reconstruction error is calculated. The reconstruction error is used as the loss function, and the loss function is minimized through back propagation and gradient descent to obtain the reconstruction error score, automatically learning the normal collaborative pattern between multimodal data;

[0022] By predicting the pre-text based on the recent The standardized multimodal data of the first time step is used to predict the multimodal data of the next time step; the prediction error is used as the loss function and minimized through back propagation and gradient descent to obtain the prediction error score, capturing the temporal dynamic characteristics between multimodal data;

[0023] The calculated reconstruction error score and prediction error score are weighted 50% each, and a joint anomaly score is obtained through weighted fusion. A threshold for the joint anomaly score is preset, and a confidence function is defined. The confidence level is calculated based on the joint anomaly score. The obtained confidence level is marked on the corresponding multimodal data, thereby obtaining a data stream with confidence labels.

[0024] A confidence threshold is preset, and confidence levels below the preset confidence threshold are recorded as abnormal confidence levels. For multimodal data with abnormal confidence levels, a freeze operation is triggered to temporarily suppress the update of the edge weights of the data in the device causal graph. For each freeze operation, the freeze duration is dynamically calculated based on the current abnormal confidence level.

[0025] Preferably, the method for obtaining the feature tensor includes:

[0026] Multi-scale convolution processing is performed on the data stream with confidence labels through a gated temporal convolutional network, and a gating mechanism is introduced to control the flow of information and extract time series features. A feature fusion path is constructed based on the device causal graph and confidence labels. The time series features of different modalities are aggregated in the device causal graph through graph path guidance to obtain a feature fusion vector. The L1 norm constraint is applied to the feature fusion vector to eliminate redundant features, and the feature dimension is compressed through sparse principal component analysis to generate a feature tensor.

[0027] Preferably, the method for obtaining the multi-strategy execution path includes:

[0028] Through the causal enhancement embedding mechanism, the feature tensor and the device causal graph are jointly mapped to the policy space to construct a joint representation vector. Based on the joint representation vector, the multi-agent curriculum learning mechanism is used to identify all policy paths under the current policy goal and generate a candidate policy space.

[0029] For each candidate policy path in the candidate policy space, a graph neural network is used to evaluate its execution utility, and the execution utility is calculated based on the execution success probability of the candidate policy path. A preset execution success probability threshold is set, and the execution utility of all candidate policy paths is evaluated. The candidate policy paths with a probability greater than the preset execution success threshold are screened out to form the final multi-policy execution path.

[0030] Preferably, the method for obtaining the multipath policy instruction set includes:

[0031] The multi-policy execution paths are mapped into a structured policy behavior tree. The joint representation vector is defined as the root node of the policy behavior tree. Taking each policy path as a guide, the state transition chain is recursively expanded based on the root node to convert each policy path into a complete branch of the policy behavior tree. A fault-tolerant distributed consensus protocol is introduced to broadcast the policy execution path of the policy behavior tree to all participating devices. When each device receives the policy execution instruction, Byzantine fault tolerance is used for consistency verification to check whether there are conflicts in the multi-policy execution paths. The policy execution paths without conflicts are retained to form a consistent path set, which is then converted into a structured instruction set to obtain the multi-path policy instruction set.

[0032] Preferably, the method for revising the preliminary group execution action plan includes:

[0033] Actions are parsed according to the multi-path policy instruction set, the policy is decomposed into device control instructions, the execution order is selected based on the device causal graph, the device control instructions and execution order are bound to the corresponding devices, and a preliminary group execution action plan is generated; the preliminary group execution action plan is corrected by calling the data stream with confidence labels, and correction instructions are generated based on the dynamic correction table to make adaptive adjustments to the control instructions of the corresponding devices.

[0034] Preferably, the method for implementing coordinated control between execution nodes includes:

[0035] The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes.

[0036] Adaptive adjustments are made to the control instructions of the corresponding devices. After completing the correction of the preliminary group execution action plan, execution feedback data is generated, and execution status data during the execution of the corrected instructions is collected at the same time. Based on the execution status data and the device causal graph, a device-instruction bipartite graph is constructed, and all control instructions are bound to the corresponding devices. A group topology optimization algorithm is used to minimize the global execution cost as the optimization goal, and resource sharing and information synchronization between devices are carried out, thereby realizing system control between each execution node.

[0037] Preferably, the method for integrated optimization management of multimodal devices includes:

[0038] Execution feedback data includes execution success / failure flags, action deviation values, feedback confidence traces, and fault trigger flags and causes. Execution status data includes node activity status, task completion rate indicators, average confidence and volatility indicators, and inter-node response delay matrix. Execution feedback data and execution status data are fused to form a multimodal tracking dataset. A time-aware neural network is used to model the state evolution of the fused multimodal tracking dataset, generating a state tracking model that covers the device operating state evolution path.

[0039] Based on the state tracking model, an execution evaluation indicator system including execution stability, strategy matching and multimodal consistency is constructed, and the equipment operation deviation analysis results and behavior consistency evaluation results are output. Based on the equipment operation deviation analysis results and behavior consistency evaluation results, a cognitive feedback pathway is constructed; based on the cognitive feedback pathway, the equipment coordination mechanism is dynamically adjusted to carry out integrated optimization management of multimodal equipment.

[0040] A multimodal device integrated management method based on intelligent AI, comprising:

[0041] S1. Use embedded causal analysis to perform real-time causal modeling on multimodal data. Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels.

[0042] S2. Based on the device causal graph and the data stream with confidence labels, a gated temporal convolutional network is used to extract time series features. The dynamic feature melting pool mechanism is used to reconstruct the feature path. The sparse regularization method is combined to compress and select high-dimensional features to generate a feature tensor.

[0043] S3. Generate multi-strategy execution paths based on feature tensors and device causal graphs; construct a policy behavior tree based on the multi-strategy execution paths through a multi-agent course learning mechanism; use a distributed consensus mechanism to verify the consistency of the policy behavior tree execution paths and output a multi-path policy instruction set;

[0044] S4 is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan; modify the preliminary group execution action plan based on the data stream with confidence labels, generate execution feedback data and execution status data, and combine the group topology optimization algorithm to achieve coordinated control between the execution nodes;

[0045] S5 is used to build a cognitive feedback pathway including multimodal state tracking and execution evaluation based on execution feedback data and execution status data, and to perform integrated optimization management of multimodal devices through an adaptive feedback mechanism.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] Real-time causal modeling accurately reflects the causal relationship between the internal operating mechanism of the equipment and external interference, which helps to deeply understand the operating principles of the equipment and provide a reliable basis for subsequent decision-making and control. Compared with traditional implicit association analysis methods, the causal relationship is highly interpretable and provides a clear logical path for fault tracing and prediction. The introduction of self-supervised anomaly detection tasks does not require manual labeling, and automatically learns the modal statistical characteristics, temporal dynamic laws, cross-modal correlation relationships and temporal and spatial consistency of normal equipment operation, solving the pain point of traditional supervision methods relying on a large number of abnormal samples and reducing data annotation costs. At the same time, combined with the multi-modal data triggering freezing operation based on the anomaly confidence level, it can effectively prevent abnormal data from interfering with the equipment's causal graph structure, ensuring the stability and reliability of the system.

[0048] The causal enhancement embedding mechanism fully considers the causal relationships between devices and policy goals, generating multi-policy execution paths that include causal dependencies. Compared with single-policy decision-making, the decision coverage in complex scenarios is significantly improved, and the policy adaptability is enhanced. Through the policy behavior tree, it gradually transitions from basic policies to complex collaborative policies, improving the efficiency of policy exploration and avoiding the cold start problem of traditional reinforcement learning.

[0049] A cognitive feedback pathway encompassing multimodal state tracking and execution evaluation is constructed, enabling integrated and optimized management of multimodal devices through an adaptive feedback mechanism. This enhances the long-term stability and intelligence of the system, enables real-time monitoring and evaluation of device operational status, and dynamically adjusts device coordination mechanisms based on evaluation results, improving device integrated management and operational efficiency.

[0050] By integrating cutting-edge technologies such as causal reasoning, self-supervised learning, and swarm topology optimization, we have built a full-link intelligent framework of "perception-feature-decision-execution-feedback", forming the advantages of cross-disciplinary technology collaboration and achieving accurate perception, efficient decision-making, precise execution, and intelligent optimization management of multimodal devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the structure of a multimodal device integrated management system based on intelligent AI of the present invention;

[0052] Figure 2 This is a flow chart of a multi-modal device integrated management method based on intelligent AI of the present invention;

[0053] Figure 3 This is a flow chart of the self-supervised anomaly detection mechanism method provided by the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1

[0056] See also Figure 1 and Figure 3 As shown, this embodiment 1 further illustrates a multimodal device integrated management system based on intelligent AI proposed by the present invention, including:

[0057] With the rapid development of artificial intelligence technology, a single modality can no longer meet the needs of complex tasks. Multimodal AI has broken through the limitations of a single modality by integrating multiple data sources such as text, images, audio, and video, significantly improving the expressiveness and generalization capabilities of the model. In the fields of industrial automation, smart home, medical diagnosis, etc., multimodal data fusion technology has become the core driving force for realizing intelligent equipment management. For example, autonomous driving systems achieve accurate perception of road environments by fusing camera, radar, and lidar data; medical diagnostic systems improve the accuracy of disease detection by integrating medical images and medical records. However, the existing multimodal equipment integrated management system has significant technical bottlenecks in data fusion, causal modeling, intelligent decision-making, and real-time control, as shown below:

[0058] In industrial scenarios, equipment often comes from different manufacturers, uses proprietary communication protocols, and has vastly different data formats (such as binary sensor signals and unstructured text logs). This necessitates customized gateways or middleware for protocol conversion during system integration, resulting in poor compatibility and high expansion costs. For example, traditional solutions for cross-modal fusion of vibration signals (high-frequency time series data) and equipment manuals (unstructured text) lack a unified spatiotemporal alignment mechanism, often leading to the loss of critical causal information due to inconsistent sampling rates and clock skew.

[0059] Traditional equipment management systems rely heavily on correlation analysis or black-box models to mine data associations. These models can only identify "data co-occurrence" and struggle to accurately reflect the causal relationship between the equipment's internal operating mechanisms and external interference (e.g., the causal relationship between abnormal motor current and bearing wear). This limitation leads to difficulties in fault tracing (a lack of a clear causal logical path makes it difficult to quickly locate the root cause), low anomaly detection efficiency (reliance on manually labeled anomaly samples, which is costly and susceptible to subjective factors), insufficient decision coverage (single policy decisions cannot adapt to complex scenarios and have poor policy adaptability), and poor system stability.

[0060] For example, when a piece of equipment experiences an abnormal temperature, traditional systems can only trigger an early warning, but it's difficult to pinpoint the root cause, whether it's excessive upstream vibration or a sudden load change. Furthermore, there's a lack of effective modeling for the temporal dynamics of multimodal data (such as periodic fluctuations) and cross-modal coupling relationships (such as the coordinated changes in speed and vibration). This results in a high rate of false alarms in anomaly detection (such as misjudgment of a single modal threshold trigger). Fault tracing relies on manual experience, making it inefficient.

[0061] Existing multimodal device integrated management systems have significant deficiencies in compatibility, real-time performance, intelligence, and security, making them difficult to meet the complex demands of industries such as industry, healthcare, and energy. To address this issue, this invention strives to provide an efficient and reliable solution for cross-industry device management through systematic innovations in protocol standardization, real-time fusion of multimodal data, intelligent decision-making, and security and privacy protection.

[0062] Therefore, in order to effectively solve the above problems, the present invention proposes a multimodal device integrated management system based on intelligent AI, including:

[0063] The autonomous perception module uses embedded causal analysis to perform real-time causal modeling on multimodal data. Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels.

[0064] The Liquid Feature Factory module uses a gated temporal convolutional network to extract time series features based on the device causal graph and confidence-labeled data streams. It then reconstructs feature paths through a dynamic feature melting pool mechanism and combines sparse regularization methods to compress and select high-dimensional features, ultimately generating feature tensors.

[0065] The path decision module generates multi-strategy execution paths based on feature tensors and device causal graphs. Through a multi-agent course learning mechanism, it constructs a policy behavior tree based on the multi-strategy execution paths. It uses a distributed consensus mechanism to verify the consistency of the execution paths of the policy behavior tree and output a multi-path policy instruction set.

[0066] The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes.

[0067] The cognitive reflection coordination module is used to build a cognitive feedback pathway including multimodal state tracking and execution evaluation based on execution feedback data and execution status data, and to perform integrated optimization management of multimodal devices through an adaptive feedback mechanism.

[0068] Methods for real-time causal modeling include:

[0069] The system collects heterogeneous data from various sensor terminals, such as images, temperature, acceleration, vibration, voltage, and current generated during the operation of each device. Based on the local clock synchronization signal, the heterogeneous data is aligned to a unified sampling time axis through interpolation. Spatial nearest neighbor interpolation is used to fill missing values, and noise filtering is performed on the completed heterogeneous data. This completes the unified spatiotemporal standardization of the heterogeneous data and obtains multimodal data.

[0070] A time window length is preset, and multimodal data is collected through a sliding time window mechanism to form a multimodal dataset. The multimodal dataset consists of multimodal data collected at all time points within the initial time window. A full-variable lag prediction model is constructed, using the multimodal dataset as input, and Granger causality test is performed to obtain a set of directed causal edges.

[0071] The multimodal dataset is input into the structural inference layer of the variational autoencoder, and the distribution characteristics of each multimodal data are encoded. With the goal of minimizing the reconstruction error and the KL divergence of the prior structure, the connection weights of the directed causal edges are calculated to obtain the edge weight set. The data contained in the multimodal dataset are used as nodes. The directed causal edge set and the edge weight set are combined to output the device causal graph, which is a directed graph reflecting the causal relationship between the internal operating mechanism of the device and external interference.

[0072] Methods for self-supervised anomaly detection mechanisms include:

[0073] Receive multimodal data collected through a sliding window mechanism and perform zero-mean unit variance normalization to eliminate numerical dimension differences, thereby obtaining standardized multimodal data. Introduce two pre-text tasks to learn normal behavior patterns in an unlabeled environment; the two pre-text tasks are pre-text reconstruction and pre-text prediction.

[0074] Learning normal behavior patterns means automatically inferring the data characteristics and temporal patterns of "devices in normal working condition" from historical data without any manual annotation (i.e., without "abnormal" or "normal" sample labels). This allows the system to construct an internal cognitive model of the device's "normal" state, specifically including intra-modal statistical characteristics, temporal dynamic patterns, cross-modal correlations, and spatial-temporal consistency.

[0075] Intramodal statistical features: Model the statistical characteristics of the amplitude distribution, variance, and spectral components of heterogeneous data (such as temperature, current, and vibration signals) acquired by each sensor terminal during normal operation. For example, the average value of a temperature sensor during normal periods is stable within a certain range, and vibration signals do not experience high-frequency mutations during normal periods.

[0076] Time series dynamics: Capturing the temporal evolution of multimodal data, such as periodic fluctuations, trend changes, or sudden jumps; measuring whether the current observation deviates from "normal" dynamics by predicting whether the data value at the next time step conforms to the historical trend;

[0077] Cross-modal correlations: Under normal conditions, different modes often have stable coupling relationships. For example, when motor speed increases, vibration and current rise simultaneously. When the load is stable, temperature and current change in a coordinated manner. After the system learns these typical coordination patterns, it can identify abnormalities when one mode is normal and another mode is deviating.

[0078] Spatial-temporal consistency: When the equipment is operating normally, the data changes coherently and smoothly in both space (between different devices or sensing points) and time. Sudden jumps and isolated abnormal points often indicate failures or abnormal events.

[0079] The standardized multimodal data is reconstructed by reconstructing the pre-text and the reconstruction error is calculated. The reconstruction error is used as the loss function, and the loss function is minimized through back propagation and gradient descent to ensure that the reconstructed data can restore the input as accurately as possible. The reconstruction error score is obtained and the normal coordination pattern between multimodal data is automatically learned;

[0080] By predicting the pre-text based on the recent The standardized multimodal data of a time step is used to predict the multimodal data of the next time step; the prediction error is used as the loss function and minimized through back propagation and gradient descent to obtain the prediction error score, thereby strengthening the learning ability of multimodal time series dynamics and capturing the time series dynamic characteristics between multimodal data;

[0081] How pre-text tasks help "normal" mode learning:

[0082] Reconstructing the pretext: The system learns to "restore" the input data stream. If the system can accurately reconstruct the data, it is likely that the compression and decoding were performed according to the normal mode. If the reconstruction error is too large, it means that the input data deviates from the normal feature distribution.

[0083] Prediction: The system is asked to "predict" the next step given data from the past few steps. If the prediction error is small, it means that the time series follows the normal dynamic pattern learned by the system; if the error is significant, it indicates that the current pattern is deviating.

[0084] Through these two tasks, we can spontaneously characterize "normal behavior" without any abnormal labels, including both the statistical characteristics of a single modality and the coupled temporal relationships between multiple modalities. Any input data that deviates from this "learned normal pattern" will produce high errors in the reconstruction or prediction phase and will be judged as a potential anomaly.

[0085] The reconstruction error score and prediction error score obtained by calculation are weighted 50% respectively, and the joint anomaly score is obtained by weighted fusion; the joint anomaly score threshold is preset, and the confidence function is defined, and the confidence is obtained according to the joint anomaly score calculation; the confidence function is ,in, is the timestamp; is a control factor used to control the mapping sensitivity of the joint anomaly score and confidence; for the joint abnormality score; is the preset joint anomaly score threshold; the obtained confidence is marked on the corresponding multimodal data, thereby obtaining a data stream with confidence labels;

[0086] A confidence threshold is preset, and the confidence level below the preset confidence threshold is recorded as abnormal confidence level. For multimodal data with abnormal confidence level, a freeze operation is triggered to temporarily suppress the update of the edge weight of the data in the device causal graph to avoid the introduction of false causal relationships by abnormal data. For each freeze operation, the freeze duration is dynamically calculated based on the current abnormal confidence level. ;in, The preset maximum freezing time; The preset minimum freezing time; The confidence threshold is preset. That is to say, if the anomaly confidence is closer to the preset confidence threshold, the freeze duration will be closer to the shortest value. Conversely, the freeze duration will be closer to the preset maximum freeze duration. The freeze operation can adaptively adjust the freeze time according to the severity of the anomaly to prevent abnormal data from continuously interfering with the device causal graph structure.

[0087] For example, in a smart wind farm environment, sensors on equipment nodes such as wind turbine blades, main shafts, gearboxes, and pitch control systems collect multimodal data streams (including vibration, temperature, current, voltage, and wind speed). Based on this data, the system constructs a causal graph of equipment operation to learn the coupling between devices and fault propagation paths. Certain abnormal data (such as sensor misalignment, electromagnetic interference, and transient jumps) can misleadingly enhance or misconnect edge weights in the causal graph, leading to misjudgments or the generation of false root cause chains.

[0088] The system targets each moment , respectively calculate the reconstruction error score (such as based on the variational autoencoder VAE output); prediction error score (such as based on the time series prediction network); set equal weight (50%) fusion to obtain the joint anomaly score :The preset joint anomaly score threshold is 0.4, the control factor is -6, then the confidence ;

[0089] The preset confidence threshold is 0.7. Since 0.658 < 0.7, it is determined to be abnormal confidence data; the preset freezing time range is the preset shortest freezing time. The preset maximum freeze time is 5 seconds. 60 seconds; dynamic calculation of freezing time The edge weight updates of the device nodes corresponding to the multimodal data will be frozen within 8.3 seconds to avoid short-term abnormalities interfering with the learning process.

[0090] Methods for obtaining feature tensors include:

[0091] Multi-scale convolution processing is performed on data streams with confidence labels through a gated temporal convolutional network, and a gating mechanism is introduced to control the flow of information to enhance the response to key change trends and temporal dependencies and extract time series features. A feature fusion path is constructed based on the device causal graph and confidence labels, and different modal time series features are aggregated in the device causal graph through graph path guidance to obtain a feature fusion vector. The L1 norm constraint is applied to the feature fusion vector to eliminate redundant features, and the feature dimension is compressed through sparse principal component analysis to generate a low-dimensional, highly expressive feature tensor.

[0092] Methods for obtaining multi-strategy execution paths include:

[0093] Through the causal enhancement embedding mechanism, the feature tensor and the device causal graph are jointly mapped to the policy space to construct a joint representation vector. Based on the joint representation vector, the multi-agent curriculum learning mechanism is used to identify all policy paths under the current policy goal and generate a candidate policy space. The candidate policy space expresses the action dependencies and causal advancement logic between devices in the form of a policy path graph.

[0094] For each candidate policy path in the candidate policy space, a graph neural network is used to evaluate its execution utility, and the execution utility is calculated based on the execution success probability of the candidate policy path. A preset execution success probability threshold is set, and the execution utility of all candidate policy paths is evaluated. The candidate policy paths with a probability greater than the preset execution success threshold are screened out to form the final multi-policy execution path.

[0095] Methods for obtaining the multipath policy instruction set include:

[0096] The multi-policy execution paths are mapped into a structured policy behavior tree. A joint representation vector is defined as the root node of the policy behavior tree. Each policy path is used as a guide, and the state transition chain is recursively expanded based on the root node to convert each policy path into a complete branch of the policy behavior tree. A fault-tolerant distributed consensus protocol (such as PBFT and Raft) is introduced to broadcast the policy execution path of the policy behavior tree to all participating devices. When each device receives the policy execution instruction, Byzantine fault tolerance is used for consistency verification to check whether there are conflicts in the multi-policy execution paths (such as event sequence conflicts and time constraint violations). The policy execution paths without conflicts are retained to form a consistent path set, which is converted into a structured instruction set to obtain the multi-path policy instruction set. The multi-path policy instruction set contains the following fields: instruction number, policy path identifier, instruction content, execution condition, instruction confidence, and expected feedback constraint.

[0097] Methods for modifying the initial group execution action plan include:

[0098] Actions are parsed according to the multi-path policy instruction set, the policy is decomposed into device control instructions, the execution order is selected based on the device causal graph, the device control instructions and execution order are bound to the corresponding devices, and a preliminary group execution action plan is generated; the preliminary group execution action plan is corrected by calling the data stream with confidence labels, and correction instructions are generated based on the dynamic correction table to make adaptive adjustments to the control instructions of the corresponding devices.

[0099] For example:

[0100] Table 1 Dynamic correction table

[0101] Confidence Category Corrective Action Example Too low (confidence < 0.3) Device task suspension and task migration The signal quality of a certain fan device is poor, so execution is suspended. Medium (0.3 ≤ confidence ≤ 0.6) Parameter adjustment, dynamic delay The heater's current feedback is abnormal, and the heating action is delayed for 5 seconds High (confidence > 0.6) Enable bypass or fault-tolerant redundancy The main power supply is overloaded and switches to the backup power supply

[0102] Methods for achieving coordinated control between execution nodes include:

[0103] The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes.

[0104] Adaptive adjustments are made to the control instructions of the corresponding devices. After completing the correction of the preliminary group execution action plan, execution feedback data is generated, and execution status data during the execution of the corrected instructions is collected at the same time. Based on the execution status data and the device causal graph, a device-instruction bipartite graph is constructed, and all control instructions are bound to the corresponding devices. A group topology optimization algorithm is used to minimize the global execution cost as the optimization goal, and resource sharing and information synchronization between devices are carried out, thereby realizing system control between each execution node.

[0105] Methods for integrated optimization management of multimodal devices include:

[0106] Execution feedback data includes execution success / failure flags (marking whether the execution was completed as planned), action deviation values (the difference between the actual execution and the original plan, such as a 3-second extension in time), feedback confidence trajectory (the trajectory of confidence changes during execution), and fault trigger flags and causes (if the system stops due to an abnormality, the fault trigger event and sensor terminal abnormality description are recorded). Execution status data is used to characterize the current overall operating status of the swarm system, including node activity status (which nodes are online, offline, or degraded), task completion rate indicators (the proportion of the current action plan completed), average confidence and volatility indicators (the stability of the current device's overall perception), and an inter-node response delay matrix (the delay of each device in responding to control signals). Execution feedback data and execution status data are fused to form a multimodal tracking dataset. A time-aware neural network is used to model the state evolution of the fused multimodal tracking dataset, generating a state tracking model that covers the evolution path of the device's operating state.

[0107] Based on the state tracking model, an execution evaluation indicator system including execution stability, strategy matching, and multimodal consistency is constructed. The results of equipment operation deviation analysis and behavior consistency evaluation are output, and a cognitive feedback pathway is constructed based on these results.

[0108] Based on the cognitive feedback pathway, the device coordination mechanism is dynamically adjusted to perform integrated optimization management of multimodal devices. Dynamic adjustment of the device coordination mechanism includes adjusting the task allocation ratio between devices, updating the communication topology, and adjusting the device working mode.

[0109] The preset joint anomaly score threshold is set by the staff by taking the average of multiple joint anomaly scores as the preset joint anomaly score threshold; similarly, the preset confidence threshold, the preset execution success probability threshold and the preset correction deviation threshold are set.

[0110] This embodiment accurately reflects the causal relationship between the internal operating mechanism of the equipment and external interference through real-time causal modeling, which helps to deeply understand the operating principle of the equipment and provide a reliable basis for subsequent decision-making and control. Compared with the traditional implicit association analysis method, the causal relationship is highly interpretable, providing a clear logical path for fault tracing and prediction. The introduction of self-supervised anomaly detection tasks does not require manual labeling, and automatically learns the modal statistical characteristics, temporal dynamic laws, cross-modal correlation relationships and temporal and spatial consistency of normal operation of the equipment, solving the pain point of traditional supervision methods relying on a large number of abnormal samples and reducing data labeling costs. At the same time, combined with the multi-modal data triggering freezing operation based on the anomaly confidence level, it can effectively prevent abnormal data from interfering with the causal graph structure of the equipment, ensuring the stability and reliability of the system.

[0111] The causal enhancement embedding mechanism fully considers the causal relationships between devices and policy goals, generating multi-policy execution paths that include causal dependencies. Compared with single-policy decision-making, the decision coverage in complex scenarios is significantly improved, and the policy adaptability is enhanced. Through the policy behavior tree, it gradually transitions from basic policies to complex collaborative policies, improving the efficiency of policy exploration and avoiding the cold start problem of traditional reinforcement learning.

[0112] A cognitive feedback pathway encompassing multimodal state tracking and execution evaluation is constructed, enabling integrated and optimized management of multimodal devices through an adaptive feedback mechanism. This enhances the long-term stability and intelligence of the system, enables real-time monitoring and evaluation of device operational status, and dynamically adjusts device coordination mechanisms based on evaluation results, improving device integrated management and operational efficiency.

[0113] By integrating cutting-edge technologies such as causal reasoning, self-supervised learning, and swarm topology optimization, we have built a full-link intelligent framework of "perception-feature-decision-execution-feedback", forming the advantages of cross-disciplinary technology collaboration and achieving accurate perception, efficient decision-making, precise execution, and intelligent optimization management of multimodal devices.

[0114] Example 2

[0115] See also Figure 2 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A multimodal device integrated management method based on intelligent AI is provided, including:

[0116] S1. Use embedded causal analysis to perform real-time causal modeling on multimodal data. Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels.

[0117] S2. Based on the device causal graph and the data stream with confidence labels, a gated temporal convolutional network is used to extract time series features. The dynamic feature melting pool mechanism is used to reconstruct the feature path. The sparse regularization method is combined to compress and select high-dimensional features to generate a feature tensor.

[0118] S3. Generate multi-strategy execution paths based on feature tensors and device causal graphs; construct a policy behavior tree based on the multi-strategy execution paths through a multi-agent course learning mechanism; use a distributed consensus mechanism to verify the consistency of the policy behavior tree execution paths and output a multi-path policy instruction set;

[0119] S4 is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan; modify the preliminary group execution action plan based on the data stream with confidence labels, generate execution feedback data and execution status data, and combine the group topology optimization algorithm to achieve coordinated control between the execution nodes;

[0120] S5 is used to build a cognitive feedback pathway including multimodal state tracking and execution evaluation based on execution feedback data and execution status data, and to perform integrated optimization management of multimodal devices through an adaptive feedback mechanism.

[0121] Since the electronic device introduced in this embodiment is an electronic device used to implement a multimodal device integrated management system and method based on intelligent AI in the embodiment of this application, based on the multimodal device integrated management system and method based on intelligent AI introduced in the embodiment of this application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application will not be introduced in detail here. As long as technical personnel in this field implement the electronic device used in the embodiment of this application based on intelligent AI and the method, they all fall within the scope of protection of this application.

[0122] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0123] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for users of ordinary skill in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal equipment integrated management system based on intelligent AI, characterized in that: include: Autonomous perception module, which uses embedded causal analysis to perform real-time causal modeling of multimodal data; Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels; The real-time causal modeling method includes: Heterogeneous data from various sensor terminals is collected and aligned to a unified sampling time axis using interpolation based on local clock synchronization signals. Missing values are then filled using the spatial nearest neighbor interpolation method. The filled heterogeneous data is then subjected to noise filtering to achieve unified spatiotemporal standardization of the heterogeneous data and obtain multimodal data. The Liquid Feature Factory module uses a gated temporal convolutional network to extract time series features based on the device causal graph and confidence-labeled data streams. It then reconstructs feature paths through a dynamic feature melting pool mechanism and combines sparse regularization methods to compress and select high-dimensional features, ultimately generating feature tensors. The path decision module generates multi-strategy execution paths based on feature tensors and device causal graphs. Through a multi-agent course learning mechanism, it constructs a policy behavior tree based on the multi-strategy execution paths. It uses a distributed consensus mechanism to verify the consistency of the execution paths of the policy behavior tree and output a multi-path policy instruction set. The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes. The cognitive reflection coordination module is used to build a cognitive feedback pathway including multimodal state tracking and execution evaluation based on execution feedback data and execution status data, and to perform integrated optimization management of multimodal devices through an adaptive feedback mechanism.

2. The multimodal equipment integrated management system based on intelligent AI according to claim 1, characterized in that: The method of real-time causal modeling further includes: A time window length is preset, and multimodal data is collected through a sliding time window mechanism to form a multimodal dataset. The multimodal dataset consists of multimodal data collected at all time points within the initial time window. A full-variable lag prediction model is constructed, using the multimodal dataset as input, and Granger causality test is performed to obtain a set of directed causal edges. The multimodal dataset is input into the structural inference layer of the variational autoencoder, and the distribution characteristics of each multimodal data are encoded. With the goal of minimizing the reconstruction error and the KL divergence of the prior structure, the connection weights of the directed causal edges are calculated to obtain the edge weight set. The data contained in the multimodal dataset is used as nodes, and the directed causal edge set and the edge weight set are combined to output the device causal graph.

3. The multimodal equipment integrated management system based on intelligent AI according to claim 2 is characterized in that: The method of the self-supervised anomaly detection mechanism includes: Receive multimodal data collected through a sliding window mechanism and perform zero-mean unit variance normalization to eliminate numerical dimension differences, thereby obtaining standardized multimodal data. Introduce two pre-text tasks to learn normal behavior patterns in an unlabeled environment; the two pre-text tasks are pre-text reconstruction and pre-text prediction. The standardized multimodal data is reconstructed by reconstructing the pre-text and the reconstruction error is calculated. The reconstruction error is used as the loss function, and the loss function is minimized through back propagation and gradient descent to obtain the reconstruction error score, automatically learning the normal collaborative pattern between multimodal data; By predicting the pre-text based on the recent The standardized multimodal data of the first time step is used to predict the multimodal data of the next time step; the prediction error is used as the loss function and minimized through back propagation and gradient descent to obtain the prediction error score, capturing the temporal dynamic characteristics between multimodal data; The calculated reconstruction error score and prediction error score are weighted 50% each, and a joint anomaly score is obtained through weighted fusion. A threshold for the joint anomaly score is preset, and a confidence function is defined. The confidence level is calculated based on the joint anomaly score. The obtained confidence level is marked on the corresponding multimodal data, thereby obtaining a data stream with confidence labels. A confidence threshold is preset, and confidence levels below the preset confidence threshold are recorded as abnormal confidence levels. For multimodal data with abnormal confidence levels, a freeze operation is triggered to temporarily suppress the update of the edge weights of the data in the device causal graph. For each freeze operation, the freeze duration is dynamically calculated based on the current abnormal confidence level.

4. The multimodal equipment integrated management system based on intelligent AI according to claim 3 is characterized in that: The method for obtaining the feature tensor includes: Multi-scale convolution processing is performed on the data stream with confidence labels through a gated temporal convolutional network, and a gating mechanism is introduced to control the flow of information and extract time series features. A feature fusion path is constructed based on the device causal graph and confidence labels. The time series features of different modalities are aggregated in the device causal graph through graph path guidance to obtain a feature fusion vector. The L1 norm constraint is applied to the feature fusion vector to eliminate redundant features, and the feature dimension is compressed through sparse principal component analysis to generate a feature tensor.

5. The multimodal equipment integrated management system based on intelligent AI according to claim 4 is characterized in that: The method for obtaining the multi-strategy execution path includes: Through the causal enhancement embedding mechanism, the feature tensor and the device causal graph are jointly mapped to the policy space to construct a joint representation vector. Based on the joint representation vector, the multi-agent curriculum learning mechanism is used to identify all policy paths under the current policy goal and generate a candidate policy space. For each candidate policy path in the candidate policy space, a graph neural network is used to evaluate its execution utility, and the execution utility is calculated based on the execution success probability of the candidate policy path. A preset execution success probability threshold is set, and the execution utility of all candidate policy paths is evaluated. The candidate policy paths with a probability greater than the preset execution success threshold are screened out to form the final multi-policy execution path.

6. The multimodal equipment integrated management system based on intelligent AI according to claim 5, characterized in that: The method for obtaining the multipath strategy instruction set includes: The multi-policy execution paths are mapped into a structured policy behavior tree. The joint representation vector is defined as the root node of the policy behavior tree. Taking each policy path as a guide, the state transition chain is recursively expanded based on the root node to convert each policy path into a complete branch of the policy behavior tree. A fault-tolerant distributed consensus protocol is introduced to broadcast the policy execution path of the policy behavior tree to all participating devices. When each device receives the policy execution instruction, Byzantine fault tolerance is used for consistency verification to check whether there are conflicts in the multi-policy execution paths. The policy execution paths without conflicts are retained to form a consistent path set, which is then converted into a structured instruction set to obtain the multi-path policy instruction set.

7. The multimodal equipment integrated management system based on intelligent AI according to claim 6, characterized in that: The method for revising the preliminary group execution action plan includes: Actions are parsed according to the multi-path policy instruction set, the policy is decomposed into device control instructions, the execution order is selected based on the device causal graph, the device control instructions and execution order are bound to the corresponding devices, and a preliminary group execution action plan is generated; the preliminary group execution action plan is corrected by calling the data stream with confidence labels, and correction instructions are generated based on the dynamic correction table to make adaptive adjustments to the control instructions of the corresponding devices.

8. The multimodal equipment integrated management system based on intelligent AI according to claim 7, characterized in that: The method for achieving coordinated control between execution nodes includes: The execution control module is used to parse the multi-path strategy instruction set and generate a preliminary group execution action plan. It modifies the preliminary group execution action plan based on the data stream with confidence labels, generates execution feedback data and execution status data, and combines the group topology optimization algorithm to achieve coordinated control between the execution nodes. Adaptive adjustments are made to the control instructions of the corresponding devices. After completing the correction of the preliminary group execution action plan, execution feedback data is generated, and execution status data during the execution of the corrected instructions is collected at the same time. Based on the execution status data and the device causal graph, a device-instruction bipartite graph is constructed, and all control instructions are bound to the corresponding devices. A group topology optimization algorithm is used to minimize the global execution cost as the optimization goal, and resource sharing and information synchronization between devices are carried out, thereby realizing system control between each execution node.

9. The multimodal equipment integrated management system based on intelligent AI according to claim 8, characterized in that: The method for integrated optimization management of multimodal devices includes: Execution feedback data includes execution success / failure flags, action deviation values, feedback confidence traces, and fault trigger flags and causes. Execution status data includes node activity status, task completion rate indicators, average confidence and volatility indicators, and inter-node response delay matrix. Execution feedback data and execution status data are fused to form a multimodal tracking dataset. A time-aware neural network is used to model the state evolution of the fused multimodal tracking dataset, generating a state tracking model that covers the device operating state evolution path. Based on the state tracking model, an execution evaluation indicator system including execution stability, strategy matching and multimodal consistency is constructed, and the equipment operation deviation analysis results and behavior consistency evaluation results are output. Based on the equipment operation deviation analysis results and behavior consistency evaluation results, a cognitive feedback pathway is constructed; based on the cognitive feedback pathway, the equipment coordination mechanism is dynamically adjusted to carry out integrated optimization management of multimodal equipment.

10. A multimodal device integrated management method based on intelligent AI, used to implement a multimodal device integrated management system based on intelligent AI according to any one of claims 1 to 9, characterized in that: include: S1. Real-time causal modeling of multimodal data using embedded causal analysis; Combined with a self-supervised anomaly detection mechanism, it outputs a device causal graph and a data stream with confidence labels; S2. Based on the device causal graph and the data stream with confidence labels, a gated temporal convolutional network is used to extract time series features. The dynamic feature melting pool mechanism is used to reconstruct the feature path. The sparse regularization method is combined to compress and select high-dimensional features to generate a feature tensor. S3. Generate multi-strategy execution paths based on feature tensors and device causal graphs; construct a policy behavior tree based on the multi-strategy execution paths through a multi-agent course learning mechanism; Use the distributed consensus mechanism to verify the consistency of the execution path of the policy behavior tree and output a multi-path policy instruction set; S4. Parse the multi-path strategy instruction set and generate a preliminary swarm execution action plan. Modify the preliminary swarm execution action plan based on the data stream with confidence labels, generate execution feedback data and execution status data, and combine the swarm topology optimization algorithm to achieve coordinated control between the execution nodes. S5. Based on the execution feedback data and execution status data, a cognitive feedback pathway including multimodal status tracking and execution evaluation is constructed, and integrated optimization management of multimodal devices is performed through an adaptive feedback mechanism.

Images