Multi-modal information fusion and knowledge driven device state management method and system

Abstract

The application provides a multi-modal information fusion and knowledge-driven device state management method and system, relates to the technical field of intelligent operation and maintenance and device health management, and realizes accurate perception and intelligent maintenance of the whole life cycle state of key devices on the apron through construction of a dynamic threshold abnormality detection system, a multi-working-condition performance degradation benchmark library and a data-knowledge dual-driven decision framework. The application effectively solves the problem of insufficient adaptability of traditional methods under complex working conditions, significantly improves the accuracy of device health evaluation and the reliability of fault prediction, and provides core technical support for intelligent operation and maintenance of an airport.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance and equipment health management technology, specifically to a multimodal information fusion and knowledge-driven equipment status management method and system. Background Technology

[0002] With the expansion of airport operations and the increasing demand for intelligent systems, apron equipment, as a core infrastructure ensuring flight safety and efficient operation, faces severe challenges in health management. Currently, apron equipment maintenance mainly relies on periodic inspections and post-failure repairs, which suffers from delayed response and resource waste. Furthermore, traditional predictive maintenance methods lack accuracy and adaptability under complex operating conditions, failing to meet the needs of modern airport's refined management.

[0003] The current apron equipment management system mainly faces the following technical bottlenecks: 1) Difficulty in fusion of multi-source heterogeneous data: Apron equipment sensors are diverse (such as voltage, current, pressure, flow, temperature, vibration, etc.), with large differences in data sampling frequency and dimensionality. Traditional threshold detection methods are difficult to adapt to dynamic operating conditions and are abnormally sensitive to sensor drift, communication noise, etc., resulting in low data quality; 2) Insufficient accuracy of health status assessment: Existing assessment models (such as threshold alarms based on a single index) do not consider the multi-condition operating characteristics and nonlinear performance degradation features of equipment. In particular, for highly coupled systems, they lack cross-equipment correlation analysis capabilities, resulting in high false alarm and false negative rates; 3) Separation between fault prediction and decision-making: Mainstream prediction methods (such as ARIMA and SVM) rely on pure data-driven approaches, ignoring prior information such as equipment topology relationships and maintenance knowledge, resulting in poor interpretability of prediction results; while reasoning systems based on expert rules are lagging behind in updates and are unable to cope with new equipment fault modes.

[0004] In recent years, multimodal information fusion and heterogeneous knowledge graph technologies have provided new approaches to solving the aforementioned problems. For example, Kalman filtering can be used for multi-sensor data fusion, but its linear assumption limits its application in nonlinear degradation scenarios; deep learning models (such as LSTM) can capture temporal features, but their generalization ability for few-sample fault data is insufficient. Furthermore, while knowledge graph technology can integrate operational experience, static graphs struggle to adapt to the dynamic evolution of equipment status. Existing research has not effectively addressed the two core issues of data-knowledge collaboration and cross-modal dynamic modeling, hindering breakthroughs in the level of intelligent apron equipment management.

[0005] In summary, existing technologies fall short of meeting the three key requirements of apron equipment management: efficient cleaning of multi-source data, accurate health assessment under complex operating conditions, and intelligent decision-making through the fusion of data and knowledge. Especially in scenarios such as extreme weather and high-load operation, the reliability of traditional systems significantly decreases. Therefore, a novel intelligent management method that integrates multimodal sensor data and domain knowledge is urgently needed. Summary of the Invention

[0006] This invention addresses the technical problems in existing apron equipment condition management technologies, such as difficulties in multi-source data fusion, insufficient accuracy in health assessment, and the disconnect between fault prediction and decision-making. It proposes a multimodal information fusion and knowledge-driven equipment condition management method and system. By constructing a dynamic threshold anomaly detection system, a multi-condition performance degradation benchmark library, and a data-knowledge dual-driven decision-making framework, it achieves accurate perception and intelligent maintenance of the entire lifecycle status of critical apron equipment. This invention effectively solves the problem of insufficient adaptability of traditional methods under complex operating conditions, significantly improves the accuracy of equipment health assessment and the reliability of fault prediction, and provides core technical support for intelligent airport operation and maintenance.

[0007] The multimodal information fusion and knowledge-driven device status management method of the present invention includes the following steps: S1. Multi-source heterogeneous data anomaly cleaning: Multi-source heterogeneous sensor data is collected in real time through the sensor array of the apron equipment. This multi-source heterogeneous sensor data is historical runtime sequence data. A sliding window algorithm is used for dynamic threshold anomaly detection, combined with wavelet transform for multi-scale noise filtering. The fault knowledge in the maintenance work order is used to classify, statistically analyze and quantify the data after multi-scale noise filtering. Based on the statistical characteristics of the data, constraint rules and anomaly thresholds based on expert experience are formulated to identify and correct the abnormal data, resulting in corrected historical runtime sequence data of the equipment. Intelligent Equipment Health Status Assessment: Based on corrected historical operating sequence data of the equipment, a performance degradation benchmark database containing multi-condition features is constructed using dynamic time warping algorithms and expert experience. On this basis, time-domain features are extracted from the benchmark database to construct an equipment state representation vector. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data to achieve equipment reliability assessment and parameter updates. Furthermore, a Weibull distribution-based full lifecycle reliability model is established based on this state representation vector, and the Weibull distribution parameters are dynamically updated using online equipment data to calculate the cumulative failure probability of the equipment in real time. Finally, by integrating the current operating sequence data, the performance degradation benchmark database, and the real-time failure probability, an unsupervised adaptive clustering algorithm is used to construct an equipment health status assessment system, enabling intelligent identification and hierarchical early warning of health status.

[0008] S3. Equipment Fault Prediction and Intelligent Decision-Making: Based on the equipment state representation vector obtained in S2, a CNN-LSTM-Transformer joint prediction model is constructed. Gated fusion and Bayesian optimization are used to predict the fault probability distribution. Furthermore, a dynamic operation and maintenance knowledge graph is constructed based on the fault triplet joint failure mode and impact analysis strategy calculated in S1. This dynamic knowledge graph is used to generate logical associations and confidence levels for equipment faults. Building upon these two steps, the fault probability distribution obtained from the prediction model and the logical associations and confidence levels obtained from the knowledge graph are fused using DS evidence theory to obtain the decision results. Finally, priority maintenance suggestions are generated by combining real-time resource constraints, and the prediction curves, propagation paths, and handling solutions are intuitively displayed through a multi-dimensional interface, supporting rapid and accurate intervention by operation and maintenance personnel.

[0009] Preferably, in step S1, the specific steps for constructing a multi-source heterogeneous data anomaly cleaning strategy are as follows: S101. Real-time monitoring data is collected through sensor arrays deployed on key apron equipment, including voltage and current signals of the berth guidance system, harmonic distortion rate of the ground power supply system, refrigerant pressure and temperature of the air conditioning unit, and flow and pressure data of water supply and drainage pipelines. S102. An adaptive threshold detection model is constructed using the sliding window Z-Score algorithm to perform anomaly detection on the collected time-series data, thus performing initial screening of abnormal data. Anomaly detection can identify sensor drift, communication interruption, and other transient interference or abrupt changes that may affect data quality, providing a preliminary screening basis for subsequent wavelet transform noise reduction processing. The dynamic decision boundary in the adaptive threshold detection model is determined by the following formula. ; In the formula, To dynamically determine the boundary; and These are the mean and standard deviation of the data within the current sliding window, respectively. The window length is set according to the device's sampling frequency. k This is the sensitivity adjustment coefficient; α The threshold change rate during operating condition switching; This is the timestamp of the last operating condition switch. t This is the current sampling time point; S103. After anomaly detection, the data undergoes multi-scale noise filtering using wavelet transform; wavelet basis functions are used to decompose the signal, retaining low-frequency useful components. To address the issue of high-frequency noise and effective signal aliasing in airport equipment, a multi-scale thresholding strategy is employed, with thresholds set for each decomposition layer. T j The calculation formula is as follows: ; In the formula, σj For the first j The standard deviation of the layer wavelet decomposition coefficients is calculated using the db4 wavelet basis function. N j This represents the number of effective coefficients at the current scale, and its value varies with the number of decomposition levels. j Increases and decreases; SNR j The first obtained through the noise power estimation module j Layer signal-to-noise ratio.

[0010] S104. For unstructured maintenance work order texts, the BERT-BiLSTM model is used for entity recognition to extract key fault description triples (faulty component, fault phenomenon, possible cause). The BERT-BiLSTM-based entity recognition model determines fault elements through joint probability output. t The predicted probability of the label for each word is: ; In the formula, W s This represents the weight matrix of the fully connected layer; The context-dependent feature vectors output by the BERT model; The sequence feature vectors extracted by the BiLSTM layer are used to capture the temporal dependencies in fault description; This represents a vector concatenation operation that merges two types of features into a hybrid feature. ；b s The bias term is used to adjust the classification boundary; the Softmax function normalizes the linear transformation result into a probability distribution, and finally outputs the first... t The probability of each word belonging to each entity label; the continuous faulty parts, fault phenomena and possible causes in the text are identified through the BIO annotation system, and a faulty part-fault phenomenon-possible cause triplet is constructed based on word position and semantic dependency relationship; S105. Based on the extracted faulty component-fault phenomenon-possible cause triplet (as a classification label), the data after multi-scale noise filtering is classified, statistically analyzed, and quantified. The statistical characteristics of data corresponding to the same type of faulty component, fault phenomenon, or cause are calculated respectively. Furthermore, based on the statistical characteristics, constraint rules and anomaly thresholds based on expert experience are formulated to identify and correct abnormal data, resulting in corrected historical operating sequence data of the equipment. The time-series data involved includes, but is not limited to: voltage and current signals of the berth guidance system, harmonic distortion rate of the ground power supply system, refrigerant pressure and temperature of the air conditioning unit, and flow and pressure data of the water supply and drainage pipelines.

[0011] Specifically, in step S2, constructing the intelligent assessment model for equipment health status includes: S201. Based on the corrected historical operating sequence data of the equipment, the Dynamic Time Warping (DTW) algorithm is used to perform pairwise similarity analysis on the historical operating sequence data of the equipment and calculate the DTW similarity score. This algorithm eliminates the timing deviation caused by operating condition fluctuations or load changes through time axis matching, ensuring the comparability of data under different operating conditions. Based on this, historical time-series data is segmented and normalized according to operating condition labels to obtain a set of historical time-series segments under the same operating condition. These segments are then categorized or clustered based on DTW similarity scores, and time-axis alignment is performed on segments of the same category to form corresponding standard operating template curves. Simultaneously, fault categories, failure modes, and occurrence time windows recorded in maintenance work orders are combined with expert experience constraint rules (including but not limited to key sensor threshold intervals, rate of change constraints, and degradation stage division rules) to label each category of segments with faults, generating typical performance degradation evolution template curves corresponding to each fault type. Further, statistical feature parameters are extracted from the standard operating template curves and typical performance degradation evolution template curves. These statistical feature parameters include at least the mean, variance, confidence interval or threshold interval, and sample size. The template curves, label information, and statistical feature parameters are stored to form an equipment performance degradation benchmark library, serving as a comparison template for subsequent health status assessment and fault prediction. The cumulative distance matrix of the dynamic time warping algorithm is used in this library. D ( i , j Constructed recursively: ; In the formula, x i The standard operating curves are taken from the equipment's historical health status database and have been Z-score standardized. y j To monitor data points in real time, the sampling interval is aligned with historical data; D ( i , j Constrain path continuity during construction to ensure the physical rationality of timing alignment.

[0012] S202. Temporal features are extracted from the sensor data in the established equipment performance degradation benchmark library to construct the equipment state representation vector. Features include mean, variance, kurtosis, and peak factor. Based on this, an initial state transition matrix (equipment state representation vector) and an observation matrix (measured multi-source sensor data) are constructed. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data and update the state representation vector, resulting in the fused equipment state representation vector. The adaptive Kalman filter model adjusts the gain matrix in real time. A weighted fusion of multi-source observations based on their reliability is implemented to improve the accuracy of state estimation; among these measures, sensor noise covariance is considered. The confidence score, calculated based on the sensor's historical error performance, is dynamically updated as follows: ; ; In the formula, Here is the Kalman gain matrix, with dimensions of... , n The number of state variables, m The number of observed variables is used to dynamically adjust the contribution weight of the observed data to the state estimation; To estimate the covariance matrix a priori, reflecting the uncertainty of state prediction, it is recursively updated through the state transition equation; H The observation matrix maps the state space to the observation space, and its elements are determined by the sensor measurement principle. The noise covariance matrix is ​​dynamically observed, and its value is determined by the sensor confidence score. The range is 0 to 1, where 1 indicates complete reliability, achieved through an exponential decay function. The calculation yielded, where γ This is the noise amplitude scaling factor. The lower limit of noise; S203. Equipment Reliability Assessment and Parameter Update: Based on the equipment state representation vector output from S202, a full lifecycle reliability model of the equipment based on the Weibull distribution is established, and the real-time failure probability is calculated. Specifically, based on the equipment state representation vector corresponding to the failure data in the performance degradation benchmark library, the initial shape parameters of the Weibull distribution are calibrated using the maximum likelihood estimation method. β With scale parameters η The real-time failure probability of the equipment is calculated by substituting the equipment's operating time into the Weibull distribution model during the online operation phase. The dynamic update expression for the Weibull distribution parameters is as follows: ; In the formula, β This is a shape parameter, and its value changes to reflect the evolution of the failure mode; η It is a scale parameter, which is positively correlated with the average lifespan of the equipment; t i The fault interval time series extracted from the performance degradation benchmark library; S204, combined with the real-time failure probability F(t) of the device calculated based on the Weibull distribution in S203, and the DTW similarity score of the performance degradation benchmark library constructed in S201 and the statistics of the device's current running sequence data, are used as input features and fed into the unsupervised adaptive clustering algorithm. The clustering algorithm analyzes the multi-dimensional features and constructs a four-level health status assessment system including normal, attention, abnormal, and dangerous, so as to realize intelligent identification of the current health status of the device.

[0013] Preferably, in step S3, equipment fault prediction and intelligent decision-making calculation are performed: S301. Construct a joint CNN-LSTM-Transformer long-range dual-channel time-series prediction model. This model integrates long-period dependencies and cross-dimensional correlation features through a gating fusion mechanism, and uses Bayesian optimization to automatically search for the optimal hyperparameter combination, thereby predicting the evolution trend of device health status over future time periods. The CNN-LSTM-Transformer joint model uses a gating mechanism to integrate multi-dimensional features, and the output weights are calculated as follows: ; In the formula, Indicates time step t The fused output results are used to predict the degradation status of the equipment; The gating weight matrix is ​​used to learn the importance of features through training. Local spatiotemporal features extracted for CNN modules; The long-period dependency features output by the LSTM module; Cross-dimensional relational features generated for the Transformer module; symbols This indicates a vector concatenation operation, which merges three types of features into a 3-feature-dimensional input. Using the Sigmoid activation function, the fusion weights are compressed to the [0, 1] interval; b g This is a bias term.

[0014] S302. Based on the faulty component-fault phenomenon-possible cause triplet extracted in S104, a dynamic operation and maintenance knowledge graph of the apron equipment is constructed using a combined Failure Mode and Effects Analysis (FMEA) strategy. In the dynamic operation and maintenance knowledge graph, the faulty component-fault phenomenon-possible cause structure forms the initial structure of nodes and edges. Potential failure modes, failure consequences, and impact paths for each component are extracted from the FMEA document and used as attribute labels for nodes in the graph. Causal propagation paths between components (e.g., "failure A may lead to failure B") are transformed into directed edges in the graph and assigned propagation probabilities. Furthermore, a graph attention mechanism from graph neural networks is introduced to model the node relationships in the graph, and the graph attention network weights are... Quantify the intensity of fault propagation, nodes i arrive j The attention coefficient is calculated as follows: ; In the formula, Represents a node i With nodes j The original attention scores between them are calculated using a single-layer feedforward neural network; a These are trainable parameter vectors used to learn potential patterns of fault propagation; W For the shared linear transformation weight matrix, the node features h i and h j Mapped to the same feature space; symbol This represents a vector concatenation operation; the LeakyReLU activation function introduces non-linearity and alleviates the gradient vanishing problem; The normalized attention weights are calculated using the softmax function for neighboring nodes. k The scores are normalized to ultimately reflect the fault originating from the node. i propagation to nodes j The probability of; S303. Decision-making is based on the prediction results of fused data and the reasoning results of the knowledge graph. Specifically, the CNN-LSTM-Transformer prediction model outputs the failure probability distribution of the device at future moments. Based on the dynamic knowledge graph constructed in S302, the failure propagation path, associated faulty components, and possible root cause nodes are identified through dynamic knowledge graph retrieval, generating the logical association and confidence level of the device failure. To enhance the robustness and credibility of the decision, the Dempster-Shafer evidence theory is used to fuse the failure probability distribution obtained by the CNN-LSTM-Transformer model with the logical association and confidence level of the failure obtained by dynamic knowledge graph reasoning. Based on the fused diagnostic results, combined with the current spare parts inventory status and available maintenance resources (such as maintenance personnel manpower load, maintenance time limit, etc.), priority ranking and optimal resource allocation suggestions for maintenance personnel are generated as the output of intelligent maintenance decision-making.

[0015] S304. Through a spatiotemporal multi-dimensional display interface, the predicted curve, fault propagation path and graded maintenance suggestions are presented simultaneously, supporting maintenance personnel to quickly locate key issues and evaluate the effectiveness of the handling measures.

[0016] Secondly, embodiments of the present invention provide a multimodal information fusion and knowledge-driven intelligent equipment status management method and system, characterized in that it includes: The data cleaning module includes a sliding window Z-Score detection unit, a wavelet denoising unit, and an expert rule verification unit. The sliding window unit uses an adaptive Z-Score algorithm to identify anomalies such as sensor drift in real time; the multi-scale denoising unit uses a wavelet basis to achieve layered signal denoising; and the expert verification unit performs logical verification based on an industry rule base. This module outputs standardized data with quality labels, providing reliable input for subsequent analysis.

[0017] The condition assessment module integrates a dynamic time warping algorithm unit, a Weibull distribution modeling unit, and a state clustering unit. The dynamic time warping algorithm unit establishes a multi-condition baseline curve library through an improved DTW algorithm; the Weibull distribution modeling unit quantifies equipment degradation patterns using multi-parameter Weibull distributions; and the state clustering unit employs Mahalanobis distance and unsupervised adaptive algorithms to achieve dynamic grading of health status. This module outputs the current health status label of the equipment, improving the speed of equipment maintenance response.

[0018] The predictive maintenance module consists of a hybrid prediction unit and a knowledge reasoning unit. The hybrid prediction unit uses a long-range model combining CNN, LSTM, and Transformer to provide early warnings of equipment failures; the knowledge reasoning unit provides root cause analysis based on a dynamically updated fault knowledge graph.

[0019] The visualization and interaction module takes equipment status assessment results and predictive decision suggestions as input, and synchronously presents equipment health status, fault propagation paths and maintenance strategies through a spatiotemporal multi-dimensional display interface. It supports maintenance personnel to interactively view detailed analysis data and evaluate the effectiveness of handling solutions.

[0020] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described multimodal information fusion and knowledge-driven intelligent device state management method.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described multimodal information fusion and knowledge-driven intelligent device state management method.

[0022] Fifthly, embodiments of the present invention provide a multimodal information fusion and knowledge-driven intelligent management method and system for device status, characterized by comprising: At least one of the following application scenarios: berth guidance system, ground power supply and air conditioning system, water supply and drainage pipelines, etc.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: This paper proposes a multimodal information fusion and knowledge-driven equipment health status management framework. An intelligent data cleaning system is constructed through dynamic threshold detection and multi-scale noise reduction techniques, enabling efficient processing of multi-source heterogeneous data. A combined approach of time series matching and state-space modeling is used to establish an equipment health assessment model adaptable to various operating conditions, improving the accuracy of status identification. By integrating time series prediction and knowledge graph reasoning techniques, a data- and knowledge-driven intelligent decision-making mechanism is built, enhancing the interpretability of decisions while ensuring prediction accuracy. Through automated knowledge extraction and dynamic update strategies, the system possesses continuous evolution capabilities, enabling rapid adaptation to changes in new equipment and operational scenarios, providing reliable technical support for the full lifecycle management of critical airport equipment.

[0024] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the present invention; Figure 2 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 3 This is a block diagram of a chip according to an embodiment of the present invention; Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0031] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0032] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0033] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0034] This invention provides a method and system for equipment status management based on multimodal information fusion and knowledge-driven approaches. The method includes: using a sliding window Z-Score algorithm combined with wavelet transform to achieve dynamic threshold anomaly detection and noise filtering of multi-source heterogeneous data; extracting fault triplet knowledge from maintenance work orders using a BERT-BiLSTM model to construct a cleaning system with expert rule constraints; establishing a multi-condition performance degradation benchmark library using a dynamic time warping algorithm; constructing a state-space model by fusing multi-source sensor data through adaptive Kalman filtering; fitting fault evolution curves based on Weibull distribution and combining with an unsupervised adaptive clustering algorithm to achieve four-level health status classification; constructing a CNN-LSTM-Transformer joint long-range time-series prediction model and a dynamic knowledge graph; and achieving intelligent decision-making by fusing data prediction and graph reasoning through DS evidence theory. This invention innovatively integrates multimodal sensor data and maintenance knowledge graphs, solving the problem of accurate equipment status assessment and fault prediction under complex operating conditions, significantly improving the intelligence level of apron equipment management, and can be widely applied to the health management of key equipment such as berth guidance systems, ground power supply systems, air conditioning units, and water supply and drainage pipelines.

[0035] Please see Figure 1The present invention relates to a multimodal information fusion and knowledge-driven equipment status management method and system, comprising the following steps: S1. Multi-source heterogeneous data anomaly cleaning: Multi-source heterogeneous sensor data is collected in real time through the sensor array of the apron equipment. This multi-source heterogeneous sensor data is historical runtime sequence data. A sliding window algorithm is used for dynamic threshold anomaly detection, combined with wavelet transform for multi-scale noise filtering. The fault knowledge in the maintenance work order is used to classify, statistically analyze and quantify the data after multi-scale noise filtering. Based on the statistical characteristics of the data, constraint rules and anomaly thresholds based on expert experience are formulated to identify and correct the abnormal data, resulting in corrected historical runtime sequence data of the equipment. S101. Real-time monitoring data is collected through sensor arrays deployed on key apron equipment, including voltage and current signals of the berth guidance system, harmonic distortion rate of the ground power supply system, refrigerant pressure and temperature of the air conditioning unit, and flow and pressure data of water supply and drainage pipelines. S102. An adaptive threshold detection model is constructed using the sliding window Z-Score algorithm to perform anomaly detection on the collected time-series data, thus performing initial screening of abnormal data. Anomaly detection can identify sensor drift, communication interruption, and other transient interference or abrupt changes that may affect data quality, providing a preliminary screening basis for subsequent wavelet transform noise reduction processing. The dynamic decision boundary in the adaptive threshold detection model is determined by the following formula. ; In the formula, To dynamically determine the boundary; and These are the mean and standard deviation of the data within the current sliding window, respectively. The window length is set according to the device's sampling frequency. k This is the sensitivity adjustment coefficient; α The threshold change rate during operating condition switching; This is the timestamp of the last operating condition switch. t This is the current sampling time point; Based on this, the identification of typical abnormal data such as sensor drift and communication interruption is achieved through joint analysis of sliding standard deviation and kurtosis coefficient; finally, a baseline library of apron equipment operating status is constructed based on the cleaned data to provide reliable data support for subsequent equipment health assessment.

[0036] S103. After anomaly detection, the data undergoes multi-scale noise filtering using wavelet transform; wavelet basis functions are used to decompose the signal, retaining low-frequency useful components. To address the issue of high-frequency noise and effective signal aliasing in airport equipment, a multi-scale thresholding strategy is employed, with thresholds set for each decomposition layer. T j The calculation formula is as follows: ; In the formula, σ j For the first j The standard deviation of the layer wavelet decomposition coefficients is calculated using the db4 wavelet basis function. N j This represents the number of effective coefficients at the current scale, and its value varies with the number of decomposition levels. j Increases and decreases; SNR j The first obtained through the noise power estimation module j Layer signal-to-noise ratio.

[0037] S104. For unstructured maintenance work order texts, the BERT-BiLSTM model is used for entity recognition to extract key fault description triples (faulty component, fault phenomenon, possible cause). The BERT-BiLSTM-based entity recognition model determines fault elements through joint probability output. t The predicted probability of the label for each word is: ; In the formula, W s This represents the weight matrix of the fully connected layer; The context-dependent feature vectors output by the BERT model; The sequence feature vectors extracted by the BiLSTM layer are used to capture the temporal dependencies in fault description; This represents a vector concatenation operation that merges two types of features into a hybrid feature. ；b s The bias term is used to adjust the classification boundary; the Softmax function normalizes the linear transformation result into a probability distribution, and finally outputs the first... t The probability of each word belonging to each entity label; the continuous faulty parts, fault phenomena and possible causes in the text are identified through the BIO annotation system, and a faulty part-fault phenomenon-possible cause triplet is constructed based on word position and semantic dependency relationship; S105. Based on the extracted faulty component-fault phenomenon-possible cause triplet (as a classification label), the data after multi-scale noise filtering is classified, statistically analyzed, and quantified. The statistical characteristics of data corresponding to the same type of faulty component, fault phenomenon, or cause are calculated respectively. Furthermore, based on the statistical characteristics, constraint rules and anomaly thresholds based on expert experience are formulated to identify and correct abnormal data, resulting in corrected historical operating sequence data of the equipment. The time-series data involved includes, but is not limited to: voltage and current signals of the berth guidance system, harmonic distortion rate of the ground power supply system, refrigerant pressure and temperature of the air conditioning unit, and flow and pressure data of the water supply and drainage pipelines.

[0038] S2. Intelligent Equipment Health Status Assessment: Based on the corrected historical operating sequence data of the equipment, a performance degradation benchmark database containing multi-condition features is constructed using a dynamic time warping algorithm and expert experience. On this basis, a device state representation vector is constructed by extracting time-domain features from the benchmark database. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data to achieve equipment reliability assessment and parameter updates. Furthermore, a full lifecycle reliability model of the equipment based on the Weibull distribution is established using this state representation vector, and the Weibull distribution parameters are dynamically updated using online equipment data to calculate the cumulative failure probability of the equipment in real time. Finally, the current operating sequence data, the performance degradation benchmark database, and the real-time failure probability are integrated, and an unsupervised adaptive clustering algorithm is used to construct an equipment health status assessment system, enabling intelligent identification and hierarchical early warning of health status.

[0039] Based on the corrected historical operating sequence data of the equipment, S201 employs a dynamic time warping algorithm to perform pairwise similarity analysis on the historical operating sequence data, calculates the DTW similarity score, and matches and aligns data of the same type of fault along the time axis. On this basis, a benchmark library of equipment performance degradation, containing time-series characteristic data curves of standard operating modes and typical fault evolution paths, is constructed in conjunction with expert experience and knowledge, serving as a comparison template for subsequent health status assessment and fault prediction. The cumulative distance matrix D(i,j) of the dynamic time warping algorithm is constructed recursively. ; In the formula, x i The standard operating curves for the equipment's historical health status database have been standardized using Z-score. y j To monitor data points in real time, the sampling interval is aligned with historical data; D ( i , j Constrain path continuity during construction to ensure the physical rationality of timing alignment; S202. Temporal features are extracted from the sensor data in the established equipment performance degradation benchmark library to construct the equipment state representation vector. Features include mean, variance, kurtosis, and peak factor. Based on this, an initial state transition matrix (equipment state representation vector) and an observation matrix (measured multi-source sensor data) are constructed. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data and update the state representation vector, resulting in the fused equipment state representation vector. The adaptive Kalman filter model adjusts the gain matrix in real time. A weighted fusion of multi-source observations based on their reliability is implemented to improve the accuracy of state estimation; among these measures, sensor noise covariance is considered. The confidence score, calculated based on the sensor's historical error performance, is dynamically updated as follows: ; ; In the formula, Here is the Kalman gain matrix, with dimensions of... , n The number of state variables, m The number of observed variables is used to dynamically adjust the contribution weight of the observed data to the state estimation; To estimate the covariance matrix a priori, reflecting the uncertainty of state prediction, it is recursively updated through the state transition equation; H The observation matrix maps the state space to the observation space, and its elements are determined by the sensor measurement principle. The noise covariance matrix is ​​dynamically observed, and its value is determined by the sensor confidence score. The range is 0 to 1, where 1 indicates complete reliability, achieved through an exponential decay function. The calculation yielded, where γ This is the noise amplitude scaling factor. The lower limit of noise; S203. Equipment Reliability Assessment and Parameter Update: Based on the equipment state representation vector output from S202, a full lifecycle reliability model of the equipment based on the Weibull distribution is established, and the real-time failure probability is calculated. Specifically, based on the equipment state representation vector corresponding to the failure data in the performance degradation benchmark library, the initial shape parameters of the Weibull distribution are calibrated using the maximum likelihood estimation method. β With scale parameters η The real-time failure probability of the equipment is calculated by substituting the equipment's operating time into the Weibull distribution model during the online operation phase. The dynamic update expression for the Weibull distribution parameters is as follows: ; In the formula, β This is a shape parameter, and its value changes to reflect the evolution of the failure mode; η It is a scale parameter, which is positively correlated with the average lifespan of the equipment; t i The fault interval time series extracted from the performance degradation benchmark library; S204, combined with the real-time failure probability F(t) of the device calculated based on the Weibull distribution in S203, and the DTW similarity score of the performance degradation benchmark library constructed in S201 and the statistics of the device's current running sequence data, are used as input features and fed into the unsupervised adaptive clustering algorithm. The clustering algorithm analyzes the multi-dimensional features and constructs a four-level health status assessment system including normal, attention, abnormal, and dangerous, so as to realize intelligent identification of the current health status of the device.

[0040] S3. Equipment Fault Prediction and Intelligent Decision-Making: Based on the equipment state representation vector obtained in S2, a CNN-LSTM-Transformer joint prediction model is constructed. The fault probability distribution is predicted through gating fusion and Bayesian optimization. In addition, a dynamic operation and maintenance knowledge graph is constructed based on the fault triplet joint failure mode and impact analysis strategy calculated in S1. The dynamic knowledge graph is used to generate equipment fault logical associations and confidence levels. Based on the above two steps, the fault probability distribution obtained from the prediction model and the fault logical associations and confidence levels obtained from the knowledge graph logical reasoning are fused through DS evidence theory to obtain the decision results.

[0041] S301. Based on the device state representation vector obtained in S202, a CNN-LSTM-Transformer joint model is constructed. This model integrates long-period dependencies and cross-dimensional correlation features through a gating fusion mechanism, and uses Bayesian optimization to automatically search for the optimal hyperparameter combination, thereby predicting the evolution trend of device health status over future time periods. The CNN-LSTM-Transformer joint model uses a gating mechanism to integrate multi-dimensional features, and the output weights are calculated as follows: .

[0042] In the formula, Indicates time step t The fused output results are used to predict the degradation status of the equipment; The gating weight matrix is ​​used to learn the importance of features through training. Local spatiotemporal features extracted for CNN modules; The long-period dependency features output by the LSTM module; Cross-dimensional relational features generated for the Transformer module; symbols This indicates a vector concatenation operation, which merges three types of features into a 3-feature-dimensional input. Using the Sigmoid activation function, the fusion weights are compressed to the [0, 1] interval; b g For bias terms; S302. Based on the faulty component-fault phenomenon-possible cause triplet extracted in S104, a dynamic operation and maintenance knowledge graph for apron equipment is constructed using a joint failure mode and impact analysis (FMEA) strategy. In the dynamic FMEA, the faulty component-fault phenomenon-possible cause structure forms the initial structure of nodes and edges. Potential failure modes, failure consequences, and impact paths for each component are extracted from the FMEA document and used as attribute labels for nodes in the graph. Causal propagation paths between components are transformed into directed edges in the graph and assigned propagation probabilities. Furthermore, a graph attention mechanism from graph neural networks is introduced to model the node relationships in the graph, and the graph attention network weights are... Quantify the intensity of fault propagation, nodes i arrive j The attention coefficient is calculated as follows: ; In the formula, Represents a node i With nodes j The original attention scores between them are calculated using a single-layer feedforward neural network; a These are trainable parameter vectors used to learn potential patterns of fault propagation; W For the shared linear transformation weight matrix, the node features h i and h j Mapped to the same feature space; symbol This represents a vector concatenation operation; the LeakyReLU activation function introduces non-linearity and alleviates the gradient vanishing problem; The normalized attention weights are calculated using the softmax function for neighboring nodes. k The scores are normalized to ultimately reflect the fault originating from the node. i propagation to nodes j The probability of; S303. Decision-making is made based on the prediction results of fused data and the reasoning results of the knowledge graph. Specifically, the CNN-LSTM-Transformer prediction model outputs the failure probability distribution of the device at future moments. Based on the dynamic knowledge graph constructed in S302, the fault propagation path, associated faulty components, and possible root cause nodes are identified through dynamic knowledge graph retrieval, generating device fault logical associations and confidence levels. The device fault logical associations consist of a relationship chain of root cause nodes—intermediate propagation nodes—faulty component nodes. This relationship chain is obtained through multi-hop path retrieval of the dynamic knowledge graph and filtered based on preset rule reasoning or graph embedding link scoring. The confidence level is determined by at least one or more factors, including: relation edge weights, path length penalties, relation type reliability, and the degree of matching with online observation evidence. Preferably, the confidence level can be expressed as the product or weighted sum of the confidence levels of each edge, and a path length attenuation coefficient is introduced to obtain the path confidence level of each candidate fault propagation path. Several paths with the highest path confidence levels are selected as the fault logical associations and confidence level outputs. The Dempster-Shafer evidence theory is used to fuse the fault probability distribution obtained by the CNN-LSTM-Transformer model with the fault logical association and confidence obtained by dynamic knowledge graph reasoning. Based on the fused diagnostic results, combined with the current spare parts inventory status and available maintenance resources of the system, priority ranking and optimal resource allocation suggestions for maintenance personnel are generated as the output of intelligent maintenance decision-making.

[0043] In another embodiment of the present invention, a multimodal information fusion and knowledge-driven equipment status management system is provided. This system can be used to implement the above-mentioned intelligent equipment status management method. Specifically, the system includes a data cleaning module, a status assessment module, a predictive maintenance module, and a visualization interaction module.

[0044] The data cleaning module takes multi-source sensor data such as voltage, current, harmonic distortion rate, and refrigerant parameters of key apron equipment as input, and uses the sliding window Z-Score algorithm and wavelet transform to construct an adaptive anomaly detection and noise reduction model to generate a standardized monitoring dataset with quality labels. The status assessment module, based on the cleaned and standardized monitoring data, designs an equipment health assessment framework that combines dynamic time warping algorithm with Weibull distribution modeling. It integrates multi-dimensional features through unsupervised adaptive clustering algorithm and outputs a four-level health status label including normal, attention, abnormal and dangerous. The predictive maintenance module takes historical operating data and real-time monitoring data of the equipment as input, and uses a long-range hybrid model combining CNN-LSTM-Transformer to predict the degradation trend of the equipment. At the same time, it combines dynamic knowledge graph to perform root cause reasoning of failures, and uses DS evidence theory to fuse the prediction results and knowledge reasoning to generate maintenance plans. The visualization and interaction module takes equipment status assessment results and predictive decision suggestions as input, and synchronously presents equipment health status, fault propagation paths and maintenance strategies through a spatiotemporal multi-dimensional display interface. It supports maintenance personnel to interactively view detailed analysis data and evaluate the effectiveness of handling solutions.

[0045] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions. The processor described in this embodiment can be used in the operation of a multimodal information fusion and knowledge-driven intelligent device status management method, including: Dynamic threshold anomaly detection and noise filtering of multi-source heterogeneous data are achieved by combining the sliding window Z-Score algorithm with wavelet transform. A cleaning system with expert rule constraints is constructed by extracting fault triple knowledge from maintenance work orders using the BERT-BiLSTM model. A multi-condition performance degradation benchmark library is established using the dynamic time warping algorithm. A state-space model is constructed by fusing multi-source sensor data through adaptive Kalman filtering. Fault evolution curves are fitted based on Weibull distribution and combined with unsupervised adaptive clustering algorithm to achieve four-level health state classification. A long-range time series prediction model and dynamic knowledge graph jointly constructed by CNN-LSTM-Transformer are built. Intelligent decision-making is achieved by fusing data prediction and graph reasoning through DS evidence theory.

[0046] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0047] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the multimodal information fusion and knowledge-driven intelligent device state management method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: Dynamic threshold anomaly detection and noise filtering of multi-source heterogeneous data are achieved by combining the sliding window Z-Score algorithm with wavelet transform. A cleaning system with expert rule constraints is constructed by extracting fault triple knowledge from maintenance work orders using the BERT-BiLSTM model. A multi-condition performance degradation benchmark library is established using the dynamic time warping algorithm. A state-space model is constructed by fusing multi-source sensor data through adaptive Kalman filtering. Fault evolution curves are fitted based on Weibull distribution and combined with unsupervised adaptive clustering algorithm to achieve four-level health state classification. A long-range time series prediction model and dynamic knowledge graph jointly constructed by CNN-LSTM-Transformer are built. Intelligent decision-making is achieved by fusing data prediction and graph reasoning through DS evidence theory.

[0048] Please see Figure 2 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the fluid composition calculation system in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here.

[0049] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 2 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0050] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, CPUs, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, quantum computing-based data processing logic units, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0051] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0052] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0053] Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0054] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0055] Please see Figure 3 The terminal device is a chip. In this embodiment, the chip 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the aforementioned generalizable unsupervised target segmentation method for dynamic apron scenes.

[0056] Additionally, chip 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of chip 600, and the communication component 650 can be configured to enable communication of chip 600, such as wired or wireless communication. Furthermore, chip 600 may also include an input / output interface 658. Chip 600 can operate on an operating system stored in memory 632.

[0057] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and illustrated in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0058] In summary, the multimodal information fusion and knowledge-driven equipment status management method and system of the present invention have the following characteristics: (1) By integrating multi-source sensor data and knowledge in the field of operation and maintenance, this invention has achieved accurate characterization of equipment health status under complex working conditions by constructing a dynamic threshold detection system and an adaptive state space model, which significantly improves the monitoring accuracy and assessment reliability of key apron equipment in variable operating environments.

[0059] (2) This invention proposes a two-way decision-making framework of “data-driven-knowledge-guided”. First, it predicts the degradation trend of equipment through a long-range model jointly composed of CNN-LSTM-Transformer. Then, it combines dynamic knowledge graphs to perform root cause reasoning of failures. It uses DS evidence theory to fuse multi-source information, which enhances the interpretability of decision-making while ensuring prediction accuracy.

[0060] (3) The present invention designs a complete equipment status management process, including modules such as multimodal data cleaning, multi-condition health assessment, and intelligent prediction and decision-making, which greatly reduces the reliance of traditional operation and maintenance on human experience.

[0061] (4) The dynamic update mechanism and adaptive learning strategy adopted in this invention enable the system to have the ability to continuously iterate and update, and can quickly adapt to new equipment and fault modes, thus solving the performance degradation problem caused by model solidification in traditional methods.

[0062] (5) Through multi-feature fusion algorithm and four-level state division mechanism, this invention effectively improves the state recognition accuracy of specific equipment (such as berth guidance system, ground power supply equipment, etc.) while maintaining the system's versatility, providing reliable technical support for intelligent operation and maintenance of airport equipment.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0065] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0066] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0069] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0070] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A multimodal information fusion and knowledge-driven equipment status management method, characterized in that, Includes the following steps: S1. Multi-source heterogeneous data anomaly cleaning: Multi-source heterogeneous sensor data is collected in real time through the sensor array of the apron equipment. This multi-source heterogeneous sensor data is historical runtime sequence data. A sliding window algorithm is used for dynamic threshold anomaly detection, combined with wavelet transform for multi-scale noise filtering. The fault knowledge in the maintenance work order is used to classify, statistically analyze and quantify the data after multi-scale noise filtering. Based on the statistical characteristics of the data, constraint rules and anomaly thresholds based on expert experience are formulated to identify and correct the abnormal data, resulting in corrected historical runtime sequence data of the equipment. S2. Intelligent Equipment Health Status Assessment: Based on the corrected historical operating sequence data of the equipment, a performance degradation benchmark database containing multi-condition features is constructed using a dynamic time warping algorithm and expert experience. On this basis, a device state representation vector is constructed by extracting time-domain features from the benchmark database. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data to achieve equipment reliability assessment and parameter updates. Furthermore, a Weibull distribution-based full lifecycle reliability model is established based on this state representation vector, and the Weibull distribution parameters are dynamically updated using online equipment data to calculate the cumulative failure probability of the equipment in real time. Finally, by fusing the current operating sequence data, the performance degradation benchmark database, and the real-time failure probability, an unsupervised adaptive clustering algorithm is used to construct an equipment health status assessment system, enabling intelligent identification and hierarchical early warning of health status. S3. Equipment Fault Prediction and Intelligent Decision-Making: Based on the equipment state representation vector obtained in S2, a CNN-LSTM-Transformer joint prediction model is constructed, and the fault probability distribution is predicted through gating fusion and Bayesian optimization. Furthermore, a dynamic operation and maintenance knowledge graph is constructed based on the joint failure mode and impact analysis strategy of the fault triplet calculated by S1. The dynamic knowledge graph is used to generate equipment fault logical associations and confidence levels. Based on the above two steps, the fault probability distribution obtained by the prediction model and the fault logical associations and confidence levels obtained by logical reasoning from the knowledge graph are fused through DS evidence theory to obtain the decision results.

2. The multimodal information fusion and knowledge-driven equipment status management method according to claim 1, characterized in that, In step S1, the multi-source heterogeneous data anomaly cleaning includes: S101. Collect time-series data through sensor arrays deployed on key apron equipment, including but not limited to voltage and current signals of berth guidance system, harmonic distortion rate of ground power supply system, refrigerant pressure and temperature of air conditioning unit and flow and pressure data of water supply and drainage pipelines. S102. An adaptive threshold detection model is constructed using the sliding window Z-Score algorithm to perform anomaly detection on the collected time-series data and conduct initial screening of anomaly data. The dynamic decision boundary in the adaptive threshold detection model is determined by the following formula. In the formula, To dynamically determine the boundary; and These are the mean and standard deviation of the data within the current sliding window, respectively. The window length is set according to the device's sampling frequency. k This is the sensitivity adjustment coefficient; α The threshold change rate during operating condition switching; This is the timestamp of the last operating condition switch. t This is the current sampling time point; S103. After anomaly detection, the data is subjected to multi-scale noise filtering using wavelet transform; the signal is decomposed using wavelet basis functions to retain useful low-frequency components, and a multi-scale thresholding strategy is adopted, with thresholds for each decomposition layer. T j The calculation formula is as follows: In the formula, σ j For the first j The standard deviation of the layer wavelet decomposition coefficients is calculated using the db4 wavelet basis function. N j This represents the number of effective coefficients at the current scale, and its value varies with the number of decomposition levels. j Increases and decreases; SNR j The first obtained through the noise power estimation module j Layer signal-to-noise ratio; S104. For unstructured maintenance work orders, the BERT-BiLSTM model is used for entity recognition to extract key fault description triples, specifically the faulty component-fault phenomenon-possible cause triples; specifically, based on the BERT-BiLSTM model, fault elements are determined through joint probability output, the first... t The predicted label probability for each word is: In the formula, W s This is the weight matrix of the fully connected layer; The context-dependent feature vectors output by the BERT model; The sequence feature vectors extracted by the BiLSTM layer are used to capture the temporal dependencies in fault description; This is a vector concatenation operation that merges two types of features into a hybrid feature. b s The bias term is used to adjust the classification boundary; the Softmax function normalizes the linear transformation result into a probability distribution, and finally outputs the first... t The probability of each word belonging to each entity label; the continuous faulty parts, fault phenomena and possible causes in the text are identified through the BIO annotation system, and a faulty part-fault phenomenon-possible cause triplet is constructed based on word position and semantic dependency relationship; S105. Based on the extracted faulty component-fault phenomenon-possible cause triplet, the data after multi-scale noise filtering is classified, statistically analyzed and quantified. The statistical characteristics of the data corresponding to the same type of faulty component, the same type of fault phenomenon or the same type of cause are calculated respectively. Furthermore, based on the statistical characteristics of the data, constraint rules and abnormal thresholds based on expert experience are formulated to identify and correct abnormal data, and the corrected historical operation sequence data of the equipment is obtained.

3. The multimodal information fusion and knowledge-driven equipment status management method according to claim 1, characterized in that, In step S2, the intelligent assessment of the device's health status specifically includes: S201. Based on the corrected historical operating sequence data of the equipment, a dynamic time warping algorithm is used to perform pairwise similarity analysis on the historical operating sequence data of the equipment, calculate the DTW similarity score, and match and align the data of the same type of fault on the time axis. On this basis, a benchmark library of equipment performance degradation containing time sequence characteristic data curves of standard operating modes and typical fault evolution paths is constructed in conjunction with expert experience and knowledge, as a comparison template for subsequent health status assessment and fault prediction. The cumulative distance matrix D(i,j) of the dynamic time warping algorithm is constructed in a recursive manner: In the formula, x i The standard operating curves for the equipment's historical health status database have been standardized using Z-score. y j To monitor data points in real time, the sampling interval is aligned with historical data; D ( i , j Constrain path continuity during construction to ensure the physical rationality of timing alignment; S202. Temporal features are extracted from the sensor data in the established equipment performance degradation benchmark library to construct the equipment state representation vector. Features include mean, variance, kurtosis, and peak factor. Based on this, an initial state transition matrix (equipment state representation vector) and an observation matrix (measured multi-source sensor data) are constructed. An adaptive Kalman filter model is then used to dynamically weight and fuse the measured multi-source sensor data and update the state representation vector, resulting in the fused equipment state representation vector. The adaptive Kalman filter model adjusts the gain matrix in real time. A weighted fusion of multi-source observations based on their reliability is implemented to improve the accuracy of state estimation; among these measures, sensor noise covariance is considered. The confidence score, calculated based on the sensor's historical error performance, is dynamically updated as follows: In the formula, Here is the Kalman gain matrix, with dimensions of . , n The number of state variables, m The number of observed variables is used to dynamically adjust the contribution weight of the observed data to the state estimation; To estimate the covariance matrix a priori, reflecting the uncertainty of state prediction, it is recursively updated through the state transition equation; H The observation matrix maps the state space to the observation space, and its elements are determined by the sensor measurement principle. The noise covariance matrix is ​​dynamically observed, and its value is determined by the sensor confidence score. The range is 0 to 1, where 1 indicates complete reliability, achieved through an exponential decay function. The calculation yielded, where γ This is the noise amplitude scaling factor. The lower limit of noise; S203. Equipment Reliability Assessment and Parameter Update: Based on the equipment state representation vector output from S202, a full lifecycle reliability model of the equipment based on the Weibull distribution is established, and the real-time failure probability is calculated. Specifically, based on the equipment state representation vector corresponding to the failure data in the performance degradation benchmark library, the initial shape parameters of the Weibull distribution are calibrated using the maximum likelihood estimation method. β With scale parameters η The real-time failure probability of the equipment is calculated by substituting the equipment's operating time into the Weibull distribution model during the online operation phase. The dynamic update expression for the Weibull distribution parameters is as follows: In the formula, β This is a shape parameter, and its value changes to reflect the evolution of the failure mode; η It is a scale parameter, which is positively correlated with the average lifespan of the equipment; t i The fault interval time series extracted from the performance degradation benchmark library; S204, combined with the real-time failure probability F(t) of the device calculated based on the Weibull distribution in S203, and the DTW similarity score of the performance degradation benchmark library constructed in S201 and the statistics of the device's current running sequence data, are used as input features and fed into the unsupervised adaptive clustering algorithm. The clustering algorithm analyzes the multi-dimensional features and constructs a four-level health status assessment system including normal, attention, abnormal, and dangerous, so as to realize intelligent identification of the current health status of the device.

4. The multimodal information fusion and knowledge-driven equipment status management method according to claim 1, characterized in that, In step S3, the equipment fault prediction and intelligent decision-making include: S301. Based on the device state representation vector obtained in S202, a CNN-LSTM-Transformer joint model is constructed. This model integrates long-period dependencies and cross-dimensional correlation features through a gating fusion mechanism, and uses Bayesian optimization to automatically search for the optimal hyperparameter combination, thereby predicting the evolution trend of device health status over future time periods. The CNN-LSTM-Transformer joint model uses a gating mechanism to integrate multi-dimensional features, and the output weights are calculated as follows: In the formula, Indicates time step t The fused output results are used to predict the degradation status of the equipment; The gating weight matrix is ​​used to learn the importance of features through training. Local spatiotemporal features extracted for CNN modules; The long-period dependency features output by the LSTM module; Cross-dimensional relational features generated for the Transformer module; symbols This indicates a vector concatenation operation, which merges three types of features into a 3-feature-dimensional input. Using the Sigmoid activation function, the fusion weights are compressed to the [0, 1] interval; b g For bias terms; S302. Based on the faulty component-fault phenomenon-possible cause triplet extracted in S104, and combined with the Failure Mode and Effects Analysis (FMEA) strategy document, a dynamic operation and maintenance knowledge graph for the apron equipment is constructed. In the dynamic operation and maintenance knowledge graph, the faulty component-fault phenomenon-possible cause structure forms the initial structure of nodes and edges. The potential failure modes, failure consequences, and impact paths of each component are extracted using the FMEA document and used as attribute labels for nodes in the graph. The causal propagation paths between components are transformed into directed edges in the graph and assigned propagation probabilities. Furthermore, the graph attention mechanism from graph neural networks is introduced to model the node relationships in the graph, and the graph attention network weights are... Quantify the intensity of fault propagation, nodes i arrive j The attention coefficient is calculated as follows: In the formula, Represents a node i With nodes j The original attention scores between the two are calculated using a single-layer feedforward neural network; a These are trainable parameter vectors used to learn potential patterns of fault propagation; W For the shared linear transformation weight matrix, the node features h i and h j Mapped to the same feature space; symbol This represents a vector concatenation operation; the LeakyReLU activation function introduces non-linearity and alleviates the gradient vanishing problem; The normalized attention weights are calculated using the softmax function on neighboring nodes. k The scores are normalized to ultimately reflect the fault originating from the node. i propagation to nodes j The probability of; S303. Decision-making based on fused data prediction results and knowledge graph reasoning results; specifically, the CNN-LSTM-Transformer prediction model outputs the failure probability distribution of the device at future moments; based on the dynamic knowledge graph constructed in S302, the fault propagation path, associated faulty components, and possible root cause nodes are identified through dynamic knowledge graph retrieval, generating device fault logical associations and confidence levels; the Dempster-Shafer evidence theory is used to fuse the fault probability distribution obtained by the CNN-LSTM-Transformer model with the fault logical associations and confidence levels obtained by dynamic knowledge graph reasoning; based on the fused diagnostic results, combined with the current spare parts inventory status and available maintenance resources of the system, priority ranking and optimal resource allocation suggestions for maintenance personnel are generated as the output of intelligent maintenance decision-making.

5. A method and system for intelligent management of equipment status, characterized in that, include: The data cleaning module includes a sliding window Z-Score detection unit, a wavelet denoising unit, and an expert rule verification unit. The sliding window unit uses an adaptive Z-Score algorithm to identify anomalies such as sensor drift in real time. The multi-scale denoising unit uses a wavelet basis to achieve layered signal denoising. The expert verification unit performs logical verification based on an industry rule base. This module outputs standardized data with quality labels, providing reliable input for subsequent analysis. The condition assessment module integrates a dynamic time warping algorithm unit, a Weibull distribution modeling unit, and a state clustering unit. The dynamic time warping algorithm unit establishes a multi-condition benchmark curve library by improving the DTW algorithm. The Weibull distribution modeling unit uses multi-parameter Weibull distribution to quantify the equipment degradation law. The state clustering unit uses Mahalanobis distance unsupervised adaptive clustering algorithm to realize dynamic classification of health status. This module outputs the current health status label of the equipment, improving the equipment maintenance response speed. The predictive maintenance module consists of a hybrid prediction unit and a knowledge reasoning unit; the hybrid prediction unit uses a long-range model jointly developed by CNN-LSTM-Transformer to provide early warning of equipment failures. The knowledge reasoning unit provides root cause analysis based on a dynamically updated fault knowledge graph.

Images