Factory equipment operation and maintenance decision-making system based on multi-modal data fusion and hybrid reasoning

Abstract

The invention relates to a factory equipment operation and maintenance decision-making system based on multi-modal data fusion and hybrid reasoning, which comprises a multi-modal data fusion framework and a hybrid reasoning model, and is characterized in that the multi-modal data fusion framework is used for integrating multi-modal data and dynamically weighting fusion features through a cross-modal attention mechanism; the hybrid reasoning model comprises a rule reasoning module, a probabilistic reasoning module and an AI enhancement module, the rule reasoning module generates a generative rule based on an expert knowledge base, the probabilistic reasoning module processes uncertainty data by adopting a dynamic Bayesian modeling equipment state transition probability, and the AI enhancement module introduces an artificial neural network to optimize rule reasoning parameters; and different factory affair scenes are adapted in combination with transfer learning. The method has the advantages that the method is suitable for power stations, power distribution rooms, fans and other scenes, and fault diagnosis, service life prediction and dynamic maintenance strategy generation are combined.

Description

Technical Field

[0001] The present invention relates to a factory equipment operation and maintenance decision-making system and method, specifically to a factory equipment operation and maintenance decision-making system based on multimodal data fusion and hybrid reasoning, belonging to the field of equipment operation and maintenance management in the electronics industry. Background Art

[0002] In the existing technology, the operation and maintenance management of equipment in the electronics industry generally includes the following forms:

[0003] (1) Human experience-driven: Relying on expert rules to judge faults, it has problems such as strong subjectivity and delayed updates.

[0004] (2) Single-modal data analysis: Using sensor time series data or image data to make fault judgments fails to effectively integrate multi-source information, resulting in one-sided decision-making basis.

[0005] (3) Static knowledge base: Updates rely on manual input and cannot adapt to dynamic scenarios such as device aging and environmental changes.

[0006] (4) Pure AI model: This model uses artificial intelligence solutions such as deep learning to perform fault diagnosis, which lacks transparency and is difficult to meet the industry's strict requirements for safety and reliability.

[0007] The defects of the above forms of existing technology mainly include:

[0008] (1) Reliance on manual experience: Manual regular inspections are inefficient and prone to missed inspections; experience-driven decision-making relies on the experience of veterans, and personnel turnover leads to knowledge gaps.

[0009] (2) Data silos: Multi-source data is scattered, and multimodal data such as sensor data, infrared images, and operation and maintenance logs are scattered in different systems and lack unified integration.

[0010] (3) Delayed response to faults: Action is taken after equipment failure, resulting in long downtime and the inability to predict the remaining life of the equipment or potential failures, resulting in unplanned downtime.

[0011] (4) Insufficient intelligence: The expert rule base relies on manual updates and is difficult to adapt to new equipment or complex failure modes. In addition, the maintenance plan is fixed and fails to consider the actual status of the equipment, which can easily lead to over-maintenance or under-maintenance.

[0012] (5) Low security and reliability: Manual recording or operation is prone to errors, and there is a lack of real-time monitoring, making it impossible to perceive changes in device status. Summary of the Invention

[0013] The present invention proposes a plant equipment operation and maintenance decision-making system based on multimodal data fusion and hybrid reasoning. Its purpose is to overcome the above-mentioned shortcomings of the existing technology and provide a high-precision, explainable and adaptive power transmission and transformation equipment decision-making system. Through multimodal data fusion and hybrid reasoning models, it realizes the full process automation of fault diagnosis, life prediction and dynamic maintenance strategy generation.

[0014] The technical solution of the present invention is: a factory equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning, including a multimodal data fusion architecture and a hybrid reasoning model, wherein the multimodal data fusion architecture is used to integrate multimodal data and dynamically weighted fusion features through a cross-modal attention mechanism; the hybrid reasoning model includes a rule reasoning module, a probabilistic reasoning module and an AI enhancement module, the rule reasoning module generates production rules based on an expert knowledge base, the probabilistic reasoning module uses dynamic Bayesian modeling of equipment state transition probabilities to process uncertain data, and the AI enhancement module introduces an artificial neural network to optimize rule reasoning parameters, combined with transfer learning to adapt to different factory scenarios.

[0015] Preferably, the multimodal data includes sensor signals, infrared images, vibration signals, and operation and maintenance logs.

[0016] Preferably, the processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning includes the following steps:

[0017] 1) The data acquisition layer collects multimodal data;

[0018] 2) Perform data preprocessing on multimodal data;

[0019] 3) Perform feature extraction and fusion on the preprocessed data to obtain multimodal data features;

[0020] 4) Using multimodal data features to perform hybrid reasoning operations by the hybrid reasoning model;

[0021] 5) Decision execution, including work order generation and knowledge base update.

[0022] Preferably, the step 2) includes image normalization and wavelet denoising.

[0023] Preferably, the step 3) includes using LSTM to extract temporal features and ResNet to extract image features, performing multimodal feature extraction and dynamic fusion.

[0024] Preferably, in the step 2), the vibration signal is preprocessed by wavelet denoising, the infrared image is preprocessed by image normalization, and the operation and maintenance log is preprocessed by text segmentation.

[0025] Preferably, after the input data into the system in step 4) is divided into three branches: rule reasoning, probabilistic reasoning and AI enhancement.

[0026] Preferably, the rule reasoning in the step 4) is based on expert experience and industry specifications, matches the predefined rule base, and decouples the business decision logic from the application code; specifically, the Drools business rule engine is used, and the Drools core architecture is divided into a rule management layer and a rule execution layer. The rule management layer compiles the rule files and rule versions into a rule base through rules. The rule execution layer constructs a KIE container as the rule engine entry based on the rule base, and inputs the multimodal data features of the sensor acquisition device into the working memory. The rule engine traverses the rule base, matches the conditions with the facts in the working memory, controls the triggering order through the rule priority and the rule flow group, executes the rule matching actions in sequence, and outputs the generated events, updated data, and called service results.

[0027] Preferably, the probabilistic reasoning in step 4) performs fault diagnosis, life prediction and risk decision-making by quantifying the possibility of the equipment state. Specifically, a dynamic Bayesian network is used to discretize time into multiple time slices, each slice represents the state of the system at a certain moment, and the same variable between adjacent time slices has a state transfer relationship.

[0028] Preferably, in the step 4), the AI enhancement uses the equipment implementation status, historical change trends, environment and load, and maintenance records as data features, which are inferred and analyzed by experts, and labels are introduced as training sets to use the fully connected network FCN and long short-term memory network LSTM model to optimize the rule base threshold, so that the system can adapt to the influence of equipment aging and environmental changes, thereby improving the accuracy of fault detection.

[0029] Preferably, the step 4) preliminarily integrates the outputs of rule reasoning, probabilistic reasoning and AI enhancement, uses weighted voting to assign weights, evaluates weight indicators based on confidence, uses reinforcement learning to optimize weights based on historical decision-making results, and increases its weight if a module makes correct decisions multiple times; each module is an independent source of evidence, and a conflict resolution mechanism is established when the conclusions of each module are inconsistent; and finally, a final decision is output based on the output decision score.

[0030] Advantages of the present invention: The system is rationally designed and applicable to scenarios such as power stations, distribution rooms, and wind turbines. It provides a plant equipment decision-making system and method that combines multimodal data fusion and a hybrid reasoning engine, including fault diagnosis, life prediction, and dynamic maintenance strategy generation. Specific advantages include the following:

[0031] 1) Technological fusion innovation: Adopting a cross-modal attention mechanism to dynamically weight and fuse multi-source features such as sensor data, infrared images, vibration signals, and text logs, this solves the problem of insufficient utilization of multi-source data of factory equipment.

[0032] 2) Explainability: Introducing dynamic Bayesian networks and rule bases to quantify the uncertainty of prediction results, enhance model transparency, and meet the stringent decision-making requirements of special fields.

[0033] 3) Dynamic update of knowledge base: Introducing incremental learning and adaptive optimization of the knowledge base to solve the problem that traditional knowledge base updates rely on manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a framework diagram of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning of the present invention.

[0035] Figure 2 It is a hybrid reasoning decision flowchart.

[0036] Figure 3 This is the core architecture diagram of the Drools rule engine.

[0037] Figure 4 This is the AI enhancement module structure diagram.

[0038] Figure 5 This is an architecture diagram of an embodiment of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below with reference to examples and specific implementation methods.

[0040] like Figure 1 As shown in FIG, the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning includes a multimodal data fusion architecture and a hybrid reasoning model, wherein:

[0041] Multimodal data fusion architecture: Integrates sensor signals (temperature, current) and infrared thermal imaging to obtain multimodal data such as infrared images, vibration signals, and operation and maintenance logs, and dynamically weights and fuses features through a cross-modal attention mechanism.

[0042] Hybrid reasoning model: The rule reasoning module is used to generate production rules based on the expert knowledge base. The probabilistic reasoning module uses dynamic Bayesian modeling to model the equipment state transition probability to process uncertain data. The AI enhancement module introduces artificial neural networks to optimize rule reasoning parameters and combines transfer learning to adapt to different factory scenarios.

[0043] The treatment methods include:

[0044] 1) The data acquisition layer collects multimodal data, including sensor signals, infrared images, vibration signals, and operation and maintenance logs;

[0045] 2) Perform data preprocessing on multimodal data;

[0046] 3) Perform feature extraction and fusion on the preprocessed data to obtain multimodal data features;

[0047] 4) Using multimodal data features for hybrid reasoning operations;

[0048] 5) Decision execution, including work order generation and knowledge base update.

[0049] Specifically,

[0050] The data acquisition layer module collects multi-source data in real time. Multimodal data includes sensor signals, infrared images, vibration signals, and operation and maintenance logs.

[0051] The multimodal data were preprocessed using image normalization, wavelet denoising and other schemes;

[0052] After using LSTM to extract time series features and ResNet to extract image features, multimodal feature extraction and dynamic fusion are performed;

[0053] Based on the extracted features, hybrid reasoning is performed by combining feature reasoning solutions such as the Drools rule engine and dynamic Bayesian networks;

[0054] Output factory equipment operation and maintenance strategies and close-loop optimization knowledge base.

[0055] Among them, vibration signals are preprocessed using wavelet denoising, infrared images are preprocessed using image normalization, and operation and maintenance logs are preprocessed using text segmentation.

[0056] Perform multimodal data fusion on the preprocessed data and dynamically assign importance weights of different modalities (sensors, images, texts), as shown in formulas (1-1) and (1-2).

[0057] α i =Softmax(W.[f sensor ;f image ;f text ])(1-1)

[0058]

[0059] Among them, f sensor is the feature vector of sensor data, with dimension d sensor ;f image is the feature vector of the infrared image, with dimension d image ;ftext is the feature vector of the text log, with dimension d text ; W is the trainable weight matrix with dimension 1×(d sensor +d image +d text ); α i is the weight of each mode; f fused is the weighted eigenvector.

[0060] Use multimodal data features to perform hybrid reasoning processes, such as Figure 2 As shown in the figure, after the input data enters the system, it is divided into three branches: rule reasoning, probabilistic reasoning, and AI enhancement.

[0061] Rule reasoning is based on expert experience and industry standards, matching a predefined rule base. This decouples business decision logic from application code, enabling independent management, dynamic adjustment, and efficient execution of complex business rules.

[0062] As an embodiment, the Drools business rule engine is used, which includes modules such as knowledge base and session. Figure 3 As shown in the figure, the Drools core architecture consists of a rule management layer and a rule execution layer. The rule management layer compiles rule files and rule versions into a rule base. The rule execution layer builds a KIE container based on the rule base as the entry point for the rule engine, inputs the multimodal data features of sensor acquisition devices into the working memory, and the rule engine traverses the rule base, matching conditions with facts in the working memory. It controls the triggering sequence through rule priorities and rule flow groups, executes rule matching actions in sequence, and outputs generated events, updated data, and invoked service results.

[0063] Probabilistic reasoning quantifies the likelihood of equipment states to facilitate fault diagnosis, lifespan prediction, and risk-based decision-making. The core of probabilistic reasoning is Bayes' theorem, as shown in Equation (1-3):

[0064]

[0065] Where P(H|E) is the posterior probability of hypothesis H given evidence E; P(E|H) is the likelihood probability of E when hypothesis H is true; P(H) is the prior probability of hypothesis H; and P(E) is the marginal probability of evidence E (normalization factor).

[0066] As an example, a dynamic Bayesian network is used to discretize time into multiple time slices. Each slice represents the state of the system at a certain moment. The same variable between adjacent time slices has a state transition relationship. Monte Carlo sampling is used to approximate the inference of the posterior distribution and generate N particles from the prior distribution P(X1). Generate new particles based on the state transition model Calculating particle weights By resampling particles according to weights, the posterior probability is approximately a weighted particle set, as shown in formula (1-4):

[0067]

[0068] As an embodiment, the AI enhancement module optimizes traditional rules and probability models by introducing artificial intelligence technology to improve the system's adaptability and efficiency in processing complex scenarios, such as Figure 4 As shown in the figure, the system uses data such as equipment status, historical trends, environment and load, and maintenance records as data features. Experts perform inference analysis and infer data labels based on historical failure cases. For example, if a transformer fails at 92°C, the analysis threshold should be changed from the original 95°C to 90°C. Using the expert-derived labels as a training set, the rule base threshold is optimized using a fully connected network (FCN) and long short-term memory (LSTM) model. This enables the system to adapt to multiple factors such as equipment aging and environmental changes, significantly improving fault detection accuracy.

[0069] The system preliminarily integrates the outputs of rule reasoning, probabilistic reasoning, and AI enhancement, and uses weighted voting to distribute weights. The distribution score is shown in formula (1-5).

[0070] Conf 综合 =ω rule ·Con f rule +ω prob ·Con f prob +ω AI ·Con f AI (1-5)

[0071] Among them, Conf 综合 is the decision score; rule is the weight of the rule reasoning result; Con frule is the confidence of the rule reasoning result; ω prob is the weight of the probability reasoning result; Conf prob is the confidence level of the probability reasoning result; ω AI Enhance the result weight for AI; Con f AI Enhance confidence in AI results.

[0072] The weight index is evaluated based on the confidence level, and reinforcement learning is used to optimize the weight based on the historical decision-making results. If a module makes correct decisions many times, its weight is increased.

[0073] As an example, each module of the system is an independent source of evidence, and a conflict resolution mechanism is established when the conclusions of the modules are inconsistent:

[0074] Confidence threshold filtering: When the confidence of a module is >90%, it will be adopted first;

[0075] Manual review trigger: When there is a conflict and the confidence level is less than 70%, an alarm is pushed to the operation and maintenance personnel;

[0076] Historical data backtracking: matching similar historical cases and recommending majority voting results.

[0077] Finally, the final decision is output based on the output decision score:

[0078] High confidence (>90%): Automatically generate a work order and trigger a shutdown instruction;

[0079] Medium confidence (60-90%): pushes operation and maintenance data, and prompts to arrange manual attachments and prepare spare parts;

[0080] Low confidence (<60%): Record data and monitor continuously.

[0081] Example

[0082] like Figure 5 As shown in the figure, the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning includes:

[0083] Data Collection Layer: The data collection layer is responsible for collecting multimodal raw data from physical devices and the environment in real time, ensuring data integrity, real-time performance, and reliability. This layer includes the collection of sensor networks, infrared thermal imager signals, and operation and maintenance logs.

[0084] Data processing layer: Cleans, converts, and stores raw data to provide high-quality structured input for upper-level decision-making. It includes a real-time stream processing engine, image preprocessing module, and ETL pipeline to process corresponding sensor networks, infrared thermal imager signals, and operation and maintenance log information.

[0085] Intelligent Decision-Making Layer: Generates diagnostic conclusions and strategies based on multimodal data fusion and hybrid reasoning models. The multimodal fusion engine uses a cross-modal attention mechanism to dynamically calculate sensor, image, and text weights and perform feature concatenation. The hybrid inference engine performs hybrid inference on the concatenated multimodal features, dynamically assigning weights for rule-based reasoning, probabilistic reasoning, and AI-enhanced reasoning, and outputs the confidence level of the fused decision.

[0086] The execution feedback layer converts decision results into executable tasks and continuously optimizes the system based on feedback data. The system includes a work order management system, a federated learning terminal, and an operations and maintenance feedback loop. The work order management system automatically generates work orders and optimizes resource scheduling. The federated learning terminal conducts local operations and maintenance training and manages the history of operations and maintenance version iterations. The operations and maintenance feedback loop provides feedback on the status of equipment after maintenance and updates the knowledge base.

[0087] The above description is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the creative concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning, characterized by: It includes a multimodal data fusion architecture and a hybrid reasoning model. The multimodal data fusion architecture is used to integrate multimodal data and dynamically weighted fusion features through a cross-modal attention mechanism. The hybrid reasoning model includes a rule reasoning module, a probabilistic reasoning module and an AI enhancement module. The rule reasoning module generates production rules based on the expert knowledge base. The probabilistic reasoning module uses dynamic Bayesian modeling to model the device state transition probability to process uncertain data. The AI enhancement module introduces an artificial neural network to optimize the rule reasoning parameters and combines transfer learning to adapt to different factory scenarios.

2. The plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 1 is characterized in that: The multimodal data includes sensor signals, infrared images, vibration signals, and operation and maintenance logs.

3. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 2 is characterized in that: The following steps are involved: 1) The data acquisition layer collects multimodal data; 2) Perform data preprocessing on multimodal data; 3) Perform feature extraction and fusion on the preprocessed data to obtain multimodal data features; 4) Using multimodal data features to perform hybrid reasoning operations by the hybrid reasoning model; 5) Decision execution, including work order generation and knowledge base update.

4. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 3 is characterized in that: In the step 2), the vibration signal is preprocessed by wavelet denoising, the infrared image is preprocessed by image normalization, and the operation and maintenance log is preprocessed by text segmentation.

5. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 4 is characterized in that: The step 3) includes using LSTM to extract time series features and ResNet to extract image features, performing multimodal feature extraction and dynamic fusion.

6. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 5 is characterized in that: After the input data in step 4) enters the system, it is divided into three branches: rule reasoning, probabilistic reasoning and AI enhancement.

7. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 6 is characterized in that: The rule reasoning in step 4) is based on expert experience and industry standards, matches the predefined rule base, and decouples the business decision logic from the application code; Specifically, the Drools business rule engine is used. The Drools core architecture is divided into the rule management layer and the rule execution layer. The rule management layer compiles the rule files and rule versions into a rule base. The rule execution layer builds a KIE container as the rule engine entry based on the rule base, and inputs the multimodal data features of the sensor collection device into the working memory. The rule engine traverses the rule base, matches the conditions with the facts in the working memory, controls the triggering order through the rule priority and rule flow group, executes the rule matching actions in sequence, and outputs the generated events, updated data, and called service results.

8. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 6 is characterized in that: In step 4), probabilistic reasoning is performed to quantify the likelihood of equipment status to perform fault diagnosis, life prediction, and risk decision-making. Specifically, a dynamic Bayesian network is used to discretize time into multiple time slices, each of which represents the state of the system at a certain moment. The same variable between adjacent time slices has a state transition relationship.

9. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to claim 6 is characterized in that: In step 4), the AI enhancement uses the equipment implementation status, historical change trends, environment and load, and maintenance records as data features, which are inferred and analyzed by experts. Labels are introduced as training sets, and the fully connected network FCN and long short-term memory network LSTM models are used to optimize the rule base threshold. This allows the system to adapt to the influence of equipment aging and environmental changes, thereby improving the accuracy of fault detection.

10. The processing method of the plant equipment operation and maintenance decision system based on multimodal data fusion and hybrid reasoning according to any one of claims 6 to 9, characterized in that: Step 4) performs a preliminary fusion of the outputs of rule-based reasoning, probabilistic reasoning, and AI enhancement, assigns weights using a weighted voting method, evaluates weight indicators based on confidence, and uses reinforcement learning to optimize weights based on historical decision-making results. If a module makes correct decisions multiple times, its weight is increased; Each module is an independent source of evidence. When the conclusions of each module are inconsistent, a conflict resolution mechanism is established; the final decision is output based on the output decision score.

Images