Intelligent patrol multi-source information fusion sudden fault diagnosis and disposal linkage system
The intelligent inspection multi-source information fusion system enables collaborative operation of drones, ground robots and fixed cameras. By combining multi-source data fusion and knowledge graphs, it solves the problems of coverage blind spots and response delays in traditional inspection modes, improves the comprehensiveness and timeliness of fault diagnosis, and builds a closed-loop linkage for the entire process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA RESOURCES NEW ENERGY INVESTMENT CO LTD SHANDONG BRANCH
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-24
AI Technical Summary
Traditional equipment inspection and fault diagnosis modes cannot achieve full-time, all-around three-dimensional coverage and cannot achieve closed-loop linkage of diagnosis-handling-feedback throughout the entire process, resulting in serious response delays.
The system employs intelligent patrol and multi-source information fusion for emergency fault diagnosis and response linkage. It includes a patrol coordination and scheduling module, a multi-source information acquisition module, a multi-source information fusion module, a fault diagnosis module, and a response linkage module. Through the collaborative operation of drones, ground robots, and fixed cameras, combined with an improved genetic algorithm and time-slice rotation mechanism, it achieves full-time domain coverage without blind spots. Furthermore, it enhances the credibility of the diagnostic model through multi-source data fusion and knowledge graphs, and constructs a closed-loop linkage for the entire process of diagnosis, response, and feedback.
It achieves full-time, all-around coverage, reduces the missed detection rate, improves the timeliness and accuracy of fault diagnosis, shortens the fault response and handling cycle, breaks down data silos, and enhances the understanding and adoption capabilities of operation and maintenance personnel.
Smart Images
Figure CN121920631A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent operation and maintenance and fault diagnosis technology, specifically involving an intelligent inspection multi-source information fusion system for sudden fault diagnosis and handling linkage. Background Technology
[0002] Against the backdrop of accelerated global digital transformation and the construction of new infrastructure, the scale of equipment in key sectors such as power, chemicals, and intelligent manufacturing continues to expand and its structure becomes increasingly complex, placing unprecedented demands on equipment operation safety and maintenance efficiency. Sudden failures, as a core risk affecting the continuous and stable operation of equipment, can not only lead to huge economic losses but also trigger safety accidents and disruptions to public services. Therefore, building an efficient and accurate intelligent inspection and fault diagnosis and handling system has become a core requirement for ensuring the safe operation of infrastructure in critical sectors.
[0003] Traditional equipment inspection and fault diagnosis models have significant limitations and are difficult to adapt to the needs of responding to sudden faults in complex scenarios. In the inspection phase, the introduction of single intelligent inspection devices (such as drones, ground robots, and fixed cameras) still has coverage blind spots: drones are good at high-altitude operations but are not suitable for long-term stationing; robots are highly adaptable to terrain but have limited field of view; fixed cameras can monitor around the clock but have blind spots and cannot achieve full-time, blind-spot-free three-dimensional coverage. In the fault diagnosis phase, fault diagnosis and handling are disconnected. Traditional systems mostly stay at the fault identification level and fail to achieve a closed-loop linkage of diagnosis-handling-feedback. After a sudden fault occurs, manual coordination of multiple departments is required to carry out handling work, resulting in serious response delays.
[0004] Therefore, an intelligent patrol system that integrates multi-source information for emergency fault diagnosis and response is needed to address the problems of existing single intelligent patrol devices, which cannot achieve full-time, blind-spot-free three-dimensional coverage and fail to achieve a closed-loop linkage of diagnosis, response, and feedback, resulting in severe response delays. Summary of the Invention
[0005] The purpose of this invention is to provide an intelligent patrol system that integrates multi-source information for emergency fault diagnosis and handling, in order to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent patrol system for multi-source information fusion in the diagnosis and handling of sudden faults, comprising:
[0007] The patrol coordination and scheduling module is used to manage various intelligent patrol devices in a unified manner. Based on preset patrol tasks or sudden fault trigger signals, it generates a coordinated patrol strategy and schedules the corresponding devices to perform patrol operations. The intelligent patrol devices include at least drones, ground robots, and fixed cameras. The patrol coordination and scheduling module includes a coverage blind spot analysis unit, a route / path planning unit, and an equipment coordination control unit.
[0008] The multi-source information acquisition module is communicatively connected to the patrol coordination and scheduling module, and is used to synchronously acquire multiple types of patrol data through the intelligent patrol equipment and sensors deployed on the equipment body. The multiple types of patrol data include image data, video data, vibration data, temperature data, and acoustic data.
[0009] The multi-source information fusion module is communicatively connected to the multi-source information acquisition module and is used to perform standardized preprocessing, feature extraction and fusion analysis on the acquired multi-type inspection data to obtain a unified set of equipment status features.
[0010] The fault diagnosis module is communicatively connected to the multi-source information fusion module. It is used to quickly identify and locate sudden faults based on the fused equipment status feature set through a lightweight intelligent diagnostic model, and output fault diagnosis results.
[0011] The handling linkage module is communicatively connected to the fault diagnosis module and the inspection coordination and scheduling module, respectively. It is used to generate handling instructions based on the fault diagnosis results and push them to the corresponding operation and maintenance terminals and departments, while receiving handling feedback information, forming a closed-loop linkage of the entire process of diagnosis-handling-feedback.
[0012] The data storage and management module communicates with each of the above modules and is used to store inspection data, fusion results, diagnostic results, treatment records and system configuration information, and provides data query and update functions.
[0013] The solution specifies that the coverage blind spot analysis unit is used to analyze the blind spots of a single device based on terrain data of the work area and the performance parameters of each intelligent patrol device, and to determine a collaborative coverage scheme; the flight path / route planning unit is used to generate the optimal flight path of the UAV and the optimal movement path of the ground robot using an improved genetic algorithm based on the preliminary location information of sudden failures, while simultaneously scheduling fixed cameras to adjust their monitoring angles; the device collaborative control unit uses a time-slice rotation mechanism to coordinate the work sequence, and achieves real-time scheduling of each intelligent patrol device through the 5G communication protocol. The work sequence coordination calculation formula is as follows:
[0014]
[0015] in, The start time of the kth device. For time slices, This is the distance-related offset (the offset increases by 0.2s for every 10 meters away from the fault area).
[0016] It is worth noting that the feature extraction process of the multi-source information fusion module includes extracting visual features of images / videos using a ResNet18 network and extracting time-domain and frequency-domain features of sensor data using wavelet packet decomposition.
[0017] Furthermore, it should be noted that the lightweight intelligent diagnostic model adopts an architecture that combines L1 regularized pruned MobileNetV2 with a knowledge graph. The knowledge graph is used to store device structure information, historical fault cases and diagnostic rules. The diagnostic results are verified by semantic similarity and explanatory text is generated. When the semantic similarity is ≥0.8, the diagnostic results are considered valid.
[0018] In a preferred embodiment, the handling linkage module includes an instruction generation unit, a multi-channel push unit, and a feedback receiving and processing unit. The instruction generation unit is used to automatically match a preset handling plan based on the fault type, severity, and location information. The multi-channel push unit pushes instructions to the operation and maintenance personnel's terminals and corresponding management departments via SMS, APP push, and background system pop-ups. The feedback receiving and processing unit is used to receive the handling progress, handling results, and supplementary on-site data uploaded by the operation and maintenance personnel, and feeds back the handling completion information to the fault diagnosis module and the data storage and management module, while triggering subsequent review and inspection tasks.
[0019] As a preferred implementation, the data storage and management module adopts a distributed storage architecture, supports the classified storage of structured and unstructured data, and also has data encryption, backup and log auditing functions.
[0020] Compared with existing technologies, the intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system provided by the present invention has at least the following beneficial effects:
[0021] The inspection and coordination module coordinates the collaborative operations of drones, ground robots, and fixed cameras. It combines an improved genetic algorithm to plan optimal routes and a time-slice rotation mechanism to coordinate timing, achieving full-time, comprehensive, and seamless three-dimensional coverage of the work area. This improves the comprehensiveness and timeliness of inspections in areas with sudden faults, reducing the missed detection rate. The multi-source information fusion module standardizes and weights various types of data, breaking down data silos and enriching the dimensions of equipment status characteristics. It also combines knowledge graphs to semantically verify and interpret diagnostic results, solving the "black box" problem of traditional deep learning models, improving model credibility, and facilitating understanding and adoption by maintenance personnel. The response and linkage module constructs a full-process mechanism of "diagnosis-response-feedback," automatically completing fault severity assessment, response plan matching, instruction push, and feedback verification. This eliminates the need for manual coordination among multiple departments, significantly shortening the fault response and handling cycle. Attached Figure Description
[0022] Figure 1 This is a structural block diagram of the intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system of the present invention;
[0023] Figure 2 This is a flowchart illustrating the usage of the intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system of the present invention. Detailed Implementation
[0024] The present invention will be further described below with reference to embodiments.
[0025] Reference Figure 1 As shown, this invention provides an intelligent patrol system for multi-source information fusion, enabling joint diagnosis and handling of sudden faults, comprising:
[0026] The patrol coordination and scheduling module is used to manage various intelligent patrol equipment such as drones, ground robots, and fixed cameras. It includes a blind spot analysis unit, a route / path planning unit, and an equipment coordination control unit.
[0027] The multi-source information acquisition module is connected to the patrol coordination and scheduling module. Through the sensors deployed on each intelligent patrol device and the device itself, it constructs a three-dimensional, blind-spot-free dynamic monitoring network in the target area, and collects multi-modal patrol data such as images, videos, vibrations, temperatures, and acoustics in real time, providing a data foundation for subsequent fusion analysis.
[0028] The multi-source information fusion module is connected to the multi-source information acquisition module and is used to perform standardized preprocessing, feature extraction and fusion analysis on the acquired multi-type inspection data to obtain a unified set of equipment status features.
[0029] The fault diagnosis module, connected to the multi-source information fusion module, is used to quickly identify and locate sudden faults based on the fused equipment status feature set through a lightweight intelligent diagnostic model, and output fault diagnosis results.
[0030] The handling linkage module is connected to the fault diagnosis module and the inspection coordination and scheduling module respectively. It is used to generate handling instructions based on the fault diagnosis results and push them to the corresponding operation and maintenance terminals and departments. At the same time, it receives handling feedback information, forming a closed-loop linkage of the entire process of "diagnosis-handling-feedback".
[0031] The data storage and management module is connected to each of the above modules and is used to store inspection data, fusion results, diagnostic results, treatment records and system configuration information, and provides data query and update functions.
[0032] Furthermore, the core function of the patrol coordination and dispatch module is to coordinate various intelligent patrol devices to achieve collaborative operations and eliminate coverage blind spots;
[0033] The blind spot analysis unit analyzes the blind spots of a single device based on terrain data of the work area (such as elevation and obstacle distribution) and performance parameters of each device (such as drone endurance, robot climbing ability, and camera monitoring range), and outputs a collaborative coverage solution.
[0034] For sudden failure scenarios, the flight path / routes planning unit uses an improved genetic algorithm to construct a multi-objective optimization model that "maximizes coverage integrity and minimizes operation time," generating the optimal flight path for the UAV and the optimal movement path for the ground robot. It also schedules fixed cameras to adjust their monitoring angles to ensure full coverage of the failure area. The multi-objective optimization function is calculated using the following formula:
[0035]
[0036] in, For the route / path decision variable matrix, These are the weighting coefficients. This refers to the actual coverage area. This represents the total area of the region to be inspected. Total collaborative work time;
[0037] The equipment collaborative control unit uses a time-slice round-robin mechanism to coordinate the operation sequence of each device, and combines it with the 5G communication protocol to achieve real-time scheduling, avoiding operation conflicts. The operation sequence coordination calculation formula is as follows:
[0038]
[0039] in, The start time of the kth device. For time slices, This is the distance-related offset (the offset increases by 0.2s for every 10 meters away from the fault area).
[0040] Furthermore, the multi-source information fusion module adopts a three-level architecture of "preprocessing - feature extraction - weighted fusion" to process multi-source data;
[0041] In the preprocessing stage, Gaussian filtering and histogram equalization are used to denoise the image data, and Kalman filtering and min-max normalization (mapped to the [0,1] interval) are used to denoise the vibration, temperature and other sensor data.
[0042] In the feature extraction stage, the ResNet18 network is used to extract 2048-dimensional visual features of images / videos, and the 8-dimensional frequency domain energy features of vibration data are extracted through 3-layer wavelet packet decomposition. The mean, variance, and peak value of temperature data are calculated as 3-dimensional time series features.
[0043] In the fusion phase, the weights of each data source are first obtained through BP neural network training, and then the multi-dimensional features are weighted and fused by combining the improved DS evidence theory to obtain a unified set of device status features, breaking down data silos.
[0044] Furthermore, the fault diagnosis module adopts a fusion architecture of "L1 regularized pruning MobileNetV2 + knowledge graph":
[0045] Among them, the architecture that combines a pruned convolutional neural network with a knowledge graph is adopted: L1 regularization is used to prune the channels of MobileNetV2 to reduce the number of model parameters;
[0046] After inputting the multi-source fusion feature set, fault features are extracted using a lightweight model. The Softmax function is then used to output the probability distribution of each fault type to preliminarily determine the fault type. The fault probability calculation formula is as follows:
[0047]
[0048] in, Let i be the probability of the i-th type of failure. The output value of the model is n, where n is the total number of fault types.
[0049] The diagnostic results are semantically similar to those obtained using a fault knowledge graph. If the similarity is ≥0.8, the verification is successful and a diagnostic explanation text is generated.
[0050] Furthermore, the handling linkage module realizes closed-loop linkage between fault diagnosis and operation and maintenance handling;
[0051] First, the severity of the fault is assessed using a quantitative formula. Then, based on the fault type, severity, and location information, a pre-set contingency plan is automatically matched to generate a handling instruction that includes fault details, handling procedures, responsible department, and time limits. The fault severity assessment formula is as follows:
[0052]
[0053] in, Severity level (1-10 points). The area affected by the fault. The total area of the work area;
[0054] Commands are pushed based on message priority rules, with an urgency coefficient of 0 for general faults and 1 for critical faults, ensuring that commands for serious faults are pushed first. Commands are pushed to maintenance terminals and relevant departments through multiple channels such as SMS, APP push, and backend system pop-ups. The message push priority calculation formula is as follows:
[0055]
[0056] in, Priority The urgency factor is 0 / 1.
[0057] After receiving the processing progress, results, and supplementary on-site data uploaded by maintenance personnel, and verifying the data integrity (uploading all required fields constitutes a complete record), the system pushes the data to the data storage and management module for archiving and triggers a review and inspection task.
[0058] Furthermore, the data storage and management module adopts a 3-node distributed storage cluster with a total storage capacity of no less than 100TB, supporting the classified storage of structured and unstructured data; it has data encryption, redundant backup, query update and log auditing functions, providing secure and reliable data support for all modules of the system.
[0059] refer to Figure 2 As shown, the usage process of the above-mentioned intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system is as follows:
[0060] Step 1: When the operating equipment detects an anomaly, it triggers the emergency fault response mechanism and sends preliminary fault location information to the inspection and coordination scheduling module;
[0061] Step 2: After receiving the location information, the patrol coordination and scheduling module, combined with the terrain and equipment performance parameters of the area, determines the collaborative coverage scheme of UAV + ground robot + fixed camera.
[0062] The route / path planning unit generates high-altitude inspection routes for UAVs and movement paths for ground robots by using an improved genetic algorithm, and schedules fixed cameras to adjust their angles to align with the fault area.
[0063] The equipment collaborative control unit allocates the operation sequence through a time-slice rotation mechanism, including the start time of the drone, the start time of the ground robot, and the synchronous start of the fixed camera, and issues operation instructions through the 5G network.
[0064] Step 3: The multi-source information acquisition module acquires busbar temperature images using an UAV infrared thermal imager, busbar vibration data and high-definition images of joints using a ground robot, and real-time video using a fixed camera; the multi-source information fusion module performs Gaussian filtering for noise reduction and histogram equalization on the images, and Kalman filtering and min-max normalization on the vibration data; it extracts ResNet18 visual features, 8-dimensional frequency domain energy features of the vibration data, and temperature mean features; it obtains the weights of the image, vibration, and temperature data sources through a BP neural network, and completes feature fusion by combining improved DS evidence theory to obtain the equipment status fusion feature set;
[0065] Step 4: The lightweight diagnostic model in the fault diagnosis module analyzes the fused feature set and outputs the probability of each fault type through the Softmax function. For example, the probability of "loose joint" is 0.92, and it is initially determined to be this fault. The knowledge graph matches the relevant entities of the fault type, and the semantic similarity calculation result is 0.86 (≥0.8), which is verified and a diagnostic explanation is generated. The diagnostic result is pushed to the handling linkage module.
[0066] Step 5: The response linkage module calculates S using the severity formula, matches it with the response plan, and generates a response instruction. This instruction is then pushed to the high-voltage maintenance team's terminals via the APP, and a pop-up notification appears in the background monitoring center. After receiving the instruction, the maintenance personnel go to the site and upload the response progress in real time via the APP. Once the fault is resolved, they upload the retest data.
[0067] Step Six: The handling linkage module verifies the completeness of the feedback data and pushes it to the data storage and management module for archiving; triggers the review and inspection task, dispatches the ground robot to retest the processed equipment, confirms that the temperature and vibration data are normal, the fault is eliminated, and a closed loop is formed.
[0068] In summary, the advantages of this invention are as follows: The inspection and coordination scheduling module coordinates the collaborative operations of drones, ground robots, and fixed cameras, combining an improved genetic algorithm to plan optimal routes and a time-slice rotation mechanism to coordinate timing, achieving full-time, blind-spot-free three-dimensional coverage of the work area, improving the comprehensiveness and timeliness of inspections in areas with sudden faults, and reducing the missed detection rate; the multi-source information fusion module standardizes and weights the fusion of various types of data, breaking down data silos, enriching the dimensions of equipment status characteristics, and combining knowledge graphs to semantically verify and interpret diagnostic results, solving the "black box" problem of traditional deep learning models, improving model credibility, and facilitating understanding and adoption by maintenance personnel; the handling linkage module constructs a full-process mechanism of "diagnosis-handling-feedback," automatically completing fault severity assessment, handling plan matching, instruction push, and feedback verification, eliminating the need for manual coordination of multiple departments and significantly shortening the fault response and handling cycle.
[0069] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. An intelligent patrol system for fusion of multi-source information to diagnose and handle sudden faults, characterized in that: include: The patrol coordination and scheduling module is used to manage various intelligent patrol devices in a unified manner. Based on preset patrol tasks or sudden fault trigger signals, it generates a coordinated patrol strategy and schedules the corresponding devices to perform patrol operations. The intelligent patrol devices include at least drones, ground robots, and fixed cameras. The patrol coordination and scheduling module includes a coverage blind spot analysis unit, a route / path planning unit, and an equipment coordination control unit. The multi-source information acquisition module is communicatively connected to the patrol coordination and scheduling module, and is used to synchronously acquire multiple types of patrol data through the intelligent patrol equipment and sensors deployed on the equipment body. The multiple types of patrol data include image data, video data, vibration data, temperature data, and acoustic data. The multi-source information fusion module is communicatively connected to the multi-source information acquisition module and is used to perform standardized preprocessing, feature extraction and fusion analysis on the acquired multi-type inspection data to obtain a unified set of equipment status features. The fault diagnosis module is communicatively connected to the multi-source information fusion module. It is used to quickly identify and locate sudden faults based on the fused equipment status feature set through a lightweight intelligent diagnostic model, and output fault diagnosis results. The handling linkage module is communicatively connected to the fault diagnosis module and the inspection coordination and scheduling module, respectively. It is used to generate handling instructions based on the fault diagnosis results and push them to the corresponding operation and maintenance terminals and departments, while receiving handling feedback information, forming a closed-loop linkage of the entire process of diagnosis-handling-feedback. The data storage and management module communicates with each of the above modules and is used to store inspection data, fusion results, diagnostic results, treatment records and system configuration information, and provides data query and update functions.
2. The intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system according to claim 1, characterized in that: The coverage blind spot analysis unit is used to analyze the blind spots of a single device based on the terrain data of the work area and the performance parameters of each intelligent inspection device, and to determine a collaborative coverage scheme. The flight path / route planning unit is used to generate the optimal flight path of the UAV and the optimal movement path of the ground robot based on the preliminary location information of sudden failures, and simultaneously schedules fixed cameras to adjust the monitoring angle. The device collaborative control unit uses a time-slice rotation mechanism to coordinate the operation sequence and realizes real-time scheduling of each intelligent inspection device through the 5G communication protocol. The operation sequence coordination calculation formula is: in, The start time of the kth device. For time slices, This is the distance-related offset (the offset increases by 0.2s for every 10 meters away from the fault area).
3. The intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system according to claim 1, characterized in that: The feature extraction process of the multi-source information fusion module includes extracting visual features of images / videos using a ResNet18 network and extracting time-domain and frequency-domain features of sensor data using wavelet packet decomposition.
4. The intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system according to claim 1, characterized in that: The lightweight intelligent diagnostic model adopts an architecture that combines L1 regularized pruned MobileNetV2 with a knowledge graph. The knowledge graph is used to store device structure information, historical fault cases and diagnostic rules. The diagnostic results are verified by semantic similarity and explanatory text is generated. The diagnostic results are considered valid when the semantic similarity is ≥0.
8.
5. The intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system according to claim 1, characterized in that: The handling linkage module includes an instruction generation unit, a multi-channel push unit, and a feedback receiving and processing unit. The instruction generation unit is used to automatically match a preset handling plan based on the fault type, severity, and location information. The multi-channel push unit pushes instructions to the operation and maintenance personnel's terminals and corresponding management departments via SMS, APP push, and background system pop-up. The feedback receiving and processing unit is used to receive the handling progress, handling results, and supplementary on-site data uploaded by the operation and maintenance personnel, and feeds back the handling completion information to the fault diagnosis module and the data storage and management module, while triggering subsequent review and inspection tasks.
6. The intelligent patrol multi-source information fusion emergency fault diagnosis and handling linkage system according to claim 1, characterized in that: The data storage and management module adopts a distributed storage architecture, supports the classified storage of structured and unstructured data, and also has data encryption, backup and log auditing functions.