Intelligent inspection monitoring method and system based on electronic vision

Abstract

The invention discloses an intelligent inspection monitoring method and system based on electronic vision, and relates to the technical field of industrial automation and intelligent monitoring, and the method comprises the steps: collecting multi-mode inspection data, and carrying out the preprocessing of the data; target identification classification and anomaly detection are carried out based on a convolutional neural network; and optimizing the inspection path by adopting a graph theory model and a shortest path algorithm. According to the intelligent inspection monitoring method based on electronic vision, the image obtained in the inspection process is analyzed through the convolutional neural network, the fault detection precision and the equipment classification accuracy are improved, the flexibility and the accuracy of anomaly detection are greatly improved through the auto-encoder model, and the detection accuracy is improved. The equipment position is regarded as the node of the graph, and the optimal inspection path is calculated by adopting the graph theory model, so that the inspection efficiency is remarkably improved, the inspection scheduling is optimized, the inspection coverage rate is improved, and better effects are achieved in the aspects of fault identification accuracy, inspection efficiency and adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial automation and intelligent monitoring, and in particular to an intelligent inspection and monitoring method and system based on electronic vision. Background Art

[0002] With the continuous improvement of industrial automation and intelligence, equipment inspection technology based on electronic vision has gradually become an important part of intelligent operation and maintenance systems. Traditional equipment inspection methods mostly rely on manual regular inspections, which are inefficient and easy to miss details. Especially in large-scale industrial scenarios, manual inspections often fail to cover all equipment states. In recent years, with the development of image processing technology, machine learning algorithms, and sensor technology, electronic vision technology has gradually been applied to the inspection field. Through cameras, thermal imagers, and a variety of sensors, it can automatically collect multimodal data such as the appearance, thermal state, and sound of the equipment, and evaluate the equipment state through intelligent analysis algorithms, which greatly improves the inspection efficiency. The successful application of deep learning algorithms such as convolutional neural networks (CNN) in image processing has enabled the inspection system based on electronic vision to show excellent performance in equipment fault detection and damage identification. However, although the existing technology can achieve preliminary equipment state recognition, there is still much room for improvement in data fusion, intelligent anomaly detection, and path optimization.

[0003] In the existing related technologies, data collection is often limited to a single type of sensor, and the advantages of multimodal data fusion are not fully utilized, resulting in incomplete and large errors in the equipment status assessment. In addition, the existing system relies on preset rules for the detection of equipment failures, lacks adaptive capabilities, and is difficult to accurately identify complex nonlinear failure modes. Especially in terms of inspection path planning, most technologies still use fixed or experience-driven path selection, fail to dynamically adjust in combination with the health status of the equipment, and the inspection efficiency is limited. In contrast, the intelligent inspection monitoring method based on electronic vision invented in this paper, through multimodal data collection and fusion, target recognition and anomaly detection of convolutional neural networks, and inspection path optimization of graph theory models and shortest path algorithms, can not only more accurately identify potential equipment failures, but also dynamically adjust the inspection frequency and path according to the health status of the equipment, significantly improving the inspection efficiency and the accuracy of fault identification. This innovative method overcomes many shortcomings of the existing technology and has good practicality and application prospects. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing electronic vision method has the problems of poor adaptability, low fault identification accuracy, poor inspection efficiency, and difficulty in accurately identifying complex nonlinear fault modes.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: an intelligent inspection and monitoring method based on electronic vision, including collecting multimodal inspection data and preprocessing the data; target recognition, classification and anomaly detection based on convolutional neural networks; and optimizing the inspection path using a graph theory model and a shortest path algorithm.

[0007] As a preferred solution of the electronic vision-based intelligent inspection and monitoring method described in the present invention, wherein: the collecting of multimodal inspection data and preprocessing of the data includes collecting image data, equipment operation status data, location and time data, sound data and historical data, and performing multimodal data fusion;

[0008] Image data includes visible light images, thermal imaging data, and multispectral images;

[0009] Equipment operating status data includes electrical parameters, temperature and humidity, and vibration data;

[0010] Location and time data includes GPS location information and timestamp;

[0011] The sound data includes the operating audio frequency;

[0012] Historical data includes equipment maintenance records and historical abnormality records;

[0013] The data collected by different sensors are synchronized to perform multimodal data fusion.

[0014] As a preferred solution of the electronic vision-based intelligent inspection and monitoring method described in the present invention, wherein: the acquisition of multimodal inspection data and preprocessing of the data includes the data being affected by ambient light changes, equipment vibrations, and sensor noise factors during the acquisition process, thereby generating noise;

[0015] Using image filtering algorithms to denoise image data includes:

[0016] By taking the median of the pixels around the pixel in the image, noise is removed and edge features are retained;

[0017] By weighted averaging the neighborhood values ​​of pixels, the image is smoothed and noise is removed;

[0018] Considering the spatial proximity and the difference of pixel values, noise is removed while retaining edge information.

[0019] As a preferred solution of the electronic vision-based intelligent inspection and monitoring method described in the present invention, the target recognition classification and anomaly detection based on the convolutional neural network includes calculating pixel by pixel on the input image by moving the convolution kernel to extract local features in the image, including edges, textures, damage or cracks on the surface of the equipment;

[0020] The convolution kernel performs pixel-by-pixel operations on the input image to detect different image features. The convolution operation model is expressed as:

[0021]

[0022] Among them, O(i, j) represents the pixel value of the image at position (i, j), (i, j) represents the coordinates of the image pixel, K(m, n) represents the convolution kernel matrix, k represents the radius of the convolution kernel, m and n represent the movement index of the convolution kernel, and I(i+m, j+n) represents the pixel value of the image at position (i+m, j+n);

[0023] After extracting image features, the convolutional neural network inputs the features into the fully connected layer for classification;

[0024] Construct a classification probability model and convert the score of each category into probability, expressed as:

[0025]

[0026] Among them, P(y=i|z) represents the probability of classifying the image into the i-th category, z=[z1,z2,...,z n ] represents the score of each category output by the neural network in the fully connected layer, z i represents the score of the image belonging to category i, and n represents the number of categories;

[0027] The classification probability model is used to output the classification probability of the equipment status and determine the health status or fault type of the equipment.

[0028] As a preferred solution of the intelligent inspection and monitoring method based on electronic vision described in the present invention, wherein: the target recognition classification and anomaly detection based on the convolutional neural network includes using an autoencoder for anomaly detection, and determining whether the current device state is abnormal by reconstructing the device data and performing error analysis;

[0029] The autoencoder model is constructed as follows:

[0030]

[0031] Where x represents the input data, f(x) represents the encoder function, and g(f(x)) represents the decoder function. Represents the reconstructed data. Represents the difference between the input data and the reconstructed data;

[0032] By minimizing The autoencoder learns the normal state of the device and determines whether there is an abnormality in subsequent detection;

[0033] To quantify the results of anomaly detection, the anomaly scoring model normalizes the reconstruction error of the autoencoder. By setting a threshold, when the value of the loss function exceeds the threshold, the device is considered to have an anomaly. The size of the score determines the severity of the anomaly. The anomaly scoring model is expressed as:

[0034]

[0035] Among them, S represents the anomaly score, and T represents the set threshold;

[0036] If S>1, it means that the equipment is abnormal, and the value of S is proportional to the severity of the abnormality.

[0037] As a preferred solution of the electronic vision-based intelligent inspection and monitoring method described in the present invention, wherein: the use of a graph theory model and a shortest path algorithm to optimize the inspection path includes setting alarm mechanisms of different levels;

[0038] Send reminders when there are minor anomalies, and issue alarms and automatically generate fault reports when there are serious anomalies;

[0039] The alarm notifies maintenance personnel via SMS, email, and APP push notifications;

[0040] The system records all detected abnormalities by time and location and saves them as inspection logs. Each fault record should include the equipment number, abnormality type, image data, abnormality degree and equipment operating parameters;

[0041] The system compares and analyzes current fault information with historical data, evaluates the frequency and trend of abnormalities, and provides a basis for equipment maintenance planning;

[0042] The inspection system uses a lossless compression algorithm to compress image and video data, and stores non-critical data in layers to save storage space;

[0043] Design a multi-dimensional database structure based on time, device location, and exception type;

[0044] The data includes original images, pre-processed images, analysis results, and equipment operating parameters;

[0045] A data transmission protocol is used to transmit data from the front-end acquisition device to the remote monitoring center, and the data is encrypted during transmission.

[0046] As a preferred solution of the electronic vision-based intelligent inspection and monitoring method described in the present invention, wherein: the use of a graph theory model and a shortest path algorithm to optimize the inspection path includes that during the inspection process, the device locations are regarded as nodes of a graph, and the edges connecting the nodes represent the distances between the devices and the inspection time;

[0047] Through the graph theory algorithm, the system determines the shortest path from the inspection starting point to each device. The graph model is expressed as:

[0048] G=(V,E)

[0049] w(e ij )

[0050] Where G represents the graph model of the inspection area, V represents the node set, V = {v1, v2, ..., v n}, each node v i represents the location of a device, E represents the edge set, and each edge e ij Represents device vi and device v j The connected path between them, w(e ij ) represents the weight of the edge, which is the inspection cost between device i and device j;

[0051] By dynamically selecting the device node with the smallest path length, the shortest path from the starting point to all target devices is found. The shortest path model is expressed as:

[0052]

[0053] Where L(P) represents the total length of path P, (i,j)∈P represents each pair of devices i and j on path P;

[0054] Initially, set the starting point v s The path length of the device node is 0, and the path length of the device node is infinite;

[0055] Select the node with the current minimum path length, select the node with the minimum path length from the unvisited nodes each time, and update the path lengths of the adjacent nodes;

[0056] Repeat the selection until the shortest paths to all devices are found;

[0057] Build a dynamic inspection frequency optimization model to dynamically adjust the inspection frequency of the equipment and automatically optimize the inspection priority based on the health status of the equipment, expressed as:

[0058]

[0059]

[0060] Among them, P(i) represents the inspection priority of device i, H(i) represents the health index of device i, F represents the optimization objective function, which is the total inspection cost of the system, N represents the total number of devices that need to be inspected, P(i) represents the inspection priority of device i, L(P i ) represents the inspection path length of device i;

[0061] Priority is inversely proportional to the health index;

[0062] The health index is proportional to the device status.

[0063] Another object of the present invention is to provide an intelligent inspection and monitoring system based on electronic vision, which can detect and score abnormalities through autoencoders, thereby solving the problem of low accuracy of fault identification in current electronic vision methods.

[0064] As a preferred solution of the intelligent patrol monitoring system based on electronic vision described in the present invention, it includes: an initialization module, a data preprocessing module, an intelligent analysis and target recognition module, and a patrol optimization module; the initialization module is used to collect multimodal patrol data; the data preprocessing module is used for analysis and target recognition; the intelligent analysis and target recognition module is used to perform target detection and classification on the preprocessed image data; the patrol optimization module is used to generate an optimal patrol path.

[0065] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of an intelligent inspection and monitoring method based on electronic vision.

[0066] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an intelligent inspection and monitoring method based on electronic vision.

[0067] Beneficial effects of the present invention: The intelligent inspection and monitoring method based on electronic vision provided by the present invention uses a convolutional neural network to analyze the images obtained during the inspection process, extracts important features such as the edge and texture of the equipment surface, improves the accuracy of fault detection and the accuracy of equipment classification, reduces the possibility of human misjudgment, and enhances the intelligence and adaptability of the system. Through the autoencoder model, it adapts to the nonlinear changes in the equipment state and dynamically adjusts according to the actual equipment operation conditions, greatly improving the flexibility and accuracy of anomaly detection. By regarding the equipment position as a node of a graph and using a graph theory model to calculate the optimal inspection path, the inspection efficiency is significantly improved, the inspection scheduling is optimized, and the inspection coverage rate is increased. It can also reduce manpower and time costs through intelligent path optimization. By establishing a multi-level alarm mechanism for mild anomalies and severe faults, it helps maintenance personnel to understand the equipment status in a timely manner and prevent the deterioration of equipment faults. At the same time, it accumulates historical data on equipment operation and maintenance, which is helpful for long-term equipment health status analysis and trend prediction. The present invention achieves better results in terms of fault identification accuracy, inspection efficiency and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0069] Figure 1 An overall flow chart of an intelligent inspection and monitoring method based on electronic vision provided for the first embodiment of the present invention.

[0070] Figure 2 An overall flow chart of an intelligent inspection and monitoring system based on electronic vision provided for the third embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0072] Example 1, reference Figure 1 , as an embodiment of the present invention, provides an intelligent inspection and monitoring method based on electronic vision, comprising:

[0073] S1: Collect multimodal inspection data and preprocess the data.

[0074] Furthermore, image data, equipment operation status data, location and time data, sound data, and historical data are collected to perform multimodal data fusion.

[0075] It should be noted that the image data includes:

[0076] Visible light imaging: Collect high-resolution visible light images of the device surface or environment to identify external damage, wear, cracks, etc.

[0077] Thermal imaging data: Thermal imaging data is used to detect the temperature distribution of equipment and identify hot spots, overheating areas, or potential electrical faults (such as overloads, short circuits).

[0078] Multispectral imaging: In addition to visible light and thermal imaging, multispectral images (such as ultraviolet and infrared) can also be collected to identify equipment anomalies that are otherwise invisible to the naked eye, such as arc discharges, radiation anomalies, etc.

[0079] Equipment operation status data includes:

[0080] Electrical parameters: The sensors collect the equipment's operating current, voltage, power factor and other parameters. These data can reflect whether the equipment's workload is normal, and can be combined with image data to analyze whether there are potential faults.

[0081] Temperature and humidity: Collect temperature and humidity information in the equipment environment to determine whether the equipment working environment meets the standards and whether there are factors such as overheating or humidity that affect the normal operation of the equipment.

[0082] Vibration data: Vibration sensors monitor the vibration frequency and amplitude of the equipment to detect abnormal vibrations. Abnormal vibrations of mechanical equipment are often a precursor to wear, looseness or component failure.

[0083] Location and time data includes:

[0084] GPS / location information: Use GPS or other positioning technologies to record the precise location of inspection equipment to facilitate subsequent location of abnormal equipment and maintenance.

[0085] Timestamp: Record accurate timestamps each time data is collected to ensure the timing of data. This helps to establish historical status data of equipment and analyze the changing trend of equipment status.

[0086] Sound data includes:

[0087] Sound collection: For some equipment, such as mechanical equipment or transformers, the operating sound frequency of the equipment is collected through sound sensors. Abnormal sounds (such as excessive noise, abnormal sound) may indicate potential problems with the equipment.

[0088] Historical data includes:

[0089] Equipment maintenance records: Collect historical maintenance records and combine them with current real-time data to help analyze whether the equipment needs further maintenance.

[0090] Historical anomaly records: Collect historical anomaly data of the device as a reference for subsequent anomaly detection, helping the intelligent analysis module improve detection accuracy.

[0091] The data collected by different sensors can be synchronized to facilitate multimodal data fusion. For example, image data and thermal imaging data can be combined to help more comprehensively analyze the status of the equipment (such as damage to the surface of the equipment due to excessive temperature).

[0092] The data preprocessing steps can be divided into the following key parts:

[0093] 1. Data denoising

[0094] Noise source: During the data collection process, the data may be affected by factors such as changes in ambient light, equipment vibration, and sensor noise itself, resulting in noise.

[0095] Processing method: Use image filtering algorithm to denoise image data. Commonly used methods are:

[0096] Median filtering: Remove salt and pepper noise and retain edge features by taking the median of the pixels around a certain pixel in the image.

[0097] Gaussian filtering: Smooth the image and reduce noise by weighted averaging of the neighborhood values ​​of pixels.

[0098] Bilateral filtering: It not only considers the spatial proximity relationship, but also the difference in pixel values, reducing noise while retaining edge information.

[0099] Multimodal data denoising: For non-image data such as sensor signals, signal processing techniques such as low-pass filtering or wavelet transform are used to eliminate high-frequency noise and retain useful low-frequency signals.

[0100] 2. Image Enhancement

[0101] Brightness and contrast adjustment: Adjust the brightness and contrast of the captured image to ensure that key details in the image can be clearly identified. Common methods include:

[0102] Histogram equalization: By adjusting the image pixel distribution, the overall contrast of the image is made more uniform and the visibility of image details is improved.

[0103] Adaptive Histogram Equalization: Enhances contrast in local areas to improve visibility of local details, especially for scenes with uneven lighting conditions.

[0104] Gamma Correction: Adjusts the gamma value of an image to make an image that is too dark or too bright more clear. Often used to enhance the visibility of damaged or abnormal areas on a device surface.

[0105] 3. Image segmentation and cropping

[0106] Region segmentation: Perform region segmentation on the acquired image or thermal imaging data to extract key components or regions. Common methods include:

[0107] Threshold segmentation: Set a threshold based on the grayscale value of the image to segment out areas of interest, such as the edge of a device or damaged parts of the surface.

[0108] Edge-based segmentation: Use edge detection algorithms (such as Sobel, Canny) to find the contours of the device surface and segment it to ensure that only key information is analyzed.

[0109] Region-based segmentation: Adjacent regions are merged into one region based on the similarity of image pixel values ​​for equipment surface detection or fault identification.

[0110] Cropping: Cropping out non-critical areas in the image, leaving only the critical parts or faulty areas of the device. This not only reduces the amount of computation required for subsequent analysis, but also allows the system to focus on key inspection areas.

[0111] 4. Image format conversion

[0112] Grayscale conversion: Convert color images to grayscale to simplify data processing, especially for edge detection and morphological analysis.

[0113] Pseudo-color processing of thermal imaging data: Thermal imaging data is usually grayscale. Pseudo-color processing can be used to display temperature differences in different colors, making it easier to identify overheated areas or temperature abnormalities in equipment.

[0114] Image resizing: If the image data is too large, the processing speed will be affected. You can adjust the image to a suitable size by image scaling or resampling.

[0115] 5. Multimodal Data Fusion

[0116] Visible light and thermal imaging image fusion: Fusion of visible light images and thermal imaging data of the same location, combined with the appearance and temperature information of the device, can more accurately identify device anomalies. For example, temperature anomalies may cause physical damage to the surface of the device, and multimodal fusion makes it easier to detect potential faults.

[0117] Data alignment and correction: Data collected by different sensors may have spatial or temporal deviations, so these data need to be aligned and corrected. For image data, geometric transformations (such as affine transformations) can be used to align multiple sensor data.

[0118] 6. Data normalization and standardization

[0119] Normalization: Normalize sensor data (such as voltage, current, temperature, etc.) to convert data of different units and ranges to the same scale to facilitate subsequent algorithm processing. For example, normalize voltage data to the [0,1] interval.

[0120] Standardization: Standardize the data and convert it into a standard normal distribution with a mean of 0 and a variance of 1. This is crucial for the training of machine learning and deep learning models, and can accelerate model convergence and improve model stability.

[0121] 7. Data labeling and storage

[0122] Image annotation: Manually or automatically annotate the captured images to mark key components of the equipment or potential abnormal areas. This helps in the subsequent training of machine learning algorithms and can establish a baseline for the health status of the equipment.

[0123] Data storage format: The pre-processed data is stored according to the inspection equipment, time, location and other dimensions to facilitate subsequent query and analysis. Image data can be stored in common formats such as JPEG and PNG, and sensor data can be stored in structured formats such as CSV and JSON.

[0124] 8. Time series data synchronization

[0125] There may be deviations in the data collection time of different sensors, so the data needs to be time-stamped synchronously to ensure that the image, thermal imaging and sensor data can correspond one to one to ensure the accuracy of subsequent analysis results.

[0126] S2: Target recognition, classification and anomaly detection based on convolutional neural networks.

[0127] Furthermore, convolutional neural networks (CNNs) are the core technology for processing image data and are particularly suitable for extracting features from images. The convolution operation is performed pixel by pixel on the input image by moving the convolution kernel (i.e., filter) to extract local features in the image, such as edges, textures, damage or cracks on the surface of the device, etc.

[0128] The convolution kernel performs pixel-by-pixel operations on the input image and can detect different image features that may be related to the surface condition of the device (such as wear, cracks, corrosion, etc.). The convolution operation can effectively reduce the complexity of the image and extract key features to provide input for subsequent classification and anomaly detection.

[0129] It should be noted that the convolution kernel performs pixel-by-pixel operations on the input image to detect different image features. The convolution operation model is expressed as:

[0130]

[0131] Among them, O(i,j) represents the pixel value of the image at position (i,j), (i,j) represents the coordinates of the image pixel, K(m,n) represents the convolution kernel matrix, k represents the radius of the convolution kernel, m and n represent the movement index of the convolution kernel, and I(i+m,j+n) represents the pixel value of the image at position (i+m,j+n).

[0132] It should also be noted that after extracting image features, the convolutional neural network will input these features into the fully connected layer for classification. The Softmax function is the key in the classification task. It converts the score of each category into a probability, thereby obtaining the most likely classification result. In the inspection system, the classification model will determine whether the equipment is in a normal state or has potential faults based on the extracted features, and output the corresponding category (such as equipment intact, worn, cracked, etc.).

[0133] Construct a classification probability model and convert the score of each category into probability, expressed as:

[0134]

[0135] Among them, P(y=i|z) represents the probability of the image being classified as the i-th category, z=[z1,z2,…,z n ] represents the score of each category output by the neural network in the fully connected layer, z i represents the score that the image belongs to category i, and n represents the number of categories.

[0136] The classification probability model is used to output the classification probability of the equipment status and determine the health status or fault type of the equipment.

[0137] Furthermore, the autoencoder is an unsupervised learning model that is mainly used to learn the low-dimensional representation of data and reconstruct it into an output close to the original data through a decoder. In the anomaly detection scenario, the autoencoder will learn how to reconstruct the normal data by training on the data of the normal state of the device. In this way, when the input device data is far from the normal state, the system cannot successfully reconstruct the input data, resulting in a large reconstruction error, and thus judging that the device is in an abnormal state.

[0138] The autoencoder model is constructed as follows:

[0139]

[0140] Where x represents the input data, f(x) represents the encoder function, and g(f(x)) represents the decoder function. Represents the reconstructed data. Represents the difference between the input data and the reconstructed data;

[0141] The autoencoder model is used to learn the data distribution of the device under normal conditions. Through the combination of the encoder and decoder, the system can reconstruct a close output when normal data is input. If the device fails or is in an abnormal state, the reconstruction error will increase significantly, and the system can determine whether there is a potential problem with the device based on the error value.

[0142] It should be noted that in order to quantify the results of anomaly detection, the anomaly scoring model normalizes the reconstruction error of the autoencoder. By setting a threshold, when the value of the loss function exceeds this threshold, the device is considered to have an anomaly. The size of the score can also be used to determine the severity of the anomaly.

[0143] The anomaly scoring model is expressed as:

[0144]

[0145] Among them, S represents the anomaly score, and T represents the set threshold;

[0146] If S>1, it means that the equipment is abnormal, and the value of S is proportional to the severity of the abnormality.

[0147] The abnormality scoring model is used to further refine the determination of the equipment status. By setting a threshold, the system can determine whether the equipment is in an abnormal state. The size of S directly reflects the severity of the abnormality. Therefore, maintenance personnel can decide whether to immediately repair or inspect the equipment based on the score size.

[0148] S3: Use graph theory model and shortest path algorithm to optimize the inspection path.

[0149] Furthermore, different levels of alarm mechanisms can be set;

[0150] Send reminders when there are minor anomalies, and issue alarms and automatically generate fault reports when there are serious anomalies;

[0151] The alarm notifies maintenance personnel via SMS, email, and APP push notifications;

[0152] The system records all detected abnormalities by time and location and saves them as inspection logs. Each fault record should include the equipment number, abnormality type, image data, abnormality degree and equipment operating parameters;

[0153] The system compares and analyzes current fault information with historical data, evaluates the frequency and trend of abnormalities, and provides a basis for equipment maintenance planning;

[0154] The inspection system uses a lossless compression algorithm to compress image and video data, and stores non-critical data in layers to save storage space;

[0155] Design a multi-dimensional database structure based on time, device location, and exception type;

[0156] The data includes original images, preprocessed images, analysis results, and equipment operating parameters;

[0157] A data transmission protocol is used to transmit data from the front-end acquisition device to the remote monitoring center, and the data is encrypted during transmission.

[0158] It should be noted that the graph model is a classic path optimization method, which is applicable to scenarios where there are different inspection costs (such as distance, time, etc.) between multiple devices. During the inspection process, the device location is regarded as a node of the graph, and the edges connecting the nodes represent the distance between the devices or the inspection time. Through the graph theory algorithm, the system can determine the shortest path from the inspection starting point to each device.

[0159] Through the graph theory algorithm, the system determines the shortest path from the inspection starting point to each device. The graph model is expressed as:

[0160] G=(V,E)

[0161] w(e ij )

[0162] Where G represents the graph model of the inspection area, V represents the node set, V = {v1,v2,…,v n}, each node v i represents the location of a device, E represents the edge set, and each edge e ij Represents the device v i and device v j The connected path between them, w(e ij ) represents the weight of the edge, which is the inspection cost between device i and device j.

[0163] It should also be noted that the shortest path model is used to calculate the optimal path from the inspection starting point to each device. Here, the Dijkstra algorithm or the A* algorithm can be used to find the shortest path from the starting point to all target devices by dynamically selecting the device node with the smallest path length.

[0164] The shortest path model is expressed as:

[0165]

[0166] Where L(P) represents the total length of path P, (i,j)∈P represents each pair of devices i and j on path P;

[0167] Initially, set the starting point v s The path length of the device node is 0, and the path length of the device node is infinite;

[0168] Select the node with the current minimum path length, select the node with the minimum path length from the unvisited nodes each time, and update the path lengths of the adjacent nodes;

[0169] Repeat the selection until the shortest paths to all devices are found;

[0170] Build a dynamic inspection frequency optimization model to dynamically adjust the inspection frequency of the equipment and automatically optimize the inspection priority based on the health status of the equipment, expressed as:

[0171]

[0172]

[0173] Among them, P(i) represents the inspection priority of device i, H(i) represents the health index of device i, F represents the optimization objective function, which is the total inspection cost of the system, N represents the total number of devices that need to be inspected, P(i) represents the inspection priority of device i, L(P i ) represents the inspection path length of device i.

[0174] Example 2, an embodiment of the present invention, provides an intelligent patrol monitoring method based on electronic vision. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0175] First, a medium-sized substation was selected as the test area. There were 10 inspection equipment in the station, including transformers, cables, switch cabinets and other equipment. The test lasted for one week. The invention system and the control system inspected the equipment in a fixed period of time (8 am to 4 pm) every day and recorded the equipment status.

[0176] The invention group system uses multimodal sensors, including high-resolution visible light cameras, thermal imagers, audio acquisition equipment, vibration sensors, and ambient temperature and humidity sensors. All data are denoised, filtered, and image enhanced through preprocessing algorithms. After data collection, the convolutional neural network model is used to analyze image features and automatically identify the status of the equipment, including surface damage, overheating, abnormal equipment operation, etc. The system performs anomaly detection through an autoencoder model and adjusts the inspection frequency in real time based on the health index of the equipment.

[0177] The control group system is based on traditional manual inspection technology, using handheld infrared devices and visible light cameras to regularly inspect the equipment and analyze the equipment operation data by regularly reading it. The inspection path is a fixed path and lacks dynamic optimization.

[0178] The invention team's system automatically schedules inspection tasks every day. The system uses graph theory models and shortest path algorithms to generate the optimal inspection path based on inspection data, the historical status of the equipment, and the health index. After each inspection, the system processes multimodal data in real time, uses convolutional neural networks to identify equipment status, and uses autoencoder anomaly detection to analyze the health of equipment operation. For abnormal equipment identified, the system sends an alarm and records the abnormal time and equipment status.

[0179] The control group used manual regular inspections, where staff manually selected equipment paths, took photos of equipment surfaces, and measured equipment temperatures and other operating parameters according to a fixed inspection plan. Due to the lack of multimodal fusion processing, the identification of abnormal equipment mainly relied on manual experience, and the inspection efficiency and path could not be dynamically adjusted based on real-time data.

[0180] Table 1 Experimental data table

[0181]

[0182] From the data in the table, we can see that the invention group is significantly better than the control group in terms of fault identification accuracy, inspection time, abnormality detection frequency and equipment health index.

[0183] In terms of fault identification accuracy, the fault identification accuracy of the invention group system equipment 1 and equipment 2 reached 98% and 96% respectively, while the fault identification accuracy of the control group equipment 1 and equipment 2 was 85% and 80% respectively. In terms of inspection time, the inspection time of the invention group system was shorter, with the inspection time of equipment 1 and equipment 2 being 45 minutes and 43 minutes respectively, while the inspection time of the control group was 70 minutes and 68 minutes respectively. In terms of abnormal detection frequency, the abnormal detection frequency of the invention group system equipment 1 and equipment 2 was 3 times / day and 2 times / day respectively, while the abnormal detection frequency of the control group was only 1 time / day. In terms of health index, the health index of the invention group system equipment 1 and equipment 2 was 0.75 and 0.80 respectively, while the health index of the control group equipment 1 and equipment 2 was lower, at 0.60 and 0.65 respectively.

[0184] In summary, the present invention achieves better results in terms of fault identification accuracy, inspection efficiency and adaptability.

[0185] Example 3, reference Figure 2 , which is an embodiment of the present invention, provides an intelligent patrol monitoring system based on electronic vision, including an initialization module, a data preprocessing module, an intelligent analysis and target recognition module, and a patrol optimization module.

[0186] The initialization module is used to collect multimodal inspection data, the data preprocessing module is used for analysis and target recognition, the intelligent analysis and target recognition module is used to perform target detection and classification on the preprocessed image data, and the inspection optimization module is used to generate the optimal inspection path.

[0187] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0188] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0189] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0190] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0191] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent inspection and monitoring method based on electronic vision, characterized in that: include: Collect multi-modal inspection data and pre-process the data; Target recognition, classification and anomaly detection based on convolutional neural networks; Graph theory model and shortest path algorithm are used to optimize the inspection path.

2. The electronic vision-based intelligent inspection and monitoring method according to claim 1, characterized in that: The collecting of multimodal inspection data and preprocessing of the data includes collecting image data, equipment operation status data, location and time data, sound data and historical data, and performing multimodal data fusion; Image data includes visible light images, thermal imaging data, and multispectral images; Equipment operating status data includes electrical parameters, temperature and humidity, and vibration data; Location and time data includes GPS location information and timestamp; The sound data includes the operating audio frequency; Historical data includes equipment maintenance records and historical abnormality records; The data collected by different sensors are synchronized to perform multimodal data fusion.

3. The electronic vision-based intelligent inspection and monitoring method according to claim 2, characterized in that: The collecting of multimodal inspection data and preprocessing of the data includes the generation of noise during the data collection process due to the influence of ambient light changes, equipment vibration, and sensor noise factors; Using image filtering algorithms to denoise image data includes: By taking the median of the pixels around the pixel in the image, noise is removed and edge features are retained; By weighted averaging the neighborhood values ​​of pixels, the image is smoothed and noise is removed; Considering the spatial proximity and the difference of pixel values, noise is removed while retaining edge information.

4. The electronic vision-based intelligent inspection and monitoring method according to claim 3, characterized in that: The target recognition classification and anomaly detection based on the convolutional neural network includes performing pixel-by-pixel calculations on the input image by moving the convolution kernel to extract local features in the image, including edges, textures, and damage or cracks on the surface of the device; The convolution kernel performs pixel-by-pixel operations on the input image to detect different image features. The convolution operation model is expressed as: Among them, O(i,j) represents the pixel value of the image at position (i,j), (i,j) represents the coordinates of the image pixel, K(m,n) represents the convolution kernel matrix, k represents the radius of the convolution kernel, m and n represent the movement index of the convolution kernel, and I(i+m,j+n) represents the pixel value of the image at position (i+m,j+n); After extracting image features, the convolutional neural network inputs the features into the fully connected layer for classification; Construct a classification probability model and convert the score of each category into probability, expressed as: Among them, P(y=i|z) represents the probability of the image being classified as the i-th category, z=[z1,z2,…,z n ] represents the score of each category output by the neural network in the fully connected layer, z i represents the score of the image belonging to category i, and n represents the number of categories; The classification probability model is used to output the classification probability of the equipment status and determine the health status or fault type of the equipment.

5. The electronic vision-based intelligent inspection and monitoring method according to claim 4, characterized in that: The target recognition classification and anomaly detection based on the convolutional neural network includes using an autoencoder to perform anomaly detection, and determining whether the current device state is abnormal by performing reconstruction error analysis on the device data; The autoencoder model is constructed as follows: Where x represents the input data, f(x) represents the encoder function, and g(f(x)) represents the decoder function. Represents the reconstructed data. Represents the difference between the input data and the reconstructed data; By minimizing The autoencoder learns the normal state of the device and determines whether there is an abnormality in subsequent detection; To quantify the results of anomaly detection, the anomaly scoring model normalizes the reconstruction error of the autoencoder. By setting a threshold, when the value of the loss function exceeds the threshold, the device is considered to have an anomaly. The size of the score determines the severity of the anomaly. The anomaly scoring model is expressed as: Among them, S represents the anomaly score, and T represents the set threshold; If S>1, it means that the equipment is abnormal, and the value of S is proportional to the severity of the abnormality.

6. The electronic vision-based intelligent inspection and monitoring method according to claim 5, characterized in that: The use of graph theory models and shortest path algorithms to optimize inspection paths includes setting alarm mechanisms at different levels; Send reminders when there are minor anomalies, and issue alarms and automatically generate fault reports when there are serious anomalies; The alarm notifies maintenance personnel via SMS, email, and APP push notifications; The system records all detected abnormalities by time and location and saves them as inspection logs. Each fault record should include the equipment number, abnormality type, image data, abnormality degree and equipment operating parameters; The system compares and analyzes current fault information with historical data, evaluates the frequency and trend of abnormalities, and provides a basis for equipment maintenance planning; The inspection system uses a lossless compression algorithm to compress image and video data, and stores non-critical data in layers to save storage space; Design a multi-dimensional database structure based on time, device location, and exception type; The data includes original images, pre-processed images, analysis results, and equipment operating parameters; A data transmission protocol is used to transmit data from the front-end acquisition device to the remote monitoring center, and the data is encrypted during transmission.

7. The electronic vision-based intelligent inspection and monitoring method according to claim 6, characterized in that: The use of graph theory model and shortest path algorithm to optimize the inspection path includes that during the inspection process, the device location is regarded as a node of the graph, and the edges connecting the nodes represent the distance between the devices and the inspection time; Through the graph theory algorithm, the system determines the shortest path from the inspection starting point to each device. The graph model is expressed as: G=(V,E) w(e ij ) Where G represents the graph model of the inspection area, V represents the node set, V = {v1,v2,…,v n }, each node v i represents the location of a device, E represents the edge set, and each edge e ij Represents the device v i and device v j The connected path between them, w(e ij ) represents the weight of the edge, which is the inspection cost between device i and device j; By dynamically selecting the device node with the smallest path length, the shortest path from the starting point to all target devices is found. The shortest path model is expressed as: Where L(P) represents the total length of path P, (i,j)∈P represents each pair of devices i and j on path P; Initially, set the starting point v s The path length of the device node is 0, and the path length of the device node is infinite; Select the node with the current minimum path length, select the node with the minimum path length from the unvisited nodes each time, and update the path lengths of the adjacent nodes; Repeat the selection until the shortest paths to all devices are found; Build a dynamic inspection frequency optimization model to dynamically adjust the inspection frequency of the equipment and automatically optimize the inspection priority based on the health status of the equipment, expressed as: Among them, P(i) represents the inspection priority of device i, H(i) represents the health index of device i, F represents the optimization objective function, which is the total inspection cost of the system, N represents the total number of devices that need to be inspected, P(i) represents the inspection priority of device i, L(P i ) represents the inspection path length of device i; Priority is inversely proportional to the health index; The health index is proportional to the device status.

8. A system using the electronic vision-based intelligent inspection and monitoring method as claimed in any one of claims 1 to 7, characterized in that: Including initialization module, data preprocessing module, intelligent analysis and target recognition module, inspection optimization module; The initialization module is used to collect multi-modal inspection data; The data preprocessing module is used for analysis and target recognition; The intelligent analysis and target recognition module is used to perform target detection and classification on the preprocessed image data; The inspection optimization module is used to generate an optimal inspection path.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the intelligent inspection and monitoring method based on electronic vision described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent inspection and monitoring method based on electronic vision described in any one of claims 1 to 7 are implemented.

Images