Equipment anomaly detection method and device, equipment and storage medium

By adopting a two-stage method on the factory site, using image acquisition equipment and pre-created template image library for matching and segmentation, combining depth-separable convolution modules and hash encoding, the high leakage of leakage with very few samples is achieved Quality identification solves the time-consuming and laborious problem of data collection and labeling in the prior art, and improves identification efficiency and accuracy.

CN118942042APending Publication Date: 2024-11-12HANGZHOU HOLLYSYS AUTOMATION
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411226417.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When the factory site identification equipment leaks, it requires a lot of manpower and time to collect data and image annotation. Especially when there are very few samples, it is difficult to achieve high-quality identification results, and the model effects are very different, so it is impossible to Meet expected algorithm performance.

Method used

Using a two-stage method, firstly, the image to be detected is obtained through the image acquisition device to match the pre-created template image library. If the match is successful, segmentation and cropping are performed to obtain the cropped image, and the pre-trained feature extraction model is input for feature extraction. , determine the exception category based on the similarity measurement and target feature library, and alarm is carried out. This method does not require training the model, reduces manpower and material resources, and uses deep separable convolution modules and hash encoding for feature extraction and matching.

Benefits of technology

With the support of very few samples, accurate identification of leaks is achieved, which reduces the time-consuming and labor-intensive data collection and labeling work, improves identification efficiency and accuracy, and overcomes the problem of model effects differences in the existing technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942042A_ABST
    Figure CN118942042A_ABST
Patent Text Reader

Abstract

The invention discloses an equipment anomaly detection method and device, equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a to-be-detected region image of target equipment, and carrying out the matching of the to-be-detected region image with a template image library; if the matching is successful, segmenting the to-be-detected area image and the successfully matched template image to obtain a first segmentation result set and a second segmentation result set; comparing the first segmentation result set with the second segmentation result set, determining a target picture and an abnormal detection position of the to-be-detected area image, and cutting the target picture based on the abnormal detection position to obtain a cut picture; inputting the cut picture into a feature extraction model for feature extraction, obtaining extracted features, determining an abnormal category of the extracted features based on similarity measurement and a target feature library, outputting the abnormal category and an abnormal detection position, and giving an alarm to complete abnormal detection of the target equipment. Therefore, the leakage and leakage of the equipment can be identified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a device anomaly detection method, device, equipment and storage medium. Background Art

[0002] Leakage is common in factory production, mainly because factories have a wide variety of special raw materials and complex process flows. Leakage often occurs only in a small local area at the beginning, but the consequences are very serious, sometimes even catastrophic. Some major accidents are caused by the leakage of flammable and explosive materials, causing fires and explosions. Civilian gas and liquefied petroleum gas tanks have similar accidents. The leakage of toxic substances causes poisoning of personnel, and in serious cases, it also affects the surrounding environment.

[0003] Early warning is essential for on-site management of equipment pipelines such as "leakage, bubbling, dripping, and leakage" in factories. The recognition and warning system is based on artificial intelligence analysis technology to automatically identify whether there is gas or liquid leakage in mechanical pipelines and equipment in the monitoring video. If a leak is detected, it will be immediately fed back to the back-end personnel for timely processing. The recognition and warning system can effectively prevent and reduce the occurrence of safety accidents and improve the efficiency of manual inspections. The intelligent recognition and warning system is safer, more convenient, time-saving, and more reliable than manual inspections. The recognition of leakage of equipment mainly uses computer vision technology. Based on a powerful convolutional neural network, it automatically learns abstract features from big data, has stronger expression and distinction capabilities, and can more accurately represent the real appearance features. At the same time, the convolutional neural network model also has a strong nonlinear mapping ability, which can effectively realize the mapping from image features to equipment environment. Image data is collected by the camera, recognized by the deep learning model, and finally the recognition results are pushed to the on-site personnel. Leak identification faces many challenges: first, the factory environment is complex, the lighting conditions are weak, and there are obstructions; second, there are a large number of devices, and various scenes need to be monitored simultaneously, requiring the system to have high-performance computing capabilities; finally, leaks come in various physical states and shapes, which places high demands on the feature extraction capabilities of the identification model.

[0004] Most of the existing solutions rely on computer vision recognition technology. For leakage that may occur in the factory, data is first collected, and then data is labeled according to different tasks, such as detection or segmentation. Then, a suitable model framework is selected for training to learn the characteristics of equipment leakage, and then the model is built, and a suitable neural network is selected for training and tuning. Finally, the trained model is deployed. When performing recognition, the system inputs a picture of the engineering site. The image goes through preprocessing, model reasoning, post-processing and other modules, and finally the recognition result is output and pushed to the system, and fed back to the back-end staff for timely processing to complete the entire visual recognition process. However, the existing solutions are mainly based on model training of massive image data. Due to the complex factory site environment, the leakage has different forms, and different raw materials have different colors, shapes, and appearances, which makes the collection of preliminary data a big problem. After the data collection is completed, image labeling is required, especially the labeling of segmentation data, which requires a lot of manpower and time costs. A small amount of sample labeling cannot achieve the expected algorithm performance, and the quality of the labeled image will also have a great impact on the algorithm effect of the model. Therefore, how to complete the recognition of leakage with high quality with the support of very few samples is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for detecting equipment anomalies, which can accurately identify leakage with the support of very few samples, and no longer rely on the collection of factory site data. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a device anomaly detection method, comprising:

[0007] Using an image acquisition device to obtain an image of the area to be detected of the target device, and matching the image of the area to be detected with a pre-created template image library;

[0008] If the match is successful, the image of the area to be detected and the template image that has been matched are segmented based on a preset segmentation model to obtain a corresponding first segmentation result set and a second segmentation result set;

[0009] Comparing the first segmentation result set with the second segmentation result set to determine a target image with an abnormality and an abnormality detection position corresponding to the image of the area to be detected, and cropping the target image based on the abnormality detection position to obtain a cropped image;

[0010] The cropped image is input into a pre-trained feature extraction model for feature extraction, the extracted features are obtained, and the abnormality category corresponding to the extracted features is determined based on the similarity measurement and the target feature library, the abnormality category and the abnormality detection position are output, and a corresponding alarm is issued to complete the abnormality detection of the target device.

[0011] Optionally, after matching the image of the area to be detected with a pre-created template image library, the method further includes:

[0012] If the image of the area to be detected fails to match the pre-created template image library, the target template image corresponding to the image of the area to be detected is added to the template image library, and the image of the area to be detected is matched with the pre-created template image library again.

[0013] Optionally, after comparing the first segmentation result set with the second segmentation result set, the method further includes:

[0014] If there is no abnormality in the image of the area to be detected, the process ends directly.

[0015] Optionally, before inputting the cropped image into a pre-trained feature extraction model for feature extraction, the method further includes:

[0016] An initial feature extraction model is constructed based on several depth-wise separable convolution modules; wherein the depth-wise separable convolution modules include depth-wise convolution, point-wise convolution, batch normalization and H-Swish activation function;

[0017] The initial feature extraction model is trained by a loss function based on metric learning to obtain a trained feature extraction model.

[0018] Optionally, before determining the abnormal category corresponding to the extracted feature based on the similarity measurement and the target feature library, the method further includes:

[0019] Collecting abnormal images of leakages existing in the production environment corresponding to the target device, and extracting features from the abnormal images using the feature extraction model to obtain target features;

[0020] Hash coding is used to convert each of the floating-point target features into binary features, and the target feature library is constructed based on the binary features.

[0021] Optionally, determining the abnormality category corresponding to the extracted feature based on the similarity measurement and the target feature library includes:

[0022] Determine the similarity measure between the extracted features and each target feature in the target feature library based on the Hamming distance;

[0023] Sorting the similarity measures to determine the maximum similarity measure;

[0024] Comparing the maximum similarity measure with a preset threshold;

[0025] If the maximum similarity measure is greater than or equal to the preset threshold, the image category corresponding to the maximum similarity measure is determined as the abnormal category corresponding to the extracted feature.

[0026] Optionally, after comparing the maximum similarity measure with a preset threshold, the method further includes:

[0027] If the maximum similarity measure is less than the preset threshold, determining the picture category corresponding to the maximum similarity measure as an unknown category;

[0028] The image category corresponding to the maximum similarity measure is added to the target feature library, and the next round of device anomaly detection is performed.

[0029] In a second aspect, the present application discloses a device for detecting anomalies of equipment, comprising:

[0030] A matching module, used to obtain an image of the area to be detected of the target device by using an image acquisition device, and match the image of the area to be detected with a pre-created template image library;

[0031] An image segmentation module, for segmenting the image of the area to be detected and the template image that has been matched successfully based on a preset segmentation model if the match is successful, so as to obtain a corresponding first segmentation result set and a second segmentation result set;

[0032] An image cropping module is used to compare the first segmentation result set and the second segmentation result set to determine a target image with an abnormality and an abnormality detection position corresponding to the image of the area to be detected, and crop the target image based on the abnormality detection position to obtain a cropped image;

[0033] The anomaly detection module is used to input the cropped image into a pre-trained feature extraction model for feature extraction, obtain the extracted features, and determine the anomaly category corresponding to the extracted features based on a similarity metric and a target feature library, output the anomaly category and the anomaly detection position, and issue a corresponding alarm to complete the anomaly detection of the target device.

[0034] In a third aspect, the present application discloses an electronic device, comprising:

[0035] Memory, used to store computer programs;

[0036] A processor is used to execute the computer program to implement the device abnormality detection method as described above.

[0037] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned device anomaly detection method.

[0038] When detecting an abnormality in a device, the present application first uses an image acquisition device to obtain an image of an area to be detected of a target device, and matches the image of the area to be detected with a pre-created template image library; if the match is successful, the image of the area to be detected and the successfully matched template image are segmented respectively based on a preset segmentation model to obtain a corresponding first segmentation result set and a second segmentation result set; then the first segmentation result set and the second segmentation result set are compared to determine a target image with an abnormality and an abnormality detection position corresponding to the image of the area to be detected, and the target image is cropped based on the abnormality detection position to obtain a cropped image; finally, the cropped image is input into a pre-trained feature extraction model for feature extraction to obtain extracted features, and the abnormality category corresponding to the extracted features is determined based on a similarity metric and a target feature library, the abnormality category and the abnormality detection position are output, and a corresponding alarm is issued to complete the abnormality detection of the target device. It can be seen that the present application obtains the image of the area to be detected through the image acquisition device, performs scene template matching, and if the match is successful, the image to be detected and the template image are input into the visual large model to obtain the segmentation results respectively, and the coordinates of the abnormal detection position are obtained after parameter screening. If the recognition is successful, the coordinates are cropped and input into the feature extraction module to obtain the image features of the object to be identified, and the pre-created leakage feature library is used for feature search to complete category recognition, and finally the identified position and category information are returned to the system for alarm. In this way, the present application adopts a two-stage method to identify leakage of equipment, which can effectively detect its location information without training. In the second stage, the feature matching algorithm is used for classification and recognition, which reduces manpower and material resources. With the support of very few samples, the identification of leakage can be completed with high quality, effectively overcoming the problems of difficult data collection on-site in factories, time-consuming and labor-intensive image annotation, and large differences in model effects in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A flow chart of a device abnormality detection method disclosed in this application;

[0041] Figure 2 A schematic diagram of a feature extraction model training disclosed in this application;

[0042] Figure 3 A schematic diagram of feature extraction disclosed in this application;

[0043] Figure 4 A flowchart of a specific device abnormality detection method disclosed in this application;

[0044] Figure 5 A schematic diagram of image recognition disclosed in this application;

[0045] Figure 6 A flowchart of a specific device abnormality detection method disclosed in this application;

[0046] Figure 7 This is a schematic diagram of the structure of a device abnormality detection device disclosed in this application;

[0047] Figure 8 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Most of the current technical solutions are deep learning training methods based on large amounts of data collection and annotation, which require a lot of manpower and material resources to mark image information, and then conduct model training, parameter tuning and other processes. It takes a long time and the recognition effect is not obvious. It requires data support, and it is difficult to achieve the expected accuracy with small sample data. In order to solve the above technical problems, this application discloses a method, device, equipment and storage medium for equipment anomaly detection, which can achieve accurate identification of leakage with the support of very few samples, and no longer rely on the collection of factory site data.

[0050] See also Figure 1 As shown, an embodiment of the present invention discloses a device abnormality detection method, including:

[0051] Step S11: using an image acquisition device to obtain an image of the area to be detected of the target device, and matching the image of the area to be detected with a pre-created template image library.

[0052] In this embodiment, when performing equipment abnormality detection, the image of the area to be identified is first obtained through the image acquisition device, and then matched with the pre-established scene templates of different time periods (this template should be a normal device image without leakage) to determine the scene of the current image acquisition. That is, the image of the area to be detected is matched with the pre-created template image library. After matching the image of the area to be detected with the pre-created template image library, if the image of the area to be detected fails to match the pre-created template image library, the target template image corresponding to the image of the area to be detected is added to the template image library, and the image of the area to be detected is matched with the pre-created template image library again.

[0053] Step S12: If the match is successful, the image of the area to be detected and the successfully matched template image are segmented based on a preset segmentation model to obtain a corresponding first segmentation result set and a second segmentation result set.

[0054] In this embodiment, if the image of the area to be detected successfully matches the pre-created template image library, the area to be detected is set according to the acquired image, and it is cropped and input into the visual segmentation model. The segmentation model is used to perform reasoning processing on the current target area image to obtain a segmentation result set of the image to be identified, and the template image that has been successfully matched is also segmented to obtain a corresponding target segmentation result set, that is, the corresponding first segmentation result set and second segmentation result set are obtained respectively.

[0055] Step S13: compare the first segmentation result set and the second segmentation result set to determine the target image with abnormalities and the abnormality detection position corresponding to the image of the area to be detected, and crop the target image based on the abnormality detection position to obtain a cropped image.

[0056] In this embodiment, after obtaining the first segmentation result set and the second segmentation result set, the two segmentation sets are compared in a weighted manner, and filtered and screened by preset parameters to obtain the final target detection position, that is, the target image with abnormalities and the abnormality detection position corresponding to the image of the area to be detected are determined, and this position is at the pixel level. After that, the target image can be cropped according to the abnormal detection position to obtain the cropped image and determine the area where the abnormality occurs. In addition, after comparing the first segmentation result set and the second segmentation result set, if there is no abnormality in the image of the area to be detected, the process ends directly. In other words, the image of the area to be detected is normal, and no further abnormality detection is required, and the current detection can be ended directly.

[0057] Step S14: input the cropped image into a pre-trained feature extraction model for feature extraction, obtain the extracted features, and determine the abnormality category corresponding to the extracted features based on the similarity measurement and the target feature library, output the abnormality category and the abnormality detection position, and make a corresponding alarm to complete the abnormality detection of the target device.

[0058] In this embodiment, before the cropped image is input into a pre-trained feature extraction model for feature extraction, that is, before feature extraction is performed on the cropped image, the model needs to be trained in advance. Feature extraction is a key step in image recognition. The feature extraction model can convert the input image into a feature vector of fixed dimension, which will be used in the subsequent vector retrieval step. In order to meet the needs of real-time reasoning in industrial scenarios, this application uses a lightweight network backbone. Figure 2 As shown, the network backbone is composed of multiple depth-separable convolution modules, which include depth-wise convolution, point-by-point convolution, batch normalization and H-Swish activation function. Depth-wise separable convolution can greatly improve computational efficiency, while reducing the amount of network computation, it can also ensure the accuracy of the network. Therefore, when training the model, the present application first constructs an initial feature extraction model based on several depth-wise separable convolution modules; wherein the depth-wise separable convolution module includes depth-wise convolution, point-by-point convolution, batch normalization and H-Swish activation function; and then trains the initial feature extraction model through a loss function based on metric learning to obtain a trained feature extraction model. It is important to know that the activation function of the backbone network uses H-Swish, which can improve the performance of the model. Compared with ReLU (Rectified Linear Unit) and Sigmoid function, H-Swish function helps alleviate the problem of gradient disappearance during back propagation. Compared with Swish function, H-Swish replaces Sigmoid function with piecewise linear function, which makes calculation simpler and reduces calculation cost. Due to these advantages, H-Swish activation function is a better choice in industrial scenarios that require efficient calculation and fast model training. Among them, the expression formula of H-Swish function is:

[0059] ;

[0060] x is the input value of the activation function. ReLU6 is a variant of the rectified linear unit ReLU. The formula is as follows:

[0061] ;

[0062] After determining the backbone network model and activation function, it is necessary to use the metric learning method to guide the network to learn good features during the training process. The goal of metric learning is to learn a transformation function to map data points from the original vector space to a new vector space, in which the distance between similar points is closer and the distance between dissimilar points is farther, that is, the feature distance of similar images is closer and the feature distance of dissimilar images is farther, so as to train the feature extraction ability of the deep convolutional neural network. The metric learning method based on classification labels learns effective feature representation by classifying each sample into the correct category. The explicit label of each sample is required to participate in the loss calculation during the learning process. In this way, the present application uses the L2-Softmax loss function, which can add a constraint to the feature during the training process. By making the features of the same category closer and the features of different categories farther apart in the normalized space, the verification signal of the feature is enhanced. In addition, the L2-Softmax loss function does not use the collaborative supervision loss used by other methods, which is convenient for engineering implementation and converges quickly during training. The formula is expressed as follows:

[0063] ;

[0064] Where M is the batch size for training, x i is the i-th image in M, f(x i ) is the output of the corresponding feature layer of the convolutional neural network, y i is the corresponding category label, W and b are the weights and biases of the last layer of the classifier network, and C is the number of categories. i ) Add a 𝐿2 constraint:

[0065] ;

[0066] in, is the constraint value.

[0067] After determining the backbone network model and the loss function of metric learning, you can start training the model. First, use the pre-trained model trained on a large-scale natural image dataset to initialize the weights of the network model, then use the collated public dataset to train the model. After the model converges, remove the last layer of the network, and use the remaining network structure to extract image features in the inference stage. Figure 2The diagram of model training and inference shown in the figure inputs pictures related to equipment leakage, such as pictures containing water stains, oil stains, water mist, sparks, etc., into the network for training. In the training phase, the entire network includes the input layer, network backbone, global average pooling layer, feature description layer, fully connected layer, and L2-Softmax loss function. In the inference phase, the feature extraction network part is used, the fully connected layer and loss function are removed, and the feature vector of the input picture is obtained from the feature description layer.

[0068] After the above process, the feature extraction model has been trained. The cropped image can be input into the pre-trained feature extraction model for feature extraction to obtain the extracted features. Then, the abnormal category corresponding to the extracted features is determined based on the similarity measurement and the target feature library. However, before that, it is necessary to collect abnormal images of leakage in the production environment corresponding to the target equipment, and use the feature extraction model to extract features from the abnormal images to obtain each target feature; use hash coding to convert each floating-point target feature into a binary feature, and build a target feature library based on the binary feature. In detail, it is necessary to build a leakage library in advance, and pre-collect the leakage situations that often occur in the on-site production environment. For example, the library should contain abnormal pictures such as water stains, oil stains, water mist, sparks, etc. The number does not need to be too many, 5-10 pictures for each category. All pictures in the picture library (including abnormal pictures such as water stains, oil stains, water mist, sparks, etc.) are feature extracted through the feature extraction model constructed above, and these features are stored in the vector library, such as Figure 3 As shown. First, the features of the leakage image library are extracted and stored in the vector library, then the feature vector is extracted from the target image, and then the feature vector of the target image is searched in the vector library, and finally the most similar category is matched. When searching, the retrieval algorithm uses HNSW (Hierarchical Navigable Small World). This method is a graph indexing method. When establishing the index, it is divided into different layers, so the retrieval accuracy is high and the speed is fast. HNSW adopts a similar idea to the skip table, skipping a large number of points far from the target point at the high level, so as to quickly locate the points closer to the target, thereby narrowing the search range. HNSW uses heuristic search to select connected neighbor nodes when composing the graph, so as to prevent the situation of disconnected graphs. Among them, the extracted features are floating point data types. When there are more features stored in the offline feature library, it takes up a large storage space, and the retrieval process will also be slow. In order to solve this problem, the present application uses hash coding to convert the features from floating point type into binary features represented by 0 or 1. This processing can not only reduce the storage space, but also greatly speed up the retrieval speed.

[0069] After that, the feature vector is searched and matched with the constructed leakage vector database. The abnormal category corresponding to the extracted feature is determined based on the similarity measurement score, and the most similar category is matched. In this process, the similarity measurement between the extracted feature and each target feature in the target feature library is determined based on the Hamming distance; each similarity measurement is sorted to determine the maximum similarity measurement; the maximum similarity measurement is compared with the preset threshold; if the maximum similarity measurement is greater than or equal to the preset threshold, the image category corresponding to the maximum similarity measurement is determined as the abnormal category corresponding to the extracted feature. Through the two-stage model recognition, the position coordinates and category of the leakage abnormal area are detected, thereby completing the recognition of the entire leakage process. If there is an abnormal scene in the area to be detected, the result is saved and an alarm is sent to the system. If not, the next round of recognition is carried out. That is to say, if the maximum similarity measurement is less than the preset threshold, the image category corresponding to the maximum similarity measurement is determined as an unknown category; the image category corresponding to the maximum similarity measurement is added to the target feature library, the target feature library is expanded, and then the next round of equipment abnormality detection is carried out.

[0070] In general, when performing device anomaly detection, the application Figure 4 As shown, the present application first obtains the image of the area to be detected through the image acquisition device, and performs scene template matching. If the match is successful, the image to be detected and the template image are input into the visual large model, and the segmentation results are obtained through the visual large model SAM (segment anything model). After parameter screening, the abnormal detection position coordinates are obtained, that is, the position of the object in the abnormal scene is detected. Then, the detected image is cropped according to the position coordinates and input into the feature extraction module to obtain the image features of the object to be identified, and the feature search is performed with the pre-created leakage feature library to complete the category recognition. In other words, in the second stage, it is passed to the feature matching model, and through feature extraction, feature vectorization, search and matching and other technologies, without the need for data labeling, model training and other processes, the location and category of equipment leakage can be effectively identified, including but not limited to general liquids (water stains, oil stains, etc.), coal ash, water mist, electric sparks, etc. Input an image to be identified such as Figure 5 (1) The visual large model SAM performs pixel-level segmentation, and the intermediate results are as follows Figure 5 (2) as shown in part, and then Figure 5 As shown in (3), the segmentation results are cropped and searched and matched after passing through the feature extraction network. The two segmentation results are searched separately to obtain the measurement results. The order is arranged from left to right according to the measurement score from small to large, and finally the largest measurement score is selected. The classification result is as follows Figure 5As shown in (4), the two categories are shui and sandbag. These categories are feature library categories created in advance, and their confidence levels are 0.95 for shui and 0.94 for sandbag respectively.

[0071] In summary, when detecting an abnormality in a device, the present application first uses an image acquisition device to obtain an image of an area to be detected of a target device, and matches the image of the area to be detected with a pre-created template image library; if the match is successful, the image of the area to be detected and the successfully matched template image are segmented respectively based on a preset segmentation model to obtain a corresponding first segmentation result set and a second segmentation result set; then the first segmentation result set and the second segmentation result set are compared to determine the target image with an abnormality and the abnormality detection position corresponding to the image of the area to be detected, and the target image is cropped based on the abnormality detection position to obtain a cropped image; finally, the cropped image is input into a pre-trained feature extraction model for feature extraction to obtain the extracted features, and the abnormality category corresponding to the extracted features is determined based on a similarity metric and a target feature library, the abnormality category and the abnormality detection position are output, and a corresponding alarm is issued to complete the abnormality detection of the target device. It can be seen that the present application obtains the image of the area to be detected through the image acquisition device, performs scene template matching, and if the match is successful, the image to be detected and the template image are input into the visual large model to obtain the segmentation results respectively, and the coordinates of the abnormal detection position are obtained after parameter screening. If the recognition is successful, the coordinates are cropped and input into the feature extraction module to obtain the image features of the object to be identified, and the pre-created leakage feature library is used for feature search to complete category recognition, and finally the identified position and category information are returned to the system for alarm. In this way, the present application adopts a two-stage method to identify leakage of equipment, which can effectively detect its location information without training. In the second stage, the feature matching algorithm is used for classification and recognition, which reduces manpower and material resources. With the support of very few samples, the identification of leakage can be completed with high quality, effectively overcoming the problems of difficult data collection on-site in factories, time-consuming and labor-intensive image annotation, and large differences in model effects in the prior art.

[0072] Based on the previous embodiment, it can be known that when determining the abnormal category, the present application determines the abnormal category corresponding to the extracted feature based on the similarity measurement and the target feature library. The most similar category can be matched in the target feature library. The specific determination process of the abnormal category will be described in detail below.

[0073] See also Figure 6 As shown, the embodiment of the present invention discloses a specific device abnormality detection method, including:

[0074] Step S21: input the cropped image into a pre-trained feature extraction model to extract features, obtain extracted features, and determine the similarity measure between the extracted features and each target feature in the target feature library based on the Hamming distance.

[0075] In this embodiment, the identified target image is cropped according to the position coordinates and input into the feature extraction model to obtain the feature vector. After that, in the established vector library, i.e., the target feature library, the extracted features are calculated with the Hamming distance of the feature vectors of all the query vectors in the library to obtain the similarity ranking. In a given set of vectors {x1, ..., xn}, an n-dimensional data is constructed. When a new vector x is input, an n-dimensional operation is performed in the vector group:

[0076] ;

[0077] in is a similarity measurement method, argmin is an index search and match operation for a given vector. The similarity measurement uses the structural Hamming distance. For a fixed length vector n, the Hamming distance is a measurement on the length vector space, which can be used to calculate the similarity of two vectors of equal length. The number of bits with different code values ​​on the corresponding bits is defined as the Hamming distance between the two code words. That is, the calculation of the similarity measurement is performed through the Hamming distance:

[0078] ;

[0079] Here i=0, 1, ..n-1, represents the Hamming distance between two vectors, It represents XOR. The specific calculation method is: compare whether each bit of the vector is the same. If it is different, the Hamming distance is increased by 1, so that the Hamming distance can be obtained. The higher the vector similarity, the smaller the corresponding Hamming distance.

[0080] Step S22: sort the similarity metrics, determine the maximum similarity metric, and compare the maximum similarity metric with a preset threshold.

[0081] In this embodiment, after calculating the similarity metric ranking by the Hamming distance, each similarity metric is ranked, the maximum similarity metric is determined, and the maximum similarity metric is compared with a preset threshold, that is, the metric corresponding to the image with the maximum similarity is compared with the threshold.

[0082] Step S33: If the maximum similarity measure is greater than or equal to the preset threshold, the image category corresponding to the maximum similarity measure is determined as the abnormal category corresponding to the extracted feature.

[0083] In this embodiment, if the maximum similarity measure is greater than or equal to a preset threshold, the image category corresponding to the maximum similarity measure is determined as the abnormal category corresponding to the extracted feature, which means that the image category corresponding to the maximum similarity measure is the abnormal category of the extracted feature, thereby completing the detection of the abnormal category.

[0084] It can be seen that the present application adopts a feature matching algorithm for classification and identification, which reduces manpower and material resources, and can complete the identification of leaks with high quality with the support of very few samples.

[0085] See also Figure 7 As shown, an embodiment of the present invention discloses a device for detecting anomalies of equipment, including:

[0086] A matching module 11 is used to obtain an image of a region to be detected of a target device by using an image acquisition device, and match the image of the region to be detected with a pre-created template image library;

[0087] An image segmentation module 12 is used to segment the image of the area to be detected and the template image that has been matched successfully based on a preset segmentation model if the match is successful, so as to obtain a corresponding first segmentation result set and a second segmentation result set;

[0088] The image cropping module 13 is used to compare the first segmentation result set and the second segmentation result set to determine the target image with abnormalities and the abnormality detection position corresponding to the image of the area to be detected, and crop the target image based on the abnormality detection position to obtain a cropped image;

[0089] The anomaly detection module 14 is used to input the cropped image into a pre-trained feature extraction model for feature extraction, obtain the extracted features, and determine the anomaly category corresponding to the extracted features based on a similarity metric and a target feature library, output the anomaly category and the anomaly detection position, and issue a corresponding alarm to complete the anomaly detection of the target device.

[0090] It can be seen that the present application obtains the image of the area to be detected through the image acquisition device, performs scene template matching, and if the match is successful, the image to be detected and the template image are input into the visual large model to obtain the segmentation results respectively, and the coordinates of the abnormal detection position are obtained after parameter screening. If the recognition is successful, the coordinates are cropped and input into the feature extraction module to obtain the image features of the object to be identified, and the pre-created leakage feature library is used for feature search to complete category recognition, and finally the identified position and category information are returned to the system for alarm. In this way, the present application adopts a two-stage method to identify leakage of equipment, which can effectively detect its location information without training. In the second stage, the feature matching algorithm is used for classification and recognition, which reduces manpower and material resources. With the support of very few samples, the identification of leakage can be completed with high quality, effectively overcoming the problems of difficult data collection on-site in factories, time-consuming and labor-intensive image annotation, and large differences in model effects in the prior art.

[0091] In some specific embodiments, the device may further include:

[0092] The matching failure module is used to add the target template image corresponding to the image of the area to be detected to the template image library if the image of the area to be detected fails to match with the pre-created template image library, and re-match the image of the area to be detected with the pre-created template image library.

[0093] In some specific embodiments, the device may further include:

[0094] The end module is used to end the process directly if there is no abnormality in the image of the area to be detected.

[0095] In some specific embodiments, the device may further include:

[0096] A model building module, used to build an initial feature extraction model based on a plurality of depth-separable convolution modules; wherein the depth-separable convolution module includes depth-wise convolution, point-wise convolution, batch normalization and H-Swish activation function;

[0097] The model training module is used to train the initial feature extraction model through a loss function based on metric learning to obtain a trained feature extraction model.

[0098] In some specific embodiments, the device may further include:

[0099] A feature extraction module is used to collect abnormal images of leakage existing in the production environment corresponding to the target device, and use the feature extraction model to extract features from the abnormal images to obtain target features;

[0100] The target feature library construction module is used to convert each of the floating point target features into a binary feature by using hash coding, and construct the target feature library based on the binary feature.

[0101] In some specific embodiments, the anomaly detection module 14 may specifically include:

[0102] A similarity measure determination unit, used to determine the similarity measure between the extracted feature and each target feature in the target feature library based on the Hamming distance;

[0103] A maximum similarity measure determination unit, used to sort the similarity measures and determine the maximum similarity measure;

[0104] A comparison unit, used for comparing the maximum similarity measure with a preset threshold;

[0105] The abnormal category determination unit is used to determine the image category corresponding to the maximum similarity metric as the abnormal category corresponding to the extracted feature if the maximum similarity metric is greater than or equal to the preset threshold.

[0106] In some specific embodiments, the device may further include:

[0107] A picture category determination module, configured to determine the picture category corresponding to the maximum similarity metric as an unknown category if the maximum similarity metric is less than the preset threshold;

[0108] The picture adding module is used to add the picture category corresponding to the maximum similarity measurement to the target feature library and perform the next round of device anomaly detection.

[0109] Furthermore, the present application also discloses an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0110] Figure 8 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the device abnormality detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0112] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0113] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the device anomaly detection method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0114] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed device abnormality detection method. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0115] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A device anomaly detection method, characterized in that: include: Using an image acquisition device to obtain an image of the area to be detected of the target device, and matching the image of the area to be detected with a pre-created template image library; If the match is successful, the image of the area to be detected and the template image that has been matched are segmented based on a preset segmentation model to obtain a corresponding first segmentation result set and a second segmentation result set; Comparing the first segmentation result set with the second segmentation result set to determine a target image with an abnormality and an abnormality detection position corresponding to the image of the area to be detected, and cropping the target image based on the abnormality detection position to obtain a cropped image; The cropped image is input into a pre-trained feature extraction model for feature extraction, the extracted features are obtained, and the abnormality category corresponding to the extracted features is determined based on the similarity measurement and the target feature library, the abnormality category and the abnormality detection position are output, and a corresponding alarm is issued to complete the abnormality detection of the target device.

2. The device abnormality detection method according to claim 1, characterized in that: After matching the image of the area to be detected with the pre-created template image library, the method further includes: If the image of the area to be detected fails to match the pre-created template image library, the target template image corresponding to the image of the area to be detected is added to the template image library, and the image of the area to be detected is matched with the pre-created template image library again.

3. The device abnormality detection method according to claim 1, characterized in that: After comparing the first segmentation result set with the second segmentation result set, the method further includes: If there is no abnormality in the image of the area to be detected, the process ends directly.

4. The device abnormality detection method according to claim 1, characterized in that: Before inputting the cropped image into a pre-trained feature extraction model for feature extraction, the method further includes: An initial feature extraction model is constructed based on several depth-wise separable convolution modules; wherein the depth-wise separable convolution modules include depth-wise convolution, point-wise convolution, batch normalization and H-Swish activation function; The initial feature extraction model is trained by a loss function based on metric learning to obtain a trained feature extraction model.

5. The device anomaly detection method according to claim 1, characterized in that: Before determining the abnormal category corresponding to the extracted feature based on the similarity measurement and the target feature library, the method further includes: Collecting abnormal images of leakages existing in the production environment corresponding to the target device, and extracting features from the abnormal images using the feature extraction model to obtain target features; Hash coding is used to convert each of the floating-point target features into binary features, and the target feature library is constructed based on the binary features.

6. The device abnormality detection method according to any one of claims 1 to 5, characterized in that: The determining the abnormal category corresponding to the extracted feature based on the similarity measurement and the target feature library includes: Determine the similarity measure between the extracted features and each target feature in the target feature library based on the Hamming distance; Sorting the similarity measures to determine the maximum similarity measure; Comparing the maximum similarity measure with a preset threshold; If the maximum similarity measure is greater than or equal to the preset threshold, the image category corresponding to the maximum similarity measure is determined as the abnormal category corresponding to the extracted feature.

7. The device anomaly detection method according to claim 6, characterized in that: After comparing the maximum similarity measure with the preset threshold, the method further includes: If the maximum similarity measure is less than the preset threshold, determining the picture category corresponding to the maximum similarity measure as an unknown category; The image category corresponding to the maximum similarity measure is added to the target feature library, and the next round of device anomaly detection is performed.

8. A device for detecting abnormality of equipment, characterized in that: include: A matching module, used to obtain an image of the area to be detected of the target device by using an image acquisition device, and match the image of the area to be detected with a pre-created template image library; An image segmentation module, for segmenting the image of the area to be detected and the template image that has been matched successfully based on a preset segmentation model if the match is successful, so as to obtain a corresponding first segmentation result set and a second segmentation result set; An image cropping module is used to compare the first segmentation result set and the second segmentation result set to determine a target image with an abnormality and an abnormality detection position corresponding to the image of the area to be detected, and crop the target image based on the abnormality detection position to obtain a cropped image; The anomaly detection module is used to input the cropped image into a pre-trained feature extraction model for feature extraction, obtain the extracted features, and determine the anomaly category corresponding to the extracted features based on a similarity metric and a target feature library, output the anomaly category and the anomaly detection position, and issue a corresponding alarm to complete the anomaly detection of the target device.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the device abnormality detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the device anomaly detection method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Crane crown block track slide wire deformation detection device, system and method

    CN121033562A