Method and apparatus for detecting device defects, electronic device, and computer readable medium

Abstract

The present disclosure relates to a device defect detection method, device, electronic device and computer readable medium. The method comprises: acquiring a real-time image of a device; inputting the real-time image into a first feature extraction model and a second feature extraction model to generate a feature extraction result; inputting the feature extraction result into a category prediction model to generate a target category; inputting the feature extraction result into a multi-scale prediction model to generate a plurality of target bounding boxes; and detecting defects of the device based on the target category and the plurality of target bounding boxes. The device defect detection method, device, electronic device and computer readable medium of the present disclosure have higher precision and faster detection speed compared with existing device defect detection technologies, and can be arranged in an edge service device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer information processing, and in particular, to a device defect detection method and device, electronic device and computer readable medium. BACKGROUND

[0002] With the rapid development of economy and society, people's living standards have made a qualitative leap, and the demand for electric energy is increasing year by year. On the one hand, it brings broad development space to power enterprises, and on the other hand, it also brings unprecedented challenges to power enterprises. The increasing demand for electricity has a certain impact on the stable operation of power equipment, and also tests the quality of power supply to power equipment. The continuous progress of science and technology has brought about the development of power grid technology and power equipment to a certain extent. In order to ensure that the power equipment can operate normally, timely find out the problems and safety hazards existing in the operation of the power system, and take effective measures to control and solve them, so as to meet the demand for electric energy in people's production and life, it is necessary to improve the daily operation and maintenance of power equipment. At present, the inspection business of the power system is mostly carried out by intelligent inspection equipment, which effectively improves the accuracy of the power equipment fault judgment of the operation and maintenance personnel, greatly reduces the workload of the operation and maintenance personnel, and reduces the cost of daily operation and maintenance of power equipment. However, the existing video monitoring system only simply transmits image information to the video monitoring interface of the control center, and the massive unstructured image information still depends on manual processing, which has the problems of weak conversion rate, low data value, and low intelligent degree. Therefore, intelligent analysis and processing of unstructured data have important data mining significance.

[0003] Object Detection provides a feasible solution to solve the above-mentioned problems. The main steps of traditional multi-object detection are to select candidate regions in the input image, extract features from these candidate regions, and then use a feature classifier for classification. When selecting candidate regions, the method of exhaustive sliding window is used, which is not targeted, has high time complexity, and has a large amount of window redundancy. Therefore, using traditional methods for multi-object detection is time-consuming and has poor detection effect.

[0004] The above information disclosed in the background section is only used to enhance the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] Therefore, the present disclosure provides a device defect detection method and device, electronic device and computer readable medium, which has higher precision and faster detection speed than existing device defect detection technology, and can be arranged in an edge service device.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, or will become more readily apparent by practice of the present disclosure.

[0007] According to an aspect of the present disclosure, a method for detecting equipment defects is provided, which includes: acquiring a real-time image of the equipment; inputting the real-time image into a first feature extraction model and a second feature extraction model to generate a feature extraction result; inputting the feature extraction result into a category prediction model to generate a target category; inputting the feature extraction result into a multi-scale prediction model to generate a plurality of target bounding boxes; and detecting defects of the equipment based on the target category and the plurality of target bounding boxes.

[0008] In an exemplary embodiment of the present disclosure, the method further includes: training a Darknet-53 network model based on a plurality of historical failure images of the equipment to generate the first feature extraction model; and training a VGG16 network model based on the plurality of historical failure images of the equipment to generate the second feature extraction model.

[0009] In an exemplary embodiment of the present disclosure, the method further includes: inputting a plurality of historical failure images of the equipment into the first feature extraction model and the second feature extraction model to generate a plurality of historical feature extraction results; training a GBDT model based on the plurality of historical feature extraction results; and fitting a category loss error in a training result based on a log-likelihood loss function to generate the category prediction model.

[0010] In an exemplary embodiment of the present disclosure, the method further includes: inputting a plurality of historical failure images of the equipment into the first feature extraction model and the second feature extraction model to generate a plurality of historical feature extraction results; training a multi-scale classification model based on the plurality of historical feature extraction results; generating a loss function in a training result based on a probability that an anchor box contains an object; and generating the multi-scale prediction model based on an optimal solution of the loss function.

[0011] In an exemplary embodiment of the present disclosure, inputting the real-time image into the first feature extraction model and the second feature extraction model to generate a feature extraction result includes: inputting the real-time image into the first feature extraction model to generate a first feature result; inputting the real-time image into the second feature extraction model to generate a second feature result; and splicing the first feature result and the second feature result to generate the feature extraction result.

[0012] In an example embodiment of the present disclosure, a Darknet-53 network model is trained based on a plurality of historical failure images of a device to generate the first feature extraction model, including: performing image labeling processing on the plurality of historical failure images of the device; inputting the labeled plurality of historical failure images into the Darknet-53 network model, wherein the Darknet-53 includes 52 convolutional layers, batch normalization layers and skip layer connections; and the Darknet-53 network model performs residual training on a sub-network based on the plurality of historical failure images to generate the first feature extraction model.

[0013] In an example embodiment of the present disclosure, a VGG16 network model is trained based on a plurality of historical failure images of a device to generate the second feature extraction model, including: performing image labeling processing on the plurality of historical failure images of the device; inputting the labeled plurality of historical failure images into the VGG16 network model; and the VGG16 network model is trained based on the plurality of historical failure images to generate the second feature extraction model.

[0014] In an example embodiment of the present disclosure, a class prediction model is generated by fitting a class loss error in a training result based on a log-likelihood loss function, including: generating a loss in a training process by using a difference between a predicted probability value and a true probability value of a class; fitting an error of the loss by the log-likelihood loss function; and generating the class prediction model when the error of the loss is minimized.

[0015] In an example embodiment of the present disclosure, a multi-scale classification model is trained based on the plurality of historical feature extraction results, including: clustering bounding boxes in the plurality of historical feature extraction results based on a clustering algorithm in the training; generating a plurality of anchor boxes according to the clustering results; and generating a plurality of target bounding boxes based on the plurality of anchor boxes.

[0016] In an example embodiment of the present disclosure, a loss function in a training result is generated based on a probability of an object contained in an anchor box, including: predicting the probability of the object contained in the anchor box based on a logistic regression to generate the loss function in the training result.

[0017] According to an aspect of the present disclosure, a device defect detection apparatus is provided, which includes: an image module configured to acquire a real-time image of a device; a feature module configured to input the real-time image into a first feature extraction model and a second feature extraction model to generate feature extraction results; a class module configured to input the feature extraction results into a class prediction model to generate a target class; a bounding box module configured to input the feature extraction results into a multi-scale prediction model to generate a plurality of target bounding boxes; and a detection module configured to detect defects of the device based on the target class and the plurality of target bounding boxes.

[0018] According to an aspect of the present disclosure, an electronic device is provided that includes one or more processors; a memory configured to store one or more programs; and the one or more programs are configured to, when executed by the one or more processors, cause the one or more processors to implement the method as above.

[0019] According to an aspect of the present disclosure, a computer-readable medium having stored thereon a computer program is provided, the computer program, when executed by a processor, implementing the method as above.

[0020] According to the device defect detection method and device, electronic device and computer readable medium of the present disclosure, a real-time image of the device is acquired; the real-time image is input into a first feature extraction model and a second feature extraction model to generate a feature extraction result; the feature extraction result is input into a category prediction model to generate a target category; the feature extraction result is input into a multi-scale prediction model to generate a plurality of target bounding boxes; and the device defect is detected based on the target category and the plurality of target bounding boxes. Compared with existing device defect detection technologies, the device defect detection method and device, electronic device and computer readable medium of the present disclosure have higher accuracy and faster detection speed, and can be arranged in an edge service device.

[0021] It should be understood that the general description above and the following detailed description are only exemplary and do not limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0023] Figure 1 is a system block diagram of a device defect detection method and device according to an example embodiment.

[0024] Figure 2 is a flowchart of a device defect detection method according to an example embodiment.

[0025] Figure 3 is a flowchart of a device defect detection method according to another example embodiment.

[0026] Figure 4 is a flowchart of a device defect detection method according to another example embodiment.

[0027] Figure 5 is a flowchart of a device defect detection method according to another example embodiment.

[0028] Figure 6 is a flowchart of a device defect detection method according to another exemplary embodiment.

[0029] Figure 7 is a schematic diagram of a device defect detection method according to another exemplary embodiment.

[0030] Figure 8 is a comparison of average precision (AP).

[0031] Figure 9 is a comparison of average precision mean (mAP).

[0032] Figure 10 is a comparison of defect detection effects.

[0033] Figure 11 is a block diagram of a device defect detection apparatus according to an exemplary embodiment.

[0034] Figure 12 is a block diagram of an electronic device according to an exemplary embodiment.

[0035] Figure 13 is a block diagram of a computer readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the figures, and descriptions of the same elements can be omitted from the descriptions of the figures.

[0037] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.

[0038] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0040] It should be understood that although the terms first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be called the second component without departing from the teachings of the present disclosure concept. As used herein, the term "and / or" includes any one and all combinations of the associated listed items.

[0041] Those skilled in the art can understand that the modules or flows in the drawings are not necessarily required for implementing the present disclosure, and therefore cannot be used to limit the protection scope of the present disclosure.

[0042] Figure 1 is a system block diagram of a device defect detection method, apparatus, electronic device and computer readable medium according to an exemplary embodiment.

[0043] As Figure 1 shown, the system architecture 10 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0044] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045] The terminal devices 101, 102, 103 can be various electronic devices with monitoring functions and supporting data transmission or calculation, including but not limited to power electronic devices, smart cameras, smart monitoring meters, etc.

[0046] The terminal device 101, 102, 103 or the server 105 may, for example, acquire a real-time image of the device; the terminal device 101, 102, 103 or the server 105 may, for example, input the real-time image into the first feature extraction model and the second feature extraction model, generate a feature extraction result; the terminal device 101, 102, 103 or the server 105 may, for example, input the feature extraction result into the category prediction model, generate a target category; the terminal device 101, 102, 103 or the server 105 may, for example, input the feature extraction result into the multi-scale prediction model, generate a plurality of target bounding boxes; the terminal device 101, 102, 103 or the server 105 may, for example, detect defects of the device based on the target category and the plurality of target bounding boxes.

[0047] The server 105 may, for example, train the Darknet-53 network model based on a plurality of historical failure images of the device to generate the first feature extraction model.

[0048] The server 105 may, for example, train the VGG16 network model based on a plurality of historical failure images of the device to generate the second feature extraction model.

[0049] The server 105 may, for example, input a plurality of historical failure images of the device into the first feature extraction model and the second feature extraction model, generate a plurality of historical feature extraction results; train the GBDT model through the plurality of historical feature extraction results; fit the category loss error in the training result based on the log-likelihood loss function to generate the category prediction model.

[0050] The server 105 may, for example, input a plurality of historical failure images of the device into the first feature extraction model and the second feature extraction model, generate a plurality of historical feature extraction results; train the multi-scale classification model through the plurality of historical feature extraction results; generate a loss function in the training result based on the probability that the anchor box contains an object; generate the multi-scale prediction model based on the optimal solution of the loss function.

[0051] The server 105 may be a server of one entity, or may also be composed of a plurality of servers. It should be noted that the device defect detection method provided in the embodiments of the present disclosure can be executed by the server 105 and / or the terminal device 101, 102, 103, and accordingly, the device defect detection apparatus can be arranged in the server 105 and / or the terminal device 101, 102, 103.

[0052] Figure 2is a flowchart of a device defect detection method according to an exemplary embodiment. The device defect detection method 20 at least includes steps S202-S210.

[0053] As shown in S202, a real-time image of the device is acquired. The real-time image of the power device can be captured by the intelligent patrol device Figure 2

[0054] In S204, the real-time image is input into a first feature extraction model and a second feature extraction model to generate a feature extraction result. For example, the real-time image can be input into the first feature extraction model to generate a first feature result; the real-time image can be input into the second feature extraction model to generate a second feature result; and the first feature result and the second feature result can be spliced to generate the feature extraction result.

[0055] In S206, the feature extraction result is input into a category prediction model to generate a target category. The category prediction model is used to predict the category of the fault, and the category of the fault can include, for example, meter damage, meter dial blur, meter normal, respirator silica discoloration, respirator silica normal, etc.

[0056] In S208, the feature extraction result is input into a multi-scale prediction model to generate a plurality of target bounding boxes. The multi-scale prediction model is used to give a plurality of regions where faults can exist. The multi-scale prediction model can identify a plurality of target bounding boxes on an image, each target bounding box can represent a region where a fault can exist, and the number of target bounding boxes can be set artificially in advance.

[0057] In S210, the defects of the device are detected based on the target category and the plurality of target bounding boxes. The defect category and the target bounding box are one-to-one corresponding to generate a final defect detection result. When the device has defects, warning information can also be automatically generated to remind the staff to handle.

[0058] According to the device defect detection method of the present disclosure, a real-time image of the device is acquired; the real-time image is input into a first feature extraction model and a second feature extraction model to generate a feature extraction result; the feature extraction result is input into a category prediction model to generate a target category; the feature extraction result is input into a multi-scale prediction model to generate a plurality of target bounding boxes; and the defects of the device are detected based on the target category and the plurality of target bounding boxes. Compared with existing device defect detection techniques, the device defect detection method has higher accuracy and faster detection speed, and can be arranged in an edge service device.

[0059] ​It should be clearly understood that the present disclosure describes how to form and use particular examples, but the principles of the present disclosure are not limited to any detail of these examples. Rather, based on the teachings of the present disclosure, these principles can be applied to a number of other embodiments.

[0060] Figure 3 is a flowchart of a device defect detection method according to another exemplary embodiment. Figure 3 The flow 30 shown is a detailed description of "generating the first feature extraction model".

[0061] As Figure 3 As shown in S302, image labeling processing is performed on a plurality of historical failure images of the device. In order to ensure that real power plant environment images are collected, the power equipment can be shot at fixed points, at fixed times, and along fixed paths. In addition, in order to enable the edge computer to quickly identify the problem equipment so as to timely repair, a large number of shots can also be taken for some equipment that has problems, such as meter damage, insulator breakage, respirator silica discoloration, etc. In one embodiment, the collected power equipment images are mainly divided into 22 categories, including meter damage, meter dial blur, meter normal, respirator silica discoloration, respirator silica normal, etc., a total of 5944 power equipment state images.

[0062] Since there may be repeated images, blurred images and damaged image files in the obtained original image data set, in order to purify the data set, the obtained original image data needs to be screened to a certain extent, and dirty data is removed to avoid unnecessary impact on the quality of the training model.

[0063] Manual labeling can be performed using the Label Image tool to label six categories of power equipment targets in each image, including respirators, meters, insulators, leaking oil components, metals, etc., and try to ensure that the overall shape of the target is labeled. After that, delete invalid sample data, rename the xml file with messy names regularly, and convert it into a VOC format data set that can be trained.

[0064] In S304, the labeled plurality of historical failure images are input into a Darknet-53 network model, wherein the Darknet-53 includes 52 convolutional layers, batch normalization layers and jump layer connections.

[0065] In S306, the Darknet-53 network model performs residual training on the sub-network based on the plurality of historical failure images to generate the first feature extraction model.

[0066] Generally, the Darknet-53 convolutional neural network has 53 convolutional layers, mainly composed of convolutional layers, batch normalization and skip connections, and the activation function uses Leaky Relu. In the embodiments of the present disclosure, the first 52 layers of Darknet-53 are used, and the last fully connected layer is removed, and the input image size is 416*416*3. The network structure introduces a residual module on the previous network structure, where each residual component has two convolutional layers and a shortcut link. The role of the residual layer is to divide the deep neural network into several sub-networks, each of which contains fewer network layers, and then uses the shortcut connection method to train the residual of each sub-network. The residual layer does not affect the size of the input and output, only calculates the residual, divides the deep network into multiple shallow networks, so it can better control gradient propagation and avoid problems such as gradient explosion or gradient disappearance.

[0067] Figure 4 is a flowchart of a device defect detection method according to another example embodiment. Figure 4 The flow 40 shown is a detailed description of "generating the second feature extraction model".

[0068] As Figure 4 As shown in S402, the plurality of historical failure images of the device are subjected to image labeling processing. The specific labeling process is as described above.

[0069] In S404, the labeled plurality of historical failure images are input into the VGG16 network model.

[0070] In S406, the VGG16 network model is trained based on the plurality of historical failure images to generate the second feature extraction model.

[0071] In the embodiments of the present disclosure, the VGG16 convolutional neural network can include 13 convolutional layers, 3 fully connected layers, and 5 pooling layers. The network structure of the VGG16 convolutional neural network has certain rules, and each is a plurality of convolutional layers followed by a pooling layer, which can reduce the height and width of the image to half of the original. In addition, there is a certain rule in the number of filter changes of the convolutional layer, which is doubled in each group of convolutional layer. The input image pixel is 224x224, the channel number is 3, the size of the convolution kernel of the convolutional layer is 3x3, the step is 1, the size of the convolution kernel of the pooling layer is 2x2, the step is 2, and the last 512-channel 7x7 feature map is subjected to a full connection operation to obtain 4096 units, and then subjected to Softmax activation.

[0072] Figure 5 is a flowchart of a device defect detection method according to another example embodiment.Figure 5 The flow 50 shown is a detailed description of "generating the category prediction model".

[0073] As Figure 5 shown, in S502, a plurality of historical fault images of the device are input into the first feature extraction model and the second feature extraction model to generate a plurality of historical feature extraction results. Since the classification effect after feature extraction by the Darknet-53 model or the VGG16 model is not ideal, the prediction accuracy is low, so in the classification subtask of target detection in the present disclosure, the category feature vectors extracted by the above two models are spliced and then input into the GBDT algorithm for feature fusion and training, so that the two features correct each other, thereby achieving the purpose of improving the target classification accuracy. The Darknet-53 model feature extraction produces 5-dimensional, 9-dimensional and 13-dimensional feature maps, and the VGG16 model feature extraction produces a 7-dimensional feature map. The three feature maps produced by the Darknet-53 model are spliced with the feature map produced by the VGG16 model to produce 12-dimensional, 16-dimensional and 20-dimensional feature matrices.

[0074] In S504, the GBDT model is trained based on the plurality of historical feature extraction results.

[0075] In S506, the category loss error in the training result is fitted based on a log-likelihood loss function to generate the category prediction model. This includes: generating the loss in the training process using the difference between the predicted probability value and the true probability value of the category; fitting the error of the loss by the log-likelihood loss function; and generating the category prediction model when the error of the loss reaches a minimum value.

[0076] Since the experimental sample output is not a continuous value but a discrete category, we cannot directly fit the category output error from the output category. Therefore, for multivariate classification, the present patent proposes a method of fitting the loss using the difference between the predicted probability value and the true probability value of the category, and using the idea of log-likelihood loss function to fit the output category loss error, as shown in formula (1), where K represents the number of categories, and if the sample output category is k, then y k = 1, otherwise y k = 0.

[0077]

[0078] The expression of the probability p k (x) of the kth category is shown in formula (2).

[0079]

[0080] Combining formulas (1) and (2), the negative gradient error of the class l corresponding to the ith sample of the tth round can be calculated as shown in formula (3).

[0081]

[0082] wherein the error is the difference between the true probability of the sample i corresponding to the class l and the predicted probability of the t-1th round. For the generated decision tree, the best negative gradient fitting value of each leaf node is shown in formula (4).

[0083]

[0084] Since the above formula is difficult to optimize, we use an approximation instead as shown in formula (5).

[0085]

[0086] As shown above, by continuously iterating, the residual error generated in the training process can be continuously reduced, the target classification accuracy can be improved, and thus the accuracy of the class prediction can be improved.

[0087] Figure 6 FIG. 6 is a flowchart of a device defect detection method according to another example embodiment. Figure 6 The flowchart 60 shown is a detailed description of "generating the multi-scale prediction model".

[0088] As shown in FIG. 6, in S602, a plurality of historical failure images of the device are input into the first feature extraction model and the second feature extraction model to generate a plurality of historical feature extraction results. Figure 6

[0089] In S604, the multi-scale classification model is trained based on the plurality of historical feature extraction results. In the training, the bounding boxes in the plurality of historical feature extraction results are clustered based on a clustering algorithm; a plurality of anchor boxes are generated according to the clustering results; and a plurality of target bounding boxes are generated based on the plurality of anchor boxes. In the same scale layer, the convolution operation is used to complete the interaction of the feature map and the local feature using 1x1 and 3x3 convolution kernels. In the present disclosure, three different scale feature matrices are taken as an example for illustration, the sizes of the three feature matrices are 12x12, 16x16 and 20x20 respectively, and then the target is detected at the three scales, and each cell uses three anchor boxes to predict three bounding boxes.

[0090] In S606, the probability of the anchor box containing the object is used to generate the loss function in the training result. It includes: using the logistic regression to predict the probability of the anchor box containing the object to generate the loss function in the training result.

[0091] ​On each cell, 4 values are predicted for each bounding box, namely the top-left corner coordinates (x, y) and the width and height (w, h) of the target, denoted as (t x ,t y ,t w ,t h ). If the center of the target is offset from the top-left corner of the image (C x ,C y ), and the anchor box has width and height (P w ,P h ), the corrected bounding box is shown in equation (6).

[0092]

[0093] The selection of anchor boxes uses a dimension clustering method. The method uses the K-means clustering algorithm to cluster the sizes of the target bounding boxes in the training set, and obtains the size of the best anchor box, so as to predict more accurate target bounding boxes. The distance metric of the K-means clustering algorithm is shown in equation (7). Wherein, box refers to the size of the bounding box sample in the data set, and centroid refers to the size of the cluster center.

[0094] d(box,centroid)=1-IOU(box,centroid) (7)

[0095] The patent uses a logistic regression to predict the probability of an anchor box containing an object. If the anchor box has a higher overlap rate with the real target bounding box than any other anchor box, the probability of this anchor box is 1; if the anchor box has an overlap rate with the real target bounding box greater than 0.5 but not the largest, this prediction is ignored. The method only assigns one anchor box to one object during training, and if the anchor box does not contain the recognized object, it does not work in the loss function.

[0096] The loss function of the patent is shown in equation (8). The loss function includes three parts: the first part l box represents the loss generated by the 4 coordinates (t x ,t y ,t w ,t h ) of each bounding box, the second part l cls represents the loss generated by the confidence, and the third part l obj represents the loss generated by the class.

[0097]

[0098] After calculating the loss value of the model, the next step is to optimize the model parameters using the loss value. The patent proposes an optimization method for the loss function, which uses the conjugate gradient method to accelerate the training convergence process of the loss function, and obtains the next direction that is conjugate to the current search direction through iteration. First, give the initial iteration value x0 and threshold ε>0, let k=0. Calculate and the initial descent direction, which satisfies Repeat the following steps. If ||g k ||<ε, stop iteration. The line search determines the calculation of step size α as shown in formula (9), where the total number of training samples is n, k=0, 1,..., n, x is the parameter object we want to optimize, and d is the search direction.

[0099]

[0100] The calculation of the updated iteration point is shown in formula (10):

[0101] x k+1 =x k +α k d k (10)

[0102] The search direction d k+1 calculated by the conjugate method satisfies formula (11), and then let k=k+1.

[0103]

[0104] In S608, the multi-scale prediction model is generated based on the optimal solution of the loss function. As shown above, by continuously iterating to solve the minimum value of the function, the minimized loss function and model parameter value are finally obtained, and the parameter optimization of the model is realized.

[0105] The scheme adopted by the present disclosure is illustrated as shown in Figure 7 In the power distribution operation target identification task, first, the device state image photographed by the intelligent patrol device used in the patrol business in the power distribution operation needs to be collected, and then the dirty data in the original power equipment image data is removed, and appropriate research samples are selected. Secondly, the image is manually labeled for power equipment using a labeling tool to obtain a power equipment target detection dataset for training a power equipment target detection model.

[0106] The target detection task includes two subtasks: one is a subtask of classifying the target, and the other is a subtask of positioning the target, wherein the purpose of the classification subtask is to add a category label to the detected target. In the classification subtask, two network models of Darknet-53 and VGG16 are respectively used to extract a feature matrix of the target on the constructed data set, the matrices output by the two models are fused by splicing, and then an integrated learning algorithm of gradient boosting decision tree (GBDT) is used to fuse and learn the output features of the two models, and a fused category label feature is output. Then, multi-scale prediction of defects is performed, the detection effect of different size objects and occluded objects is effectively enhanced by predicting in three scales, and a leap layer connection is introduced to strengthen the convergence effect, and a random multi-scale training method is used to enhance the robustness of the algorithm. Then, a loss function optimization method of the conjugate gradient method is used to speed up the training convergence process of the loss function. Finally, related experiments and analysis are performed to evaluate the target detection effect.

[0107] The device defect detection method of the present disclosure proposes a multi-target detection optimization method for the power equipment operation and maintenance field. Based on data preprocessing of power equipment defect image data, the method uses an integrated learning technology-based multi-target detection optimization method to improve the efficiency of multi-target detection.

[0108] The device defect detection method of the present disclosure uses Darknet-53 and VGG16 models as feature extractors to extract multi-target features in the training data set, and then uses an integrated learning algorithm of gradient boosting decision tree (GBDT) to fuse and learn the output features of the two models, and then predicts the multi-target in unknown image instances to realize target detection in monitoring applications.

[0109] The device defect detection method of the present disclosure proposes an integrated learning method based on gradient boosting decision tree (GBDT) to fuse and learn the output features of the weak classifier model based on deep learning, and finally trains a strong classifier for multi-target detection.

[0110] The device defect detection method of the present disclosure proposes an optimization loss function method based on the conjugate gradient method, which uses the conjugate gradient method to speed up the loss function training convergence process and improve the classification quality.

[0111] Experimental setup and data set

[0112] The experiment uses Win10, GPU with GTX2080 and 16G memory and CUDA / CUDNN. The experimental data set is 1432 power equipment state images after screening, including six types of equipment state images such as respirator, meter, insulator, leakage oil, foreign matter and metal corrosion. The training / test / validation data division of the collected data set is performed, 72% of the entire data set is used for model training as the training set, 20% of the entire data set is used for model testing as the test set, and the remaining 8% is used as the validation set.

[0113] The sample satisfying the condition shown in formula (12) is a positive sample.

[0114]

[0115] Wherein, T represents the real target frame originally labeled in the test set, P represents the candidate frame detected by the training model, and area represents the area of the frame.

[0116] Experimental index

[0117] In order to evaluate the performance of multi-target detection, the average precision (AP) and the average precision mean (mAP) are used as evaluation indexes, as shown in formula (13) and formula (14).

[0118]

[0119] AP Class The average precision of the class reflects the precision rate of the target detection model obtained by training on a certain label class.

[0120]

[0121] mAP Class The sum of the average precision of each class and the total number of classes reflects the average precision rate of the target detection model obtained by training on all label classes.

[0122] Wherein P Class The precision of the class reflects the ratio of the positive examples that are actually positive examples in the predicted positive examples, and the definition formula is shown in formula (15).

[0123]

[0124] Comparative method

[0125] YOLOv3-tiny as a comparative method, which does not have a residual layer, only uses two different scales of YOLO output layer, and adopts a 7-layer network for feature extraction. YOLOv3-tiny has a faster execution speed than many existing methods, and is widely used in real-time multi-target detection.

[0126] Experimental results and analysis

[0127] In this experiment, the technical solution proposed in the present disclosure and the YOLOv3-tiny method for comparison are used for training and testing, and 100 iterations are performed on the power equipment dataset. Taking the five categories of meter damage, meter dial blur, meter normal, respirator silica gel discoloration, and respirator silica gel normal as examples for analysis, the AP values and mAP values of the two methods are compared. Figure 8 and Figure 9 The experimental results of the AP values and mAP values of the two target detection methods on the five categories are shown in (a) and (b), respectively.

[0128] As shown in (a) and (b), the target detection model used in the technical solution of the present disclosure has higher average precision in the five categories: meter damage, meter dial blur, meter normal, respirator silica gel discoloration, and respirator silica gel normal, compared with the YOLOv3-tiny method. The mAP value based on YOLOv3 is significantly higher than the mAP value based on YOLOv3-tiny. Figure 8 Figure 9

[0129] Figure 10 (a) and (b) show the actual detection effect of the same four images under the two target detection methods. Among them, Figure 10 (a) is the scheme in the present disclosure, Figure 10 (b) is the YOLOv3-tiny method. From the detection results in the figure, it can be seen that the technical solution of the present disclosure has higher detection accuracy in power equipment detection. Figure 10 Through the relevant experimental results, compared with the current popular methods, the technical solution of the present disclosure can meet the requirements of power equipment defect analysis, has higher average precision and average precision mean, and is effective in the analysis and application of multi-target detection based on power equipment images. The power equipment defect analysis framework proposed in the present disclosure has higher precision and improves the performance of defect analysis applications.

[0130]

[0131] ​​​Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above-mentioned functions defined by the above-mentioned method provided by the present disclosure are executed. The program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0132] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not for limiting purposes. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0133] The following is a device embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the device embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0134] Figure 11 is a block diagram of a device defect detection apparatus according to an exemplary embodiment. As shown in Figure 11 The device defect detection apparatus 110 includes an image module 1102, a feature module 1104, a category module 1106, a bounding box module 1108, and a detection module 1110.

[0135] The image module 1102 is configured to acquire a real-time image of a device;

[0136] The feature module 1104 is configured to input the real-time image into a first feature extraction model and a second feature extraction model, and generate a feature extraction result;

[0137] The category module 1106 is configured to input the feature extraction result into a category prediction model, and generate a target category;

[0138] The bounding box module 1108 is configured to input the feature extraction result into a multi-scale prediction model, and generate a plurality of target bounding boxes;

[0139] The detection module 1110 is configured to detect defects of the device based on the target category and the plurality of target bounding boxes.

[0140] The device defect detection device according to the present disclosure acquires a real-time image of a device; inputs the real-time image into a first feature extraction model and a second feature extraction model to generate feature extraction results; inputs the feature extraction results into a category prediction model to generate a target category; inputs the feature extraction results into a multi-scale prediction model to generate a plurality of target bounding boxes; and detects defects of the device based on the target category and the plurality of target bounding boxes. Compared with existing device defect detection technologies, the device defect detection device has higher accuracy and faster detection speed, and can be arranged in an edge service device.

[0141] Figure 12 is a block diagram of an electronic device according to an exemplary embodiment.

[0142] The electronic device 1200 according to this implementation of the present disclosure will be described below with reference to Figure 12 Figure 12 The displayed electronic device 1200 is merely an example and should not impose any limitation on the functions and usage range of the embodiments of the present disclosure.

[0143] As shown in Figure 12 , the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 can include, but are not limited to, at least one processing unit 1210, at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), a display unit 1240, and the like.

[0144] The storage unit stores program codes that can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps described in the present specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1210 can perform the steps as shown in Figures 2 to 6

[0145] The storage unit 1220 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 12201 and / or a cache memory 12202, and can further include a read-only memory (ROM) 12203.

[0146] The storage unit 1220 can further include program / utilities 12204 having a set of (at least one) program modules 12205, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include implementation of a network environment.

[0147] ​​Bus 1230 can be one or more of several types of bus structure including a memory bus or memory controller, a peripheral bus, a graphics bus, a processor or local bus using any of a variety of bus architectures.

[0148] Electronic device 1200 can also communicate with one or more external devices 1200' such as a keyboard or pointing device, a display, a printer, etc. using one or more of buses 1230. Communication with one or more external devices 1200' can occur using input / output (I / O) interface(s) 1250. I / O interface 1250 can enable electronic device 1200 to interface to various peripheral devices, such as a printer, disk drives, or the like. I / O interface 1250 can also interface to various input and output devices such as display, keyboard, mouse, etc. In some embodiments, I / O interface 1250 can include one or more buses, as described above. In other embodiments, I / O interface 1250 can be implemented using a combination of buses.

[0149] From the foregoing description, it will be apparent to a person skilled in the art that the example embodiments described herein can be implemented in software and / or can be implemented as software in combination with the necessary hardware. Moreover, as software is involved in the described embodiments, it will thus be appreciated that a computer, processor or controller, or other Figure 13 product in the form of a software program can be stored in any medium (e.g., RAM, flash memory, ROM, EEPROM, and / or the like) or transmitted via a network, which can be implemented with any suitable digital and / or analog communication medium now known or later developed including, but not limited to, the Internet, intranet(s), Local Area Network(s) (LAN(s)), Wide Area Network(s) (WAN(s)), Metropolitan Area Network(s) (MAN(s)), and the like.

[0150] The software product can be in any combination of one or more of the readable medium. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0151] The computer readable storage medium can include a data signal transported over a carrier wave and can be baseband or propagated along with carriers. The program code embodied on the computer readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, and the like, or any suitable combination of the foregoing.

[0152] The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0153] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the device, the computer readable medium enables the following functions: acquiring a real-time image of the device; inputting the real-time image into a first feature extraction model and a second feature extraction model to generate a feature extraction result; inputting the feature extraction result into a category prediction model to generate a target category; inputting the feature extraction result into a multi-scale prediction model to generate a plurality of target bounding boxes; and detecting defects of the device based on the target category and the plurality of target bounding boxes.

[0154] Those skilled in the art can understand that the above modules can be distributed in the device according to the description of the embodiments, and can also be changed to be in one or more devices different from the embodiments. The modules of the above embodiments can be combined into one module, or can be further split into a plurality of sub-modules.

[0155] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0156] The example embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

1. A method for detecting equipment defects, characterized in that, include: Acquire real-time images of the device; This equipment is an electrical device; The Darknet-53 network model was trained based on multiple historical fault images of the device to generate the first feature extraction model; The VGG16 network model was trained based on multiple historical fault images of the device to generate a second feature extraction model. The real-time image is input into the first feature extraction model and the second feature extraction model to generate feature extraction results; The feature extraction results are input into the category prediction model to generate the target category; The feature extraction results are input into a multi-scale prediction model to generate multiple target bounding boxes; Defects in the device are detected based on the target category and the multiple target bounding boxes.

2. The method as described in claim 1, characterized in that, Also includes: Multiple historical fault images of the equipment are input into the first feature extraction model and the second feature extraction model to generate multiple historical feature extraction results; The GBDT model is trained using the results of the extraction of the aforementioned historical features; The class prediction model is generated by fitting the class loss error in the training results based on the log-likelihood loss function.

3. The method as described in claim 1, characterized in that, Also includes: Multiple historical fault images of the equipment are input into the first feature extraction model and the second feature extraction model to generate multiple historical feature extraction results; The multi-scale classification model is trained using the results of the extraction of the aforementioned historical features; The loss function in the training results is generated based on the probability that the anchor box contains an object. The multi-scale prediction model is generated based on the optimal solution of the loss function.

4. The method as described in claim 1, characterized in that, The real-time image is input into a first feature extraction model and a second feature extraction model to generate feature extraction results, including: The real-time image is input into the first feature extraction model to generate the first feature result; The real-time image is input into the second feature extraction model to generate the second feature result; The first feature result and the second feature result are concatenated to generate the feature extraction result.

5. The method as described in claim 1, characterized in that, The Darknet-53 network model is trained based on multiple historical fault images of the device to generate the first feature extraction model, including: Image annotation processing is performed on multiple historical fault images of the device; The labeled historical fault images are input into the Darknet-53 network model, which includes 52 convolutional layers, batch normalization layers, and hop connections. The Darknet-53 network model performs residual training on the sub-networks based on multiple historical fault images to generate the first feature extraction model.

6. The method as described in claim 1, characterized in that, The VGG16 network model is trained based on multiple historical fault images of the device to generate the second feature extraction model, including: Image annotation processing is performed on multiple historical fault images of the device; The labeled historical fault images are input into the VGG16 network model; The VGG16 network model is trained on multiple historical fault images to generate the second feature extraction model.

7. The method as described in claim 2, characterized in that, The class prediction model is generated by fitting the class loss error in the training results to the log-likelihood loss function, including: The loss during training is generated by using the difference between the predicted probability value and the true probability value of the category; Fit the error of the loss described in the log-likelihood loss function; The category prediction model is generated when the error of the loss is minimized.

8. The method as described in claim 3, characterized in that, The multi-scale classification model is trained using the extracted historical features, including: During training, the bounding boxes in the multiple historical feature extraction results are clustered based on a clustering algorithm; Multiple anchor boxes are generated based on the clustering results; Multiple target borders are generated based on the multiple anchor point boxes.

9. The method as described in claim 8, characterized in that, The loss function in the training results is generated based on the probability that the anchor box contains an object, including: The probability of an object being contained in an anchor box is predicted based on logistic regression to generate the loss function in the training results.

10. A device for detecting equipment defects, characterized in that, include: The image module is used to acquire real-time images of the device; the device is a power equipment. The feature module is used to input the real-time image into the first feature extraction model and the second feature extraction model to generate feature extraction results; the Darknet-53 network model is trained based on multiple historical fault images of the device to generate the first feature extraction model; The VGG16 network model was trained based on multiple historical fault images of the device to generate the second feature extraction model. The category module is used to input the feature extraction results into the category prediction model to generate the target category; The bounding box module is used to input the feature extraction results into the multi-scale prediction model to generate multiple target bounding boxes; The detection module is used to detect defects in the device based on the target category and the plurality of target bounding boxes.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images