State Automatic Detection Method and Device Based on the Fusion of Deep Learning and Morphology

Abstract

The present invention relates to a method and device for automatic state detection based on the fusion of deep learning and morphology. First, target localization and initial type analysis are performed on the protection pressure plate image frame, and the images of type I and type II protection pressure plates are automatically recognized to obtain the recognition status. Then, the feature parameter extraction method is used to extract image features to obtain feature parameters. Furthermore, the recognition status and the feature status are fused based on the fusion information to determine the final detection status of type I and type II protection pressure plates. The method and device for automatic state detection based on the fusion of deep learning and morphology provided by the present invention can effectively detect the states of different types of protection pressure plates, with accurate recognition, high applicability, good robustness, which is helpful for the monitoring and control of the real-time state of protection pressure plates. The scheme is efficient, practical, objective and accurate, and can be used to assist in the inspection of protection pressure plates and electricity in the power system, thus better ensuring the safety of the power system.

Description

Technical Field

[0001] The present invention relates to the field of image analysis and processing, and in particular to a method and device for automatic state detection algorithm based on the fusion of deep learning and morphology. Background Art

[0002] The protection pressure plate, also known as the protection link, is the bridge and link connecting the external wiring of the protection power device, which is related to whether the protection function and action outlet can play a normal role. In the power system, the normal operation and maintenance are inseparable from the relay protection system. The protection pressure plate is an important equipment component to ensure the normal operation of the power system and an important part of the relay protection inspection. Therefore, it is of great significance to select a convenient, fast, accurate and appropriate method to monitor and inspect the state of the protection pressure plate for the safety performance evaluation of the power system. However, the traditional manual inspection has a large workload, and the long-term repetitive work is extremely easy to cause visual fatigue, resulting in low efficiency and even misoperation. With the rapid development of society, the power system is also developing towards the trend of intelligence and informatization. Therefore, using the method of artificial intelligence for auxiliary detection of protection pressure plates can reduce the manual pressure while improving the reliability of the system.

[0003] In recent years, image processing technology has been gradually used for the automatic detection of the state of protection pressure plates. For example, the prior art improves the OTSU algorithm, that is, threshold processing to eliminate the influence of the shadow area brought by light, and proposes to use the minimum circumscribed rectangle algorithm based on Graham to realize the recognition of the state of the pressure plate; or for the high-light interference brought by the reflection of the protection panel cabinet, the two-dimensional maximum inter-class is used to detect the high-light area, the high-light is eliminated by improving the sparse algorithm, and then the state is recognized by the minimum circumscribed rectangle; there are also those that realize the automatic recognition of the state of the pressure plate and its corresponding text label based on image processing technology. By obtaining the row number and column number where the pressure plate is located and comparing them with the pressure plate position information in the database, the recognition accuracy rate is obtained.

[0004] Chinese Patent CN111915509A discloses a method for identifying the state of a protection pressure plate based on image processing de-shadow optimization. The method for identifying the state of a protection pressure plate based on image processing de-shadow optimization includes graying the color image of the protection pressure plate to convert it into a gray image, and then enhancing the contrast and binaryzation of the gray image to eliminate the shadow area. The convex hull of each protection pressure plate switch is obtained through the principle of the Graham algorithm, and then the convex hull is connected into a rectangle by the principle of the minimum circumscribed rectangle to obtain the rectangle area. A threshold is set for the rectangle area. If the rectangle area is greater than the threshold, it is determined to be thrown out, otherwise it is put in. This method can effectively reduce the influence of shadow interference, has low requirements for the quality of the collected images and strong robustness; it can effectively reduce the influence of shadow interference and accurately identify the operating state of the pressure plate in the image.

[0005] In the state recognition of protection pressure plates, both traditional image processing methods and deep learning have achieved good recognition results. However, traditional image processing algorithms are vulnerable to environmental factors, such as environmental factors like light and shadow, which bring difficulties to recognition. At the same time, in the existing technology, usually only the state of one type of protection pressure plate is recognized, while in the actual operation environment, there are often multiple types of relay protection pressure plates, and the states of multiple types of protection pressure plates need to be automatically detected.

[0006] Therefore, in order to meet the automatic recognition requirements of multiple types of protection pressure plates and make the recognition and inspection processes more intuitive, efficient, and low-cost, there is an urgent need for a new recognition method to protect pressure plates and conduct power inspections in the power system, so as to better ensure the safety performance of the power system. Summary of the Invention

[0007] In order to accurately recognize the types and states of different protection pressure plates, the present invention proposes an automatic detection method for the state of protection pressure plates based on the YOLOv5 deep learning network and the fusion features of color space and morphology, realizing the effective recognition of the states of two types of protection pressure plates.

[0008] The technical solution adopted by the present invention is as follows:

[0009] An automatic state detection method based on the fusion of deep learning and morphology, used for automatically detecting protection pressure plates, characterized by including the following steps: S01, performing target localization and initial type analysis on the collected image frames of protection pressure plates, obtaining the corresponding types of target protection pressure plates in each image frame, respectively denoted as type I and type II; S02, preprocessing the collected images of type I protection pressure plates, and automatically recognizing their working states using a deep learning target network to obtain the first recognition state; S03, extracting image features of type I protection pressure plates using a feature parameter extraction method based on cubic spline interpolation and color space features to obtain the first feature parameters; S04, detecting the first feature state corresponding to the protection pressure plate based on the first feature parameters, and fusing the first recognition state and the first feature state based on the fusion information to determine the final detection state of type I protection pressure plates; S05, automatically recognizing the state of type II protection pressure plates using an improved deep learning network to obtain the second recognition state; S06, extracting image features of type II protection pressure plates using a feature parameter extraction method based on the fusion features of color space and morphology to obtain the second feature parameters; S07, determining the final detection state of type II protection pressure plates.

[0010] Further, the method for S07 to determine the final detection state of type II protection pressure plates: detecting the second feature state corresponding to type II protection pressure plates based on the second feature parameters, and fusing the second recognition state and the second feature state based on the fusion information.

[0011] Preferably, step S01 includes:

[0012] First, perform Sobel edge detection on the input image frame of the protection pressure plate. After obtaining the edge information of the picture, perform maximum and minimum filtering on it. At the same time, after obtaining the binary information of the image through global threshold processing, remove small connected regions according to the area of the connected regions, and then obtain each protection pressure plate object in the picture through the circumscribed rectangle of the connected region. Finally, determine whether the target protection pressure plate belongs to type I or type II by performing line detection on each object.

[0013] Preferably, the preprocessing of the image collected for the type I protection pressure plate in step S02 includes:

[0014] Perform Sobel operator extraction on the image in the x-direction, y-direction, and 135° direction respectively, then merge the three images proportionally to obtain the final feature image after gradient processing, and finally use the final feature image as the input of the target network for recognition.

[0015] Preferably, step S03 includes:

[0016] S301, first use the image resolution reduction method based on cubic spline interpolation for the original image;

[0017] S302, secondly, preprocess the protection pressure plate image using the histogram equalization method based on contrast-limited adaptive histogram equalization;

[0018] S303, after performing median filtering on the image, segment the image based on the HSV color space, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain the final binary segmentation result;

[0019] S304, use the method for extracting shape feature parameters based on the circumscribed rectangle parameters of the connected region to extract shape feature parameters of the protection pressure plate, so as to optimize and supplement the recognition results of deep learning.

[0020] Preferably, step S04 includes:

[0021] S401, detect the first feature state corresponding to the protection pressure plate based on the first feature parameter;

[0022] Obtain the first feature parameters h, w, S1, and S2 of the type I protection pressure plate through the foregoing steps, where h and w are respectively the length and width of the circumscribed rectangle of the protection pressure plate, S1 and S2 are respectively the feature values obtained according to the height h and width w, S1 is the area of the circumscribed rectangle of the protection pressure plate, S2 is the average value of the areas of the circumscribed rectangles of all detected protection pressure plates, and n is the number of rectangular frames, as shown in formulas (5)-(6):

[0023] S1 = h * w (5)

[0024]

[0025] Then, the state A of the type-I protection pressure plate is judged by the single-parameter threshold judgment method, and the selection formula is as shown in formula (7):

[0026]

[0027] Automatically select according to the parameter range, and two states A1 and A2 corresponding to the type-I protection pressure plate can be obtained, where A1 is the input state and A2 is the output state;

[0028] S402, automatic detection of the protection pressure plate state based on the fused state information;

[0029] After the extraction of the first characteristic parameter is completed, there is a rectangular frame information A for the target protection pressure plate r (x, y, h, w), and at the same time, there is a rectangular area B of the target protection pressure plate in the first recognition state result r ;

[0030] Perform an intersection judgment on A r and B r . If the intersection of A r and B r is empty, the final state of the type-I protection pressure plate is A i ; if the intersection of A r and B r is not empty, the final state of the type-I protection pressure plate is B i , and finally, the final detection state of all objects in the protection pressure plate image is E, as shown in formula (8):

[0031]

[0032] Preferably, the automatic recognition of the state of the type-II protection pressure plate using the improved deep learning network in step S05 includes:

[0033] In order to suppress the interference of the complex background of the type-II protection pressure plate, a channel and spatial convolutional block attention model, a channel attention module, and a spatial attention module are introduced after the CSP module of the YOLOv5 network model.

[0034] Preferably, step S06 includes:

[0035] S601, first, automatically detect the highlight area of the original image based on pixel differences, and use the fast marching algorithm to repair the highlight area of the image;

[0036] S602, secondly, use the watershed algorithm to achieve foreground segmentation of the protection pressure plate;

[0037] S603, perform image segmentation based on the HSV color space, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain the final binary segmentation result;

[0038] S604, use the method for extracting the shape feature parameters of the pressure plate based on the circumscribed rectangle of the connected domain to extract the feature parameters of the pressure plate switch, so as to optimize and supplement the classification results of deep learning.

[0039] Preferably, step S07 includes:

[0040] S701, based on the obtained second feature parameters, obtain the second feature state A' through the automatic detection method of the protection pressure plate state based on parameter features;

[0041] After inputting the width w and height h of the second feature parameters, first perform multi-parameter calculation to obtain the feature values a, S3, and S4, where a is the ratio of h to w, S3 is the area of the circumscribed rectangle, and S4 is the average area of the circumscribed rectangles of all detected objects, as shown in formulas (14)-(16):

[0042]

[0043] S3 = h * w (15)

[0044]

[0045] Then, through the selection of connected domains based on multi-parameter thresholds, remove the small connected domains in the binary region, and finally judge the second feature state A' of the type II protection pressure plate through the single-parameter threshold judgment algorithm. The judgment formula is as shown in formula (17):

[0046]

[0047] According to the automatic selection of the parameter range, three states A1', A2', and A3' of the type II protection pressure plate state can be obtained, where A1' is the withdrawn state, A2' is the standby state, and A3' is the inserted state;

[0048] S702, perform automatic detection of the pressure plate state based on the fusion state information for the second feature state A' and the second recognition state B' to obtain the final type II protection pressure plate state E';

[0049] According to the rectangular position information A r '(x, y, h, w) of the rectangular frame obtained from the second feature parameters, and at the same time, there may be a rectangular area B r ' of the target protection pressure plate in the second recognition state result. Perform an intersection judgment on A r ' and B r ';

[0050] If A r ' and B r ' have an empty intersection, then the final state is A i '; if A r ' and B r ' have a non-empty intersection, then the final state is B i ', and the final detection state of all type-II protection pressure plate objects in the image is E', as shown in Equation (18):

[0051]

[0052] A state automatic detection device based on the fusion of deep learning and morphology is used for automatically detecting the states of type-I and type-II protection pressure plates.

[0053] In the present invention, a state automatic detection method based on the fusion of deep learning and morphology effectively combines the YOLOV5 deep learning network with the fusion features of the color space and morphology for automatically detecting the types and states of protection pressure plates, including:

[0054] First, perform target localization and initial type analysis on the collected protection pressure plate image frames to obtain the corresponding types of the target protection pressure plates in each image frame, denoted as type-I and type-II respectively.

[0055] For the input type-I protection pressure plate image, the present invention proposes a state automatic detection method for type-I protection pressure plates that integrates a deep learning network. After inputting the type-I protection pressure plate image, first obtain the first feature parameter through a feature parameter extraction method based on cubic spline interpolation and color space features, and obtain the first feature state A through a state automatic detection method based on the first parameter feature; at the same time, use the YOLOv5m deep learning network to identify the state of the protection pressure plate to obtain the first recognition state B; finally, perform automatic detection of the pressure plate state on the second feature state A and the second recognition state B based on the fused state information to obtain the final state E of the type-I protection pressure plate.

[0056] In order to achieve more accurate measurement of the state of type-II protection pressure plates, the present invention proposes an automatic detection method for the state of type-II protection pressure plates based on the fusion of improved YOLOV5 with color space and morphological features. First, the basic YOLOv5 network is improved by introducing a spatial and channel convolutional attention model to enhance the saliency of the fault target to be detected, and the state of the pressure plate is detected to obtain the second recognition state B'. Secondly, the second feature parameter is obtained through the method of extracting the feature parameters of the protection pressure plate based on the fusion of color space and morphological features, and the second feature state A' is obtained through the automatic detection method of the protection pressure plate state based on parameter features. Then, an automatic detection method for the state of type-II pressure plates that integrates a deep learning network is proposed. The second feature state A' and the second recognition state B' are used to automatically detect the state of the pressure plate based on the fused state information to obtain the final state E' of the type-II protection pressure plate.

[0057] On the other hand, the present invention also provides a state automatic detection device based on the fusion of deep learning and morphology. The device is a detection device composed of module units corresponding to the steps of any of the foregoing state automatic detection methods, and is used to automatically detect the type and state of the protection pressure plate.

[0058] To sum up, due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows:

[0059] The state automatic detection method and device based on the fusion of deep learning and morphology provided by the present invention can effectively detect the states of different types of protection pressure plates, with accurate recognition, high applicability, and good robustness. It helps to monitor and control the real-time state of the protection pressure plates. The solution is efficient, practical, objective, and accurate, and can be used to assist in the inspection of protection pressure plates and electricity in the power system, thereby better ensuring the safety performance of the power system. Description of the Drawings

[0060] Figure 1 is the flowchart of the automatic detection method for the state of the protection pressure plate provided by the embodiment of the present invention.

[0061] Figure 2-(a) is a diagram of a type-I protection pressure plate;

[0062] Figure 2-(b) is a diagram of a type-II protection pressure plate.

[0063] Figure 3 is the flowchart of the automatic detection method for the type of protection pressure plate based on shape features provided by the embodiment of the present invention.

[0064] Figure 4-(a) is a diagram of the withdrawn state of a type-I protection pressure plate;

[0065] Figure 4-(b) is a diagram of the inserted state of a type-I protection pressure plate.

[0066] Figure 5It is a schematic diagram of the result of gradient processing by the Sobel operator provided by an embodiment of the present invention.

[0067] Figure 6 It is a network structure diagram of the YOLOv5 provided by an embodiment of the present invention.

[0068] Figure 7 It is a schematic diagram of the detection result of the type-I protection pressure plate state based on YOLOv5 provided by an embodiment of the present invention.

[0069] Figure 8 It is a flow chart for extracting the characteristic parameters of the type-I protection pressure plate based on cubic spline interpolation and color space features provided by an embodiment of the present invention.

[0070] Figure 9-(a) is the histogram equalization effect diagram of the type-II pressure plate switch provided by an embodiment of the present invention: original image;

[0071] Figure 9-(b) is the histogram equalization effect diagram of the type-II pressure plate switch provided by an embodiment of the present invention: the type-I protection pressure plate image after being processed by cubic spline interpolation.

[0072] Figure 10-(a) is the schematic diagram of the hole filling effect provided by an embodiment of the present invention: the image before filling;

[0073] Figure 10-(b) is the schematic diagram of the hole filling effect provided by an embodiment of the present invention: the image after filling.

[0074] Figure 11 It is the binary image of the protection pressure plate after morphological processing provided by an embodiment of the present invention.

[0075] Figure 12 is the schematic diagram of the shape characteristic parameters of the protection pressure plate provided by an embodiment of the present invention.

[0076] Figure 13-(a) is the schematic diagram of the state classification of the type-II protection pressure plate provided by an embodiment of the present invention: the image in the input state;

[0077] Figure 13-(b) is the schematic diagram of the state classification of the type-II protection pressure plate provided by an embodiment of the present invention: the image in the standby state;

[0078] Figure 13-(c) is the schematic diagram of the state classification of the type-II protection pressure plate provided by an embodiment of the present invention: the image in the withdrawn state.

[0079] Figure 14 It is the improved YOLOv5 network model diagram provided by an embodiment of the present invention.

[0080] Figure 15 It is a flow chart for extracting the characteristic parameters of the type-II protection pressure plate based on the fusion features of color space and morphology provided by an embodiment of the present invention.

[0081] Figure 16-(a) is the high-gloss repair effect diagram of the pressing plate switch provided by the embodiment of the present invention: the original diagram of the pressing plate switch;

[0082] Figure 16-(b) is the high-gloss repair effect diagram of the pressing plate switch provided by the embodiment of the present invention: the effect diagram after high-gloss repair of the pressing plate switch.

[0083] Figure 17-(a) is the foreground segmentation effect diagram of the pressing plate switch provided by the embodiment of the present invention: the original diagram of the pressing plate switch;

[0084] Figure 17-(b) is the foreground segmentation effect diagram of the pressing plate switch provided by the embodiment of the present invention: the effect diagram after foreground segmentation of the pressing plate switch.

[0085] Figure 18 is the binary image of the pressing plate switch after morphological processing provided by the embodiment of the present invention.

[0086] Figure 19 is the parameter schematic diagram of the method for extracting the shape feature parameters of the protection pressing plate based on the circumscribed rectangle of the connected domain provided by the embodiment of the present invention.

[0087] Figure 20 is the schematic diagram of the automatic detection result of the pressing plate state based on the fusion state information: (a) the automatic detection result diagram of the pressing plate state based on parameter features; (b) the detection result diagram of the type II protection pressing plate state based on the YOLOv5m network; (c) the automatic detection result diagram of the protection pressing plate state based on the fusion state information.

[0088] Figure 21-(a) is the schematic diagram of the detection experimental results of the protection pressing plate based on each network state provided by the present invention: the detection result diagram of the type I protection pressing plate;

[0089] Figure 21-(b) is the schematic diagram of the detection experimental results of the protection pressing plate based on each network state provided by the present invention: the detection result diagram of the type II protection pressing plate. Detailed implementation manners

[0090] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0091] In the following embodiments, the type I protection pressing plate and the type II protection pressing plate will be selected as the target protection pressing plates to illustrate the specific solutions in detail. In other embodiments, the target protection pressing plate may also be other types of target protection pressing plates, or non-target protection pressing plates in similar scenarios. It is only the analysis object targeted by the technical solution of the present invention and shall be subject to the application of the present invention solution and the solution of the corresponding technical problems.

[0092] Example 1

[0093] As Figure 1 shown in the overall flowchart, this embodiment is a method for automatic state detection based on the fusion of deep learning and morphology. The YOLOV5 deep learning network is effectively combined with the fusion features of color space and morphology to automatically detect the type and state of protection pressure plates. The method includes the following steps:

[0094] S01. Perform target localization and initial type analysis on the collected protection pressure plate image frames, and obtain the corresponding types of the target protection pressure plates in each image frame, which are respectively denoted as type I and type II.

[0095] Before detecting the state of the protection pressure plate, it is necessary to first detect the type of the protection pressure plate. This embodiment proposes an automatic detection method for the type of protection pressure plate based on shape features to achieve the type detection of the protection pressure plate, thereby laying a foundation for subsequent state detection steps for different protection pressure plates of each type.

[0096] The types of protection pressure plates targeted in this embodiment are two common types: type I and type II, as shown in Figures 2(a) and (b) respectively.

[0097] It can be seen that the shape of the type I protection pressure plate is directly a rectangular structure, while the type II protection pressure plate includes a circular structure at the top and bottom and a square structure for connecting the two circular structures at both ends. The different structural characteristics of the two types determine that the methods for detecting their states also have different focuses.

[0098] The type detection of type I and type II protection pressure plates is carried out by using an automatic detection method for the type of protection pressure plate based on shape features, as Figure 3 shown.

[0099] First, perform sobel edge detection on the input protection pressure plate image frame. After obtaining the edge information of the picture, perform maximum and minimum filtering processing on it. At the same time, obtain the binary information of the image through global threshold processing, remove small connected regions according to the connected region area, and then obtain each protection pressure plate object in the picture through the circumscribed rectangle of the connected region. Finally, judge whether the target protection pressure plate belongs to type I or type II by performing line detection on each object.

[0100] S02. Preprocess the collected images of type I protection pressure plates, and use the deep learning target network to automatically identify their working states to obtain the first identification state.

[0101] The states of the type I protection pressure plates are divided into two types, namely "withdrawn" and "inserted", as shown in Figures 4(a) and (b) respectively.

[0102] Preprocess the collected image frames of the Type-I protection pressure plate. Mainly aiming at the state characteristics of the Type-I protection pressure plate, use the Sobel operator to perform gradient processing on the image first to obtain the processed feature image. Since there is a reverse 45-degree angle in the "withdrawn" state of the Type-I protection pressure plate, in this embodiment, the Sobel operator is extracted in the x-direction, y-direction, and 135° direction of the image respectively, and then the three images are merged in proportion to obtain the final feature image after gradient processing. Finally, the final feature image is used as the input of the target network for recognition.

[0103] The sub-images in the three directions and the merged final feature image are respectively as Figure 5 (a)-(d) shown.

[0104] In this embodiment, the YOLOv5 object detection network is used as the target network, which mainly includes four parts: the input end, the backbone network, the detection neck, and the prediction layer. Its network block diagram is as Figure 6 shown.

[0105] When the final feature image after preprocessing enters the input end, the YOLOv5 object detection network uses Mosaic data augmentation to enrich the data set; at the same time, perform adaptive anchor boxes on the data and adaptive scaling on the pictures to improve the algorithm speed.

[0106] The preprocessed image data set enters the backbone network, which mainly includes 4 modules: Focus, CBL, CSP1-X, and SSP. The CBL layer is the smallest component in YOLOV, which consists of a conv (convolution layer), a BN (batch normalization layer), and a Leaky-relu activation function.

[0107] After the dataset is sliced in the Focus structure, it enters the concat (concatenation layer) to improve the convolution speed. Then, after passing through the CBL layer, where the convolutional layer extracts the features of the input dataset and the Bn layer normalizes the structure, it finally passes through the activation function and enters the next convolutional layer. The dataset undergoes further processing through CSP1-x to optimize the gradient information in the network. CSP1-x draws on CSPNet (Cross Stage Partial Network) to divide the input into two parts. One part goes through the residual components x times and then undergoes convolution, while the other part directly undergoes convolution. Finally, these two parts are concatenated. The output data then passes through SPP (Spatial Pyramid Pooling layer), which downsamples through convolution and the max pooling layer to transform inputs of different sizes into outputs of the same size. The detection neck is the fusion part of the network, combining and transmitting the feature combinations to the prediction layer. In the Neck layer, the network adopts a structure that combines FPN and PAN. The top-down FPN transmits and fuses the high-level feature information through upsampling; the bottom-up PAN conveys the localization features, and the two are fused to enhance the network's ability to fuse features. At the same time, the CSP2-x structure is also added to the Neck network for feature fusion. Finally, at the output end, the network uses GIOU_Loss as the loss function and filters the target boxes through non-maximum suppression.

[0108] Finally, the YOLOv5m object recognition network is used to implement the state detection example of the type-I protection pressure plate as Figure 7 shown.

[0109] S03, the feature parameter extraction method using cubic spline interpolation and color space features extracts image features from the type-I protection pressure plate to obtain the first feature parameter;

[0110] Since all the protection pressure plate images are taken by the mobile phones of substation staff, the illumination in the substation main control room is uneven, and the image quality is greatly affected by conditions such as illumination, distance, and shooting angle. There are a large number of highlights and shadows interfering in the pressure plate images collected in this experimental application scenario, and the deep learning network does not learn comprehensively enough about images in various states, so there will be inaccurate classification situations. To obtain a more accurate pressure plate segmentation result, in this embodiment, a rectangular protection pressure plate feature parameter extraction method that combines color space and morphological features is further proposed on the basis of the deep learning network segmentation result to optimize the deep learning classification result of the protection pressure plate.

[0111] The overall flowchart of the feature parameter extraction method using cubic spline interpolation and color space features to extract image features from the type-I protection pressure plate is as Figure 8 shown:

[0112] S301. First, reduce the resolution of the original image based on cubic spline interpolation, so as to reduce the computational amount and improve the image processing efficiency without affecting subsequent image segmentation and feature parameter extraction.

[0113] Since the actual use environment of the protection pressure plate has a unified standard, and the background of the protection pressure plate picture is relatively simple. According to the requirements of the protection pressure plate state recognition of the present invention, it is necessary to segment the foreground and background of the picture, and then realize the extraction of the state features of the protection pressure plate. During the extraction of the pressure plate feature parameters, the requirement for the image accuracy is not high. Therefore, in order to improve the operation speed, reduce the computational amount, and improve the recognition efficiency, first use the image resolution reduction method based on cubic spline interpolation to process the input original image.

[0114] Use cubic spline interpolation to fit discrete points. Comparing 3 line connection methods, the curve fitted by the cubic spline interpolation method is more in line with the actual situation. The implementation process of the corresponding interpolation algorithm is as follows.

[0115] Assume there are n + 1 data nodes (x0, y0), (x1, y1), (x2, y2), …, (x n , y n ).

[0116] a. Calculate the step size h i = x i+1 - x i (i = 0, 1, …, n - 1);

[0117] b. Substitute the data nodes and the specified first and last endpoint conditions into the matrix equation;

[0118] c. Solve the matrix equation to obtain the second derivative value h i ;

[0119] d. Calculate the coefficients of the spline curve: a i = y i ;

[0120]

[0121] where i = 0, 1, …, n - 1;

[0122] e. In each sub - interval x i ≤ x ≤ x i+1 , create the equation

[0123] g i (x) = a i + b i (x - x i ) + c i (x - x i ) 2 + di (x - x i ) 3 (4).

[0124] S302. Secondly, use the histogram equalization method based on restricted contrast adaptation to preprocess the protection pressure plate image. This step can improve the foreground and background pixel differences in large highlight or shadow areas, reducing interference for subsequent image segmentation steps;

[0125] Since all the pressure plate pictures are taken by mobile phones, affected by the shooting equipment and the on-site environment, the background colors of the collected images vary greatly, and there are some highlight or shadow areas, which will cause serious interference to subsequent color space conversion and image foreground segmentation. Therefore, performing histogram equalization on the image in image processing is extremely crucial for subsequent pressure plate state detection.

[0126] To distinguish the foreground protection pressure plate and the background as much as possible, divide the input image into sub-regions and then perform contrast stretching. To prevent large highlight or shadow areas from having a greater impact on the overall image quality, set the threshold of the histogram distribution, and "uniformly" disperse the distribution exceeding this threshold onto the probability density distribution, thereby restricting the increase of the transfer function (cumulative histogram). To reduce the discontinuity between each sub-region, use the bilinear interpolation algorithm to connect each sub-region, and finally achieve the preprocessing of the protection pressure plate image, as shown in Figure 9.

[0127] S303. After median filtering the image, segment the image based on the HSV color space, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain the final binary segmentation result;

[0128] Since the colors of type I protection pressure plate switches are all red, yellow, and white, the extraction of color features can achieve the foreground segmentation of the pressure plate switch area. Therefore, in this embodiment, use the image segmentation method based on the HSV color space, convert the image to the HSV space and then perform threshold processing. After threshold processing, superimpose the red, yellow, and blue channels to display the complete binary processing effect.

[0129] S304. Finally, use the method of extracting shape feature parameters based on the circumscribed rectangle parameters of the connected domain to extract the shape feature parameters of the protection pressure plate, so as to optimize and supplement the classification results of deep learning.

[0130] Since there are a large number of interferences in the binary-processed protection pressure plate image, such as small isolated points, discontinuous edges of type I protection pressure plate switches, and missing centers caused by reflection at the center position of the protection pressure plate, it has a greater impact on subsequent protection pressure plate feature extraction. Therefore, it is necessary to perform morphological processing on the connected domain to obtain an effective foreground segmentation image.

[0131] In this step, first, the broken edges of the rectangular protection pressing plate in the binary image are connected through erosion and dilation, then the image is subjected to hole filling to ensure the area integrity of the switch, and finally, the isolated points in the image are removed, so as to obtain a binary image with the foreground and background completely separated.

[0132] The processing effect of hole filling is shown in Figure 10.

[0133] After performing binary threshold processing on the protection pressing plate image, it can be found that there are some small "noise points" in the segmented image, and these relatively small noise points need to be removed to facilitate feature extraction of the image later.

[0134] To remove the noise points and extract the contour information in the binary image, the areas with smaller areas are filled with black, and the effect after black filling is as Figure 11 shown.

[0135] After obtaining the segmented area of the protection pressing plate image, it is necessary to extract the circumscribed rectangle of the connected domain of the protection pressing plate. According to the 8-connected neighborhood of each pixel point, the starting coordinates x, y of the upper left corner of the circumscribed rectangle of each connected domain, the width w, height h of the circumscribed rectangle, and the number of pixels of each connected domain can be obtained.

[0136] Since the type-I protection pressing plate has two states, namely "inserted" and "withdrawn". Therefore, after segmenting the protection pressing plate image, the area feature of the circumscribed rectangle of the connected region is extracted, and by setting the threshold parameter and removing the parts with obviously non-compliant areas, the characterization of the two states can be realized. The schematic diagrams of the corresponding shape feature parameters for the two states are shown in Figures 12(a)-(b) respectively.

[0137] S04, Detect the first feature state corresponding to the protection pressing plate based on the first feature parameter, and fuse the first recognition state and the first feature state based on the fusion information to determine the final detection state of the type-I protection pressing plate.

[0138] For the input type-I protection pressing plate image, first, the first feature parameter is obtained through the feature parameter extraction method based on cubic spline interpolation and color space features, and the first feature state A is obtained through the state automatic detection method based on the first parameter feature; at the same time, the improved YOLOv5m deep learning network is used to identify the state of the protection pressing plate to obtain the first recognition state B; finally, the first recognition state and the first feature state are passed through the automatic detection method of the pressing plate state based on the fusion state information to obtain the final detection state of the protection pressing plate.

[0139] S401, Detect the first feature state A corresponding to the protection pressing plate based on the first feature parameter;

[0140] The first characteristic parameters h, w, s, and S of the protection pressure plate can be obtained through the foregoing steps, where h and w are respectively the length and width of the circumscribed rectangle of the protection pressure plate, and s and S are the characteristic values obtained according to the height h and width w. Among them, s is the area of the circumscribed rectangle of the protection pressure plate, and S is the average value of the areas of the circumscribed rectangles of all detected protection pressure plates, as shown in Equation (5-6).

[0141] s = h * w (5)

[0142]

[0143] Then, the single-parameter threshold judgment method is used to judge the state A of the Type-I protection pressure plate, and the selected formula is as shown in Equation (7):

[0144]

[0145] Automatically selected according to the parameter range, two states A1 and A2 corresponding to the Type-I protection pressure plate can be obtained, where A1 is the "put-in" state and Z2 is the "withdrawn" state.

[0146] S402, automatic detection of the pressure plate state based on the fused state information;

[0147] In order to more effectively detect the state of the I protection pressure plate, this embodiment proposes an automatic detection method for the state of the Type-I protection pressure plate that integrates a deep learning network. After inputting the image of the Type-I protection pressure plate, first, the first characteristic parameters are obtained through the method for extracting the characteristic parameters of the protection pressure plate based on the fused color space and morphological features, and the first characteristic state A is obtained through the method for automatically detecting the state of the protection pressure plate based on the parameter features; at the same time, the YOLOv5m deep learning network is used to identify the state of the protection pressure plate to obtain the first recognition state B; finally, the second characteristic state A and the second recognition state B are automatically detected for the pressure plate state based on the fused state information to obtain the final state E of the Type-I protection pressure plate.

[0148] After the extraction of the first characteristic parameters is completed, there is a rectangular frame information Ar(x, y, h, w) for the target Type-I protection pressure plate. At the same time, there may be a rectangular area Br of the target protection pressure plate in the result of the first recognition state. Combine A r with B r to perform an intersection judgment. If the intersection of A r and B r is empty, the final state of the Type-I protection pressure plate is A i ; if the intersection of A r and B r is not empty, the final state of the Type-I protection pressure plate is B i , and finally the final detection state of all objects in the protection pressure plate image is E, as shown in Equation (8):

[0149]

[0150] In this way, the final state E of the type-I protection pressure plate can be obtained.

[0151] For the type-II protection pressure plate, compared with the type-I protection pressure plate, the geometric shape of the type-II protection pressure plate and the shooting background are more complex. There are three states of the type-II protection pressure plate, as shown in Figs. 13(a)-(b), namely the "input", "standby", and "input" states.

[0152] In order to more accurately measure the state of the type-II protection pressure plate, this embodiment proposes an automatic detection method for the state of the type-II protection pressure plate based on the improved YOLOV5 and the fusion features of the color space and morphology. First, improvements are made on the yolov5 basic network, and a spatial and channel convolutional attention model is introduced to enhance the saliency of the fault target to be detected and detect the state of the pressure plate. Secondly, an algorithm for extracting the characteristic parameters of the type-II protection pressure plate based on the fusion features of the color space and morphology is proposed to obtain the characteristic parameters of the protection pressure plate. Then, an automatic detection method for the state of the type-II pressure plate that integrates the deep learning network is proposed, that is, the state of the protection pressure plate is obtained by using the automatic detection algorithm based on the parameter characteristics, and the state information is fused with the state of the deep learning result to realize the recognition of the state of the type-II protection pressure plate.

[0153] S05, automatically recognize the state of the type-II protection pressure plate by using the improved deep learning network to obtain the second recognition state;

[0154] In order to suppress the interference of the complex background of the type-II protection pressure plate, this embodiment introduces a spatial and channel convolutional attention model on the basis of the basic YOLOv5 recognition network to enhance the saliency of the fault target to be detected.

[0155] As Figure 14 shown, after the CSP module of the YOLOv5 network model, a channel and spatial convolutional block attention model, a channel attention module, and a spatial attention module are introduced to make the extracted features more refined and improve the performance of the target recognition model.

[0156] The input feature map F(H×W×C) is respectively subjected to global max pooling and global average pooling based on the width and height dimensions to obtain two feature maps of 1×1×C. Then, they are respectively fed into a two-layer neural network (MLP). The number of neurons in the first layer is C / r (r is the reduction rate), the activation function is Relu, the number of neurons in the second layer is C, and this two-layer neural network is shared. Then, the features output by the neural network MLP are summed, and then passed through a sigmoid activation operation to generate the final channel attention feature, that is, M_c. Finally, M_c and the input feature map F are multiplied element-wise to generate the input feature required by the spatial attention module. The feature map output by the channel attention module is used as the input feature map of this module. First, a global max pooling and global average pooling based on the channel are performed to obtain two feature maps of H×W×1. Then, these two feature maps are concatenated based on the channel, and then passed through a 7×7 convolution operation to reduce the dimension to 1 channel, that is, H×W×1. Then, it passes through sigmoid to generate the spatial attention feature, that is, M_s. Finally, this feature and the input feature of this module are multiplied to obtain the finally generated feature.

[0157] S06, the feature parameter extraction method based on the color space and morphological fusion features extracts image features from the type II protection pressure plate to obtain the second feature parameter;

[0158] Since all the protection pressure plate images are taken by the mobile phones of substation staff, the illumination in the substation main control room is uneven, and the image quality is greatly affected by conditions such as illumination, distance, and shooting angle. There are a large number of highlights and shadows interfering in the pressure plate images collected in the application scenario of this experiment. The deep learning network does not learn pictures in various states comprehensively enough, so the classification will be inaccurate. To obtain a more accurate state recognition result, this embodiment further proposes a type II protection pressure plate feature parameter extraction method based on the color space and morphological fusion features to optimize the deep learning classification result of the protection pressure plate.

[0159] The overall process is as Figure 15 shown. The type II protection pressure plate feature parameter extraction method based on the color space and morphological fusion features includes the following steps:

[0160] S601, first, automatically detect the highlight area of the original image based on pixel differences, and use the fast marching algorithm to repair the highlight area of the image;

[0161] Since all the protection pressure plate images are taken by mobile phones, affected by the shooting equipment and the on-site environment, the collected images are greatly interfered by reflections, shadows, etc. Therefore, in the preprocessing, first detecting and repairing the highlight area of the protection pressure plate image is extremely crucial for the subsequent detection of the protection pressure plate state.

[0162] There are significant differences in pixel values between the highlight positions of the protection pressure plate image and the surrounding adjacent areas, which are more obvious in the grayscale image. To reduce the computational amount and improve the image processing speed, a threshold range is first set for the grayscale image of the pressure plate, and a binary mask image of the highlight area is established, so as to automatically detect the highlight area of the pressure plate image.

[0163] After detecting the highlight area, according to the same position relationship between the mask image and the highlight area of the original image, an image inpainting method based on the fast marching method is used to process the highlight area of the protection pressure plate image.

[0164] This fast marching method starts to inpaint from the boundary of the non-zero area of the mask image, continuously updates the boundary of the inpainting area dynamically, gradually marches into the area, inpaints the adjacent pixels of each target pixel, and fills all the contents inside the boundary. When each target pixel is inpainted, the pixel is replaced by the normalized weighted sum of all known pixels in the neighborhood. The corresponding weight setting mechanism is as follows:

[0165] More weights are given to the pixels close to the inpainted point, the normal line close to the boundary, and the pixels located on the boundary contour.

[0166] Once a pixel is inpainted, it will move to the next nearest pixel using the fast marching method.

[0167] The weight function used in this step is shown in formula (9).

[0168]

[0169] Among them, w(p,q) is the weight function, which is used to limit the weight sizes of each pixel in the neighborhood. The calculation formula of w(p,q) is shown in formula (10).

[0170] w(p,q)=dir(p,q)·dst(p,q)·lev(p,q) (10)

[0171] Formulas (11)-(13) respectively calculate the values of the direction factor dir(p,q), the geometric distance factor dst(p,q), and the level set distance factor lev(p,q) of each target pixel point:

[0172]

[0173] Among them, d0 and T0 are the distance parameter and the level set parameter respectively, and the general values are 1. The direction factor dir(p,q) ensures that the closer to the normal direction The pixel pair contributes the most to point p; the geometric distance factor dst(p,q) ensures that the pixel points closer to point p contribute more to point p; the level set distance factor lev(p,q) ensures that the known pixel points closer to the contour line of the damaged area passing through point p contribute more to point p.

[0174] An example of the result after automatic detection of the high-light area and image repair of the protection pressure plate image is shown in Figure 16.

[0175] S602. Secondly, use the watershed algorithm to implement the foreground segmentation of the protection pressure plate, which can avoid the interference of shadows in the protection pressure plate image and achieve a better foreground segmentation effect.

[0176] After the high-light repair process of the original pressure plate image, in order to extract the pressure plate switch area, it is necessary to perform foreground segmentation on the image. Since all the protection pressure plate images are collected by mobile phone photography, the lighting conditions are poor, and due to the limitations of the mobile phone performance, the quality of the pictures is poor and there is a lot of noise. After denoising using traditional median, mean, Gaussian and other filtering methods, the image quality still cannot meet the requirements of subsequent processing. Based on the watershed algorithm, the background of the protection pressure plate image with large noise and low image quality can be turned black, so as to achieve a better foreground segmentation effect.

[0177] The watershed algorithm is mainly used to extract nearly uniform cluster-like targets from the background, so as to characterize the areas with small gray-scale changes. Since the substation control room's substation protection pressure plates are all installed on iron cabinets and the background color is relatively uniform, there are mainly shadow interferences caused by uneven lighting after the high-light repair process. Therefore, in this embodiment, a foreground segmentation algorithm based on the watershed algorithm is used. First, a random seed is selected in the background, and starting from the seed, edge points are searched according to the pixel gradient transformation within the self-set gray-scale threshold. All points that meet the threshold are regarded as the same "reservoir", and all pixel points in the "reservoir" are marked as black, while the colors of other areas remain unchanged, so as to achieve a good foreground segmentation effect. The foreground segmentation effect is shown in Figure 17.

[0178] S603. Based on the HSV color space, perform image segmentation, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain the final binary segmentation result.

[0179] Due to factors such as uneven lighting in the substation control room and other contents in the captured pictures, after foreground segmentation of some pressure plate images, there are still many irrelevant area interferences. And the colors of the pressure plate switches are all red, yellow, and blue. The extraction of color features can further achieve the segmentation of the pressure plate switch area. Therefore, in this embodiment, an image segmentation method based on the HSV color space is used to perform threshold processing after converting the image to the HSV space.

[0180] After the picture of the pressure plate switch is transformed in the HSV color space, a preliminary segmented image is obtained. However, there are still some problems in the segmented image, such as the disconnected switch edges, the incomplete filling of the switch center area, and some isolated noise points in the background, which bring great interference to the extraction of the shape feature parameters of the pressure plate in the subsequent steps. To solve the above problems, this algorithm uses the method of morphological processing. First, the binary image is processed through erosion and dilation to remove the broken edges in the binary image. Then, hole filling and isolated point removal are performed on the image to obtain a binary image in which the foreground and background are completely separated.

[0181] Through dilation and erosion, the switch edges can be connected and small spikes can be removed; in the isolated point removal step, the contour information in the binary image is first extracted, and the areas with smaller areas are filled with black; in the hole filling step, the pixel point set within the closed circle composed of the 8-connected lattice points inside the binary image is processed, and all the internal areas of the connected domain are changed to white. After morphological processing, the obtained binary image is as Figure 18 shown.

[0182] S604. Finally, the method for extracting the shape feature parameters of the pressure plate switch based on the circumscribed rectangle of the connected domain is used to extract the feature parameters of the pressure plate switch, so as to optimize and supplement the classification results of deep learning.

[0183] After obtaining the segmented area of the pressure plate switch image, it is necessary to extract the circumscribed rectangle of the connected domain of the pressure plate switch. According to the 8-connected neighborhood of each pixel point, the starting position coordinates (x, y) of the upper left corner of the circumscribed rectangle of each connected domain, the height h and width w of the circumscribed rectangle are obtained. The schematic diagram of the parameters is as Figure 19 shown.

[0184] S07. Based on the second feature parameter, the second feature state corresponding to the type II protection pressure plate is detected, and the second recognition state and the second feature state are fused based on the fusion information to determine the final detection state of the type II protection pressure plate.

[0185] To improve the state detection result of the type II protection pressure plate, this embodiment proposes an automatic detection method for the state of the type II protection pressure plate that integrates a deep learning network. After inputting the image of the type II protection pressure plate, first, the second feature parameter is obtained through the method for extracting the feature parameters of the protection pressure plate based on the fusion of the color space and morphological features, and the second feature state A' is obtained through the automatic detection method for the state of the protection pressure plate based on the parameter features; at the same time, the improved YOLOv5m deep learning network is used to identify the state of the protection pressure plate to obtain the second recognition state B'; finally, the second feature state A' and the second recognition state B' are used for automatic detection of the pressure plate state based on the fusion state information to obtain the final state E' of the type II protection pressure plate.

[0186] The overall flowchart is asFigure 14 As shown, it includes:

[0187] S701, based on the obtained second feature parameter, the second feature state A' is obtained through the automatic detection method of the protection pressure plate state based on parameter features; at the same time, the YOLOv5m deep learning network is used to identify the state of the protection pressure plate to obtain state B;

[0188] After inputting the width w and height h of the second feature parameter, multi-parameter calculations are first performed to obtain the feature values a, s, and S. Among them, a is the ratio of h to w, s is the area of the circumscribed rectangle, and S is the average value of the areas of the circumscribed rectangles of all detected objects, as shown in formulas (14)-(16).

[0189]

[0190] s = h * w (15)

[0191]

[0192] Then, through the connected component selection based on multi-parameter thresholds, small connected components in the binary region are removed, and finally, the second feature state A' of the type-II protection pressure plate is judged by the single-parameter threshold judgment algorithm, and the judgment formula is as shown in formula (17).

[0193]

[0194] According to the automatic selection based on the parameter range, three states A1', A2', and A3' of the type-II protection pressure plate state can be obtained, where A1' is the "withdrawn" state, A2' is the "standby" state, and A3' is the "inserted" state.

[0195] S702, the second feature state A' and the second recognition state B' are used to automatically detect the pressure plate state based on the fused state information to obtain the final type-II protection pressure plate state E'.

[0196] According to the rectangular position information A r '(x, y, h, w) of the obtained rectangle in the second feature parameter, and at the same time, there may be a rectangular area B r ' of the target protection pressure plate in the second recognition state result. The intersection of A r ' and B r ' is judged. If the intersection of A r ' and B r ' is empty, then the final state is A r '; if the intersection of A r ' and B r ' is not empty, then the final state is B i ', and the final detection state of all type-II protection pressure plate objects in the image is E', as shown in formula (18).

[0197]

[0198] The final detection status of the Type II protection pressure plate is shown in Figure 21.

[0199] Embodiment 2

[0200] This embodiment is a state automatic detection device based on the fusion of deep learning and morphology. The device is a detection device composed of module units corresponding to the state automatic detection method steps in any of the foregoing embodiments, and is used for automatically detecting the type and state of the protection pressure plate.

[0201] Embodiment 3

[0202] To better illustrate the excellent effects of the solution provided by the present invention, this embodiment is described through experimental test results. The experimental video data was taken at xxx of the State Grid. The experimenters took a total of 380 pictures of protection pressure plates in different environments and times, including 104 pictures of Type I protection pressure plates and 216 pictures of Type II protection pressure plates.

[0203] (1) Experimental results and analysis of the state detection of Type I protection pressure plates

[0204] In this experiment, the YOLOv5m network was used to identify the state of the Type I protection pressure plate. There were a total of 104 pictures of Type II protection pressure plates, 74 of which were used for network training, and the remaining 30 pictures were used for testing. The test results are shown in Table 1.

[0205] Table 1 State detection results of Type 2 protection pressure plates based on YOLOv5m

[0206]

[0207]

[0208] As can be seen from Table 1, among a total of 1089 Type I protection pressure plate objects, the "input" state was 309, and the "exit" state was 780. The correct detection of the target state was achieved, and the accuracy rate reached 100%.

[0209] (2) Experimental results and analysis of the state detection of Type II protection pressure plates

[0210] In this experiment, after the preliminary state detection of the protection pressure plate was performed through the YOLOv5m network, the state automatic detection algorithm of the Type II protection pressure plate that integrates the deep learning network was added to further improve the detection accuracy.

[0211] 2.3.1 Experimental results of the state detection of Type II protection pressure plates based on YOLOv5m

[0212] The experiment conducts a preliminary status detection of the protection pressure plates through the YOLOv5m network. There are a total of 216 pictures of type II protection pressure plates, and each picture contains several protection pressure plates. 151 of these pictures are used for network training, and the remaining 65 pictures are used for testing. The detection results are shown in Table 2.

[0213] Table 2 Status Detection Results of Type II Protection Pressure Plates Based on YOLOv5m

[0214]

[0215] As can be seen from Table x, among the 1709 type II protection pressure plates, there are 664 in the "spare" state, 596 in the "withdrawn" state, and 449 in the "inserted" state. Among them, the detection accuracy rate for the "withdrawn" state is 0.9161, slightly lower than the accuracy rates of 0.9246 and 0.9955 for the "spare" and "inserted" states; at the same time, the missed detection rate of 0.0772 for the "withdrawn" state is slightly higher than the missed detection rates of the "spare" and "inserted" states.

[0216] (3) Experimental Results of Status Detection of Type II Protection Pressure Plates Based on the Fusion Features of YOLOv5m and Color Space and Morphology

[0217] After adding the automatic detection method for the status of type II protection pressure plates to the fusion deep learning network, the status detection results of type II protection pressure plates are shown in Table 3.

[0218] Table 3 Experimental Results of Status Detection of Type II Protection Pressure Plates Based on the Fusion Features of YOLOv5m and Color Space and Morphology

[0219]

[0220] As can be seen from Table 3, the experiment effectively improves the status detection accuracy rate and reduces the missed detection rate. Among them, the detection accuracy rate of the protection pressure plates in the "spare" state has increased by 2.87%, and the missed detection rate has decreased by 2.4%; the detection accuracy rate of the protection pressure plates in the "withdrawn" state has increased by 5.37%, and the missed detection rate has decreased by 4.87%.

[0221] (4) Comparison with the Existing Technology

[0222] In this embodiment, the classic Faster-RCNN deep learning network and the SSD (single shot multibox detector) deep learning network are used to compare with the experimental results of YOLOv5 in this article. The accuracy rates obtained by each network are shown in Figure 21.

[0223] As can be seen from Figure 21, the state detection results of using YOLOv5 for the two types of protection pressure plates are better than those of the Faster-RCNN and SSD deep learning networks. When using Fater-RCNN to identify the state of Type I protection pressure plates, the accuracy rates of the "inserted" and "withdrawn" states are 20.06% and 66.54% respectively; when using the SSD network to identify the state of Type I protection pressure plates, the recognition accuracy rates of its "inserted" and "withdrawn" states are 63.11% and 86.02% respectively. When using Fater-RCNN to identify the state of Type II protection pressure plates, the accuracy rates of its "inserted", "withdrawn", and "spare" states are 71.06%, 66.78%, and 48.34% respectively; when using the SSD network to identify the state of Type II protection pressure plates, the accuracy rates of its "inserted", "withdrawn", and "spare" states are 82.98%, 83.39%, and 84.41% respectively. Compared with the Fater-RCNN deep learning network and the SSD deep learning network, using YOLOv5 has higher detection accuracy.

[0224] In view of the current situation of protection pressure plate detection, the present invention proposes a state automatic detection scheme based on the fusion of deep learning and morphology for detecting the states of two types of protection pressure plates. First, the classification of two types of protection pressure plates is realized through the automatic detection of protection pressure plate categories based on shape features; second, the YOLOv5m deep learning network is used to realize the state detection of Type I protection pressure plates, and a high accuracy rate is obtained; then, an automatic state detection algorithm for Type II protection pressure plates based on the fusion features of YOLOv5, color space, and morphology is proposed to realize the state detection of Type II protection pressure plates, and the accuracy rates of its three states of "spare", "withdrawn", and "inserted" are all above 95%. Experiments show that the method proposed by the present invention can effectively detect the states of different types of protection pressure plates, with accurate recognition, high applicability, and good robustness, which is helpful for the monitoring and control of the real-time states of protection pressure plates.

[0225] All features disclosed in this specification, or all steps in the disclosed methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A method for automatic state detection based on the fusion of deep learning and morphology, which is used for automatic detection of protection pressure plates, and is characterized in that, The method includes the following steps: S01, perform target positioning and initial type analysis on the collected protection pressure plate image frames, obtain the corresponding types of the target protection pressure plates in each image frame, and record them as type I and type II respectively; S02, preprocess the collected images of type I protection pressure plates, and use a deep learning target network to automatically identify their working states to obtain a first recognition state; S03, extract image features of type I protection pressure plates by using a feature parameter extraction method based on cubic spline interpolation and color space features to obtain a first feature parameter; S04, detect the first feature state corresponding to the protection pressure plate based on the first feature parameter, and fuse the first recognition state and the first feature state based on the fusion information to determine the final detection state of the type I protection pressure plate; S05, use an improved deep learning network to automatically identify the state of the type II protection pressure plate to obtain a second recognition state; S06, extract image features of the type II protection pressure plate by using a feature parameter extraction method based on the fusion features of color space and morphology to obtain a second feature parameter; S07, determine the final detection state of the type II protection pressure plate; Step S03 includes: S301, first use an image resolution reduction method based on cubic spline interpolation for the original image; S302, secondly, preprocess the protection pressure plate image by using a histogram equalization method based on contrast-limited adaptive histogram equalization; S303, after median filtering the image, segment the image based on the HSV color space, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain a final binary segmentation result; S304, extract the shape feature parameters of the protection pressure plate by using a method for extracting parameters of the circumscribed rectangle of the connected domain, so as to optimize and supplement the recognition result of deep learning; Step S06 includes: S601, first automatically detect the highlight area of the original image based on pixel difference, and use the fast marching algorithm to repair the highlight area of the image; S602, secondly, use the watershed algorithm to achieve foreground segmentation of the protection pressure plate; S603, segment the image based on the HSV color space, and perform morphological processing on the initial segmentation image obtained by superimposing the three-channel binary images to obtain a final binary segmentation result; S604, extract the feature parameters of the pressure plate switch by using a method for extracting the feature parameters of the pressure plate shape based on the circumscribed rectangle of the connected domain, so as to optimize and supplement the classification result of deep learning.

2. The method for automatic state detection based on the fusion of deep learning and morphology according to claim 1, and is characterized in that, The method for S07 to determine the final detection state of the type II protection pressure plate: detect the second feature state corresponding to the type II protection pressure plate based on the second feature parameter, and fuse the second recognition state and the second feature state based on the fusion information.

3. The method for automatic state detection based on the fusion of deep learning and morphology according to claim 2, and is characterized in that, Step S01 includes: First, perform Sobel edge detection on the input image frame of the protection pressure plate. After obtaining the edge information of the picture, perform maximum and minimum filtering on it. At the same time, obtain the binary information of the image through global threshold processing. Then, remove small connected regions according to the area of the connected regions. Next, obtain each protection pressure plate object in the picture through the circumscribed rectangle of the connected region. Finally, determine whether the target protection pressure plate belongs to type I or type II by performing line detection on each object.

4. The method for automatic state detection based on the fusion of deep learning and morphology according to claim 3, and is characterized in that, The preprocessing of the image collected for the type I protection pressure plate in step S02 includes: Perform Sobel operator extraction on the image in the x-direction, y-direction, and 135° direction respectively. Then, merge the three images proportionally to obtain the final feature image after gradient processing. Finally, use the final feature image as the input of the target network for recognition.

5. The method for automatic state detection based on the fusion of deep learning and morphology according to claim 2, and is characterized in that, Step S04 includes: S401, detect the first feature state corresponding to the protection pressure plate based on the first feature parameter; Obtain the first feature parameters h, w, S1, and S2 of the type I protection pressure plate through the foregoing steps, where h and w are the length and width of the circumscribed rectangle of the protection pressure plate respectively, S1 and S2 are the feature values obtained according to the height h and width w respectively, S1 is the area of the circumscribed rectangle of the protection pressure plate, S2 is the average value of the areas of the circumscribed rectangles of all detected protection pressure plates, and n is the number of rectangular frames, as shown in formulas (5)-(6): S1 = h * w (5) Then, judge the state A of the type I protection pressure plate through the single-parameter threshold judgment method, and select the formula as shown in formula (7): Automatically select according to the parameter range to obtain two states A1 and A2 corresponding to the type I protection pressure plate, where A1 is the input state and A2 is the output state; S402, automatically detect the state of the protection pressure plate based on the fused state information; After the extraction of the first characteristic parameter is completed, there is a rectangular frame information A for the target protection pressure plate r (x, y, h, w), and at the same time, there is a rectangular area B of the target protection pressure plate in the first recognition status result r ; Intersect A r with B r to perform an intersection judgment. If the intersection of A r and B r is empty, the final state of the Type I protection pressure plate is A i ; if the intersection of A r and B r is not empty, the final state of the Type I protection pressure plate is B i , and finally the final detection state of all objects in the protection pressure plate image is E, as shown in Equation (8):

6. The automatic state detection method based on the fusion of deep learning and morphology according to claim 1, characterized in that In step S05, the automatic recognition of the state of the type II protection pressure plate by using the improved deep learning network includes: In order to suppress the interference of the complex background of the type II protection pressure plate, a channel and spatial convolutional block attention model, a channel attention module, and a spatial attention module are introduced after the CSP module of the YOLOv5 network model.

7. The automatic state detection method based on the fusion of deep learning and morphology according to claim 1, characterized in that Step S07 includes: S701, based on the obtained second feature parameter, obtain the second feature state A' through the automatic detection method of the protection pressure plate state based on the parameter feature; After inputting the second feature parameters width w and height h, first perform multi-parameter calculation to obtain the feature values a, S3, and S4, where a is the ratio of h to w, S3 is the area of the circumscribed rectangle, and S4 is the average value of the areas of the circumscribed rectangles of all detected objects, as shown in formulas (14)-(16): S3 = h * w (15) Then, remove small connected regions in the binary region through the connected region selection based on multi-parameter thresholds. Finally, judge the second feature state A' of the type II protection pressure plate through the single-parameter threshold judgment algorithm, and the judgment formula is as shown in formula (17): Automatically select according to the parameter range to obtain three states A1', A2', and A3' of the type II protection pressure plate state, where A1' is the output state, A2' is the standby state, and A3' is the input state; S702. Automatically detect the state of the pressure plate for the final Type II protection pressure plate state E' by using the second characteristic state A' and the second recognition state B' based on the fused state information. According to the rectangular position information A of the obtained rectangular frame in the second characteristic parameter r '(x, y, h, w), and there may be a rectangular area B of the target protection pressure plate in the second recognition status result r ', compare A r ' with B r ' to perform an intersection judgment; If A r ' and B r ' have an empty intersection, then the final state is A i '; if A r ' and B r ' have a non-empty intersection, then the final state is B i ', and the final detection state of all type-II protection pressure plate objects in the image is E', as shown in Equation (18):

8. An automatic state detection device based on the fusion of deep learning and morphology, characterized in that The device is a detection device composed of module units corresponding to the steps of the automatic detection method for any state in claims 1-7, and is used for automatically detecting the states of Type I and Type II protection pressure plates.

Images