Protection pressing plate on-off state detection method for identifying pressing plate name based on OCR model

Abstract

The invention discloses a protection pressing plate on-off state detection method for identifying a pressing plate name based on an OCR model, and the method comprises the steps: obtaining a protection pressing plate image, constructing a sample set, and dividing the sample set into a training sample set and a test sample set; performing image processing on the obtained sample set, generating a target candidate box, calculating a candidate box score value S1, a color feature score S2 and a spatial feature score S3, and determining a commissioning and quitting score S; according to the obtained on-off score S, adopting an XGBoost model to obtain a threshold value Sth of the on-off score, comparing S with Sth, and judging according to a comparison result to obtain the on-off state of the pressing plate; identifying the name of the protection pressing plate by using an OCR neural network; and taking the obtained on-off state of the protection pressing plate and the obtained name of the protection pressing plate as a final detection result of the on-off state of the protection pressing plate. According to the invention, intelligent detection of the state of the protection pressing plate is realized, the pressing plate detection efficiency is improved, stable operation of a power system is guaranteed, and the problems of low efficiency and error proneness of manual checking of switching of the relay protection pressing plate are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of relay protection, and particularly relates to a method for detecting the switching state of protection pressure plates based on OCR model to identify the names of pressure plates. Background Art

[0002] In the actual operation scenario of the current relay protection field in hydropower plants, the traditional method of manually checking the switching state of protection pressure plates has defects. There are a large number of relay protection pressure plates in hydropower plants. When substation operation personnel perform the checking task, they need to check each pressure plate one by one. This process not only has an extremely heavy workload, requiring one-by-one verification with a huge workload, but also highly relies on manual operation and has strong mechanical repeatability. Due to factors such as long-term fatigue operation and difficulty in continuously concentrating attention in the personal mental state of the staff, situations such as incorrect verification and missed verification are likely to occur. This not only greatly reduces work efficiency, but also poses a serious threat to the accuracy and safety of the operation of the power system. Once an error occurs during the verification process, when a fault occurs in the power system, the relay protection equipment may not be able to respond in a timely and accurate manner, thereby causing the scope of the power outage accident to expand, resulting in more serious economic losses and social impacts.

[0003] With the rapid development of computer technology, the exploration of using computer vision and artificial intelligence technologies to intelligently identify protection pressure plates has gradually emerged. However, there are still bottlenecks to be broken through in the existing related technical solutions. First, the complexity of many algorithms is relatively high, making it difficult to meet the demand for rapid data processing in the power operation scenario where real-time requirements are extremely strict. In the emergency moment of a sudden power system failure, timely and accurate judgment of the protection pressure plate status is crucial for making precise decisions. However, the long processing process of existing complex algorithms will seriously delay the timing and cannot provide timely and effective support for subsequent fault handling operations. Second, the production of the dataset and the model training stage often consume a large amount of time. In practical applications, a hydropower plant may need to quickly update and adjust the protection pressure plate configuration according to the actual situation. However, the current technology has obvious deficiencies in this regard, and often when the model is available, the layout of the power system has already changed, resulting in a significant reduction in the applicability of the model. Moreover, the dataset constructed for a specific scenario and the trained model have poor universality. Once the application scenario changes, such as replacing a different hydropower plant or the configuration of power components is slightly different, it is necessary to re-make the dataset and train the model, greatly increasing the cost investment. Third, most of the existing technologies focus on the discrimination of the pressure plate's on / off state, but generally neglect the identification of the protection pressure plate name. In actual operation and maintenance work, when staff need to quickly locate a specific pressure plate and check the pressure plates related to the fault, the inability to accurately identify the pressure plate name will make the work difficult to carry out, greatly hindering the efficient development of operation and maintenance work and having an adverse impact on subsequent intelligent management and data analysis. And the present invention can make up for these deficiencies and bring innovation to the relay protection pressure plate management in hydropower plants with its unique advantages. Summary of the Invention

[0004] The technical problem of the present invention is that the on / off of the existing relay protection pressure plate relies on manual verification, which has the disadvantages of low efficiency and poor accuracy, and is significantly restricted by factors such as the mental state of the staff. On the other hand, existing related technologies such as the protection pressure plate image preprocessing algorithm and the detection method based on the Mask_RCNN algorithm have problems such as complex algorithms, difficulty in meeting real-time processing requirements, long time-consuming for dataset production and model training, weak universality, and imperfect functions, lacking the identification of the pressure plate name.

[0005] The object of the present invention is to solve the above problems and provide a detection method for the on / off state of protection pressure plates that identifies the pressure plate name based on the OCR model, which solves the problems of low efficiency and poor accuracy in relying on manual verification for the on / off state of protection pressure plates in hydropower plants, as well as the problems of complex algorithms, poor real-time performance, weak universality, and imperfect functions in existing related technologies, and realizes intelligent identification by using computer vision to improve work efficiency and quality.

[0006] To achieve the above object, the technical solution provided by the present invention is a method for detecting the input and output state of a protection pressure plate based on an OCR model to identify the name of the pressure plate, including the following steps: Step 1: Obtain the protection pressure plate image, construct a sample set, and divide the sample set into a training sample set and a test sample set; For the acquisition of the protection pressure plate image, the obtained protection pressure plate image is an original image, including the protection pressure plate and the yellow label above the protection pressure plate. The yellow label contains the protection pressure plate number and name information. Use the monitoring video or the track inspection robot to collect the video image of the target protection pressure plate, transmit it to the server through the network, manually mark the input state of the sample, and divide the sample into a training sample set and a test sample set according to a ratio; Step 2: Perform image processing on the sample set obtained in Step 1 to generate a target candidate box, calculate the candidate box score value S1, the color feature score S2, and the spatial feature score S3, and determine the input and output score S; Step 2 includes the following sub-steps: Step 2.1: Convert the protection pressure plate image obtained in Step 1 into a grayscale image; Step 2.2: Use the median filtering algorithm to denoise the grayscale image obtained in Step 2.1; Step 2.3: Convert the protection pressure plate image obtained in Step 1 into an HSV image; Step 2.4: Extract the foreground area of the protection pressure plate according to the HSV color characteristics of the protection pressure plate; Step 2.5: Use the Canny edge detection algorithm to extract the edge information from the grayscale image obtained after being processed in Step 2.2; Step 2.6: Combine the result of the foreground area of the protection pressure plate obtained in Step 2.4 and the Canny edge detection result obtained in Step 2.5 through a logical OR operation to obtain the complete foreground area of the protection pressure plate; Step 2.7: Generate a target candidate box for the foreground area of the protection pressure plate obtained in Step 2.6 by using the connected component analysis method; Preferably, Step 2.7 includes the following sub-steps: Step 2.7.1: Sort the candidate boxes according to the area size; Step 2.7.2: Select the candidate box with the largest area and compare the intersection over union (IOU) with other candidate boxes respectively. If the IOU overlap is greater than or equal to 0.3, delete the smaller candidate box; Step 2.7.3: Starting from the second largest candidate box, repeat the method of Step 2.7.2 to delete the smaller candidate boxes until the end.

[0007] Step 2.8: Calculate the width-to-height ratio of the candidate bounding boxes obtained in Step 2.7 as R1, and perform a function mapping on R1 to obtain the candidate bounding box score S1; Preferably, for the candidate bounding boxes, calculate the width-to-height ratio R1 of the candidate bounding boxes, perform a function mapping on R1 to obtain the candidate bounding box score S1. The candidate bounding box score S1 represents the confidence of the platen input calculated based on the width-to-height ratio. The calculation formula for the candidate bounding box score S1 is: ; Step 2.9: Calculate the color feature score S2 based on the foreground image of the protection platen framed by the candidate bounding boxes obtained in Step 2.7; Preferably, calculate the center point coordinates of the foreground image of the protection platen as (x, y), the maximum coordinate value of the foreground image of the protection platen in the vertical direction is y_max, select the coordinate points (x, y), (x, y + 1).... (x, y_max) as a set, calculate the average pixel of this set in the RGB space, calculate the Euclidean distance D between the average pixel and the coordinate point (x, y) in the RGB space, and obtain the color feature score S2 through the function mapping of the Euclidean distance D. The calculation formula for the function f(D) is: ; Step 2.10: Extract the foreground image of the yellow label from the HSV image obtained in Step 2.3, and calculate the spatial feature score S3 based on the foreground image of the yellow label and the foreground image of the protection platen framed by the candidate bounding boxes obtained in Step 2.7; Preferably, obtain the line L1 connecting the lower left coordinate and the lower right coordinate of the yellow label, obtain the line L2 connecting the upper left coordinate and the lower left coordinate of the foreground image of the protection platen, calculate the angle α between L2 and L1, and obtain the spatial feature score S3 through the function transformation of the angle α. The calculation formula for the spatial feature score S3 is: ; Step 2.11: Calculate the input / output score S: S = k1*S1 + k2*S2 + (1 - k1 - k2)*S3, where k1 and k2 are the weights of S1 and S2 respectively.

[0008] Step 3: According to the input / output score S obtained in Step 2, use the XGBoost model to obtain the threshold S of the input / output score th , compare S with S th in size, and judge the input / output state of the platen based on the comparison result; Calculating the threshold S of the input / output score using the XGBoost model according to the input / output score S in Step 2 th , and judging the input / output state specifically includes the following steps Step 3.1: Determine the parameters of the XGBoost model; Step 3.2: Train the XGBoost model with the training sample set; Step 3.3: Test the XGBoost model with the test sample set; Step 3.3.1: Set the threshold search range and initialize the threshold variable; Step 3.3.2: Use the trained XGBoost model to predict the test set, and judge the input / output status according to the current threshold S th and compare the input / output scores; Step 3.3.3: Traverse all thresholds S th ; Step 3.4: Analyze the curve of the metric changing with the threshold S th and select the optimal threshold; The XGboost algorithm contains multiple decision trees, which combine a group of weak learners into a strong learner. Given a data set , is an input vector containing S1, S2, S3, is the sample label, the input status is recorded as 1, and the output status is recorded as 0, is the sample serial number, with a total of samples, and represents the th regression tree, is the parameter set for the regression tree; The predicted value of the XGboost algorithm is expressed as: ; where FK( ) represents the output function of the XGboost algorithm, is the number of regression trees; solve by minimizing the objective function, and the objective function is constructed according to the loss function of the actual value and the predicted value , ; ; In the formula, is the objective function; ( ) is the loss function; is the regularization function; , are both hyperparameters; is the number of leaves; is the weight of the leaf node; Train the XGBoost model with the training sample set and rewrite the objective function as: ; In the formula, represents the objective function of the s-th round of training; represents the predicted value of the s-th round; n represents the number of samples; represents the predicted value of the (s - 1)-th round of training; For the loss function in the objective function, expand it using Taylor's formula to obtain the second derivative: ; In the formula , respectively represent the first-order and second-order coefficients; ; ; Set the threshold search range from 0 to 1, initialize the threshold variable, and the search step size is 0.01; Use the trained model for the test sample set to obtain the on / off score predicted by the model, and judge the on / off state according to the current threshold compared with the on / off score; According to the set search step size, change the value of the threshold in turn, repeat step 3.3.2, judge the on / off state on the test set for all threshold values, and record the accuracy of the model under each threshold; Analyze the accuracy change curve with the threshold, and select the optimal threshold; The accuracy The calculation formula is: ; Step 4: Use the OCR neural network to recognize the name of the protection pressure plate; The method of using the OCR neural network to recognize the name of the protection pressure plate is: Step 4.1: For each candidate box, crop the corresponding part of the original image of the candidate box. The cropped original image part is the original foreground image of the protection pressure plate, and calculate the angle between the line connecting the upper left corner coordinate and the lower right corner coordinate of the original foreground image of the protection pressure plate and the vertical direction; Step 4.2: Adaptively rotate the corresponding candidate box to make the pressure plate basically in the vertical direction according to the angle obtained in step 4.1; Step 4.3: Use the PaddleOCR tool to recognize the name of the protection pressure plate for the rotated cropped image obtained in step 4.2.

[0009] Step 5: Take the on / off state of the protection pressure plate obtained in step 3 and the name of the protection pressure plate obtained in step 4 together as the final detection result of the on / off state of the protection pressure plate.

[0010] Compared with the prior art, the beneficial effects of the present invention include: (1) The foreground extraction method of the protection pressure plate of the present invention by fusing HSV color features and Canny edge features processes and generates candidate boxes and feature data, calculates the withdrawal score of the pressure plate according to the feature data, determines the threshold of the withdrawal score using the XGBoost model, judges the withdrawal state of the pressure plate according to the withdrawal score of the pressure plate and its threshold, and then uses the OCR neural network to identify the name of the pressure plate, obtaining the detection result of the withdrawal state of the pressure plate with the name of the pressure plate, realizing the intelligent detection of the state of the protection pressure plate, improving the efficiency and intelligence level of the pressure plate detection, and ensuring the stable operation of the power system; the present invention solves the problems of low efficiency and error-proneness in manual verification of the withdrawal of relay protection pressure plates and the defects of existing related technologies.

[0011] (2) Different from the prior art that only relies on single or a small number of simple features to judge the withdrawal state of the pressure plate, the present invention innovatively uses three key features: the aspect ratio of the candidate box, the color feature of the protection pressure plate, and the foreground space feature of the protection pressure plate. This multi-dimensional and highly targeted feature combination can capture the subtle state changes of the protection pressure plate more comprehensively and accurately, achieving real-time detection effects, making the accuracy of the withdrawal state judgment a qualitative leap compared with the prior art, and providing a more reliable basis for the operation and maintenance of the power system.

[0012] (3) The present invention uses the XGBoost model to obtain the threshold of the withdrawal score S, which is more rigorous and scientific. From setting the search range, initializing variables, to model prediction, traversing the threshold and selecting the best threshold, the threshold can accurately adapt to different working conditions of the pressure plate, significantly improving the accuracy of the withdrawal state judgment, reducing the risk of power outage accidents, and enhancing the intelligent level of power operation and maintenance, overcoming the drawbacks of arbitrary and inaccurate threshold determination in the prior art.

[0013] (4) In the aspect of identifying the name of the protection pressure plate, the present invention makes the pressure plate basically in the vertical direction by angle adaptive rotation, solving the recognition problem caused by the shooting angle. The OCR neural network is used to identify the rotated image, and the name of the pressure plate is fed back to the original image. This method greatly improves the recognition accuracy compared with the existing name recognition technologies that ignore or are simple and rough, facilitating the operation and maintenance personnel to quickly locate and check the pressure plate, and helping the intelligent and efficient management of relay protection. Description of the Drawings

[0014] The following further illustrates the present invention in conjunction with the drawings and embodiments.

[0015] Figure 1 It is a schematic flow chart of the method for detecting the withdrawal state of the protection pressure plate based on the OCR model in Embodiment 1 of the present invention.

[0016] Figure 2 It is a flow chart of the method for extracting feature data from the sample set in Embodiment 1 of the present invention.

[0017] Figure 3 This is the foreground area effect diagram of the extraction of the red protection pressure plate in the first embodiment of the present invention.

[0018] Figure 4 This is the comparison diagram of the effect of removing redundant candidate boxes in the first embodiment of the present invention.

[0019] Figure 5 This is the schematic diagram of the detection result of the input and withdrawal state of the protection pressure plate in the first embodiment of the present invention.

[0020] Figure 6 This is the schematic diagram of the detection result of the input state of the protection pressure plate in the first embodiment of the present invention, which is the input result.

[0021] Figure 7 This is the schematic diagram of the detection result of the input and withdrawal state of the protection pressure plate in the first embodiment of the present invention, which is the withdrawal result. Detailed implementation manners

[0022] Embodiment 1: As Figure 1 shown, the method for detecting the input and withdrawal state of the protection pressure plate based on the OCR model to identify the name of the pressure plate includes the following steps: Step 1: Obtain the protection pressure plate image, construct a sample set, and divide the sample set into a training sample set and a test sample set; Obtain the protection pressure plate image as the original image, which includes the protection pressure plate and the yellow label above the protection pressure plate. The yellow label contains the protection pressure plate number and name information. Use the monitoring video or the track inspection robot to collect the video image of the target protection pressure plate, transmit it to the server through the network, manually mark the input state of the sample, use the random number generation algorithm to shuffle the sample set, and allocate it to the training sample set and the test sample set according to the ratio of 8:2; Step 2: Perform image processing on the sample set obtained in Step 1 to generate target candidate boxes, calculate the candidate box score value S1, the color feature score S2, and the spatial feature score S3, and determine the input and withdrawal score S; As Figure 2 shown, Step 2 includes the following sub-steps: Step 2.1: Convert the protection pressure plate image obtained in Step 1 into a grayscale image. The implementation method uses the cvtColor (img1, COLOR_BGR2GRAY) function of the OpenCV open source library, where img1 represents the input image data, and COLOR_BGR2GRAY represents the conversion of the image from the BGR color space to the grayscale color space; Step 2.2: Denoise the grayscale image obtained in Step 2.1 using the median filtering algorithm. The implementation uses the medianBlur(img2, 5) function of the OpenCV open-source library, where img2 represents the input grayscale image data; Step 2.3: Convert the protection pressure plate image obtained in Step 1 into an HSV image. The implementation uses the cvtColor(img3, COLOR_BGR2HSV) function of the OpenCV open-source library, where img3 represents the input image data, and COLOR_BGR2GRAY indicates that the image is converted from the BGR color space to the HSV color space; Step 2.4: Extract the foreground area of the protection pressure plate using the HSV color features of the protection pressure plate. The implementation uses the inRange(hsv_img, lower, upper) function of the OpenCV open-source library. Here, hsv_img is the image that has been converted to the HSV color space, lower is used to set the lower limit value of the color range to be extracted, and upper is used to set the upper limit value of the color range to be extracted; In the case where the protection pressure plate has multiple colors, set different HSV thresholds to extract the foreground multiple times, and use the OR operation to obtain the complete foreground area of the protection pressure plate. As Figure 3 shown, when extracting the red protection pressure plate, collect the HSV values of the red protection pressure plate multiple times, and set the HSV range of the inRange function to: [0, 43, 46]-[10, 255, 255], then the extraction of the red pressure plate can be completed; Step 2.5: Extract edge information from the grayscale image obtained after Step 2.2 using the Canny edge detection algorithm. The implementation uses the Canny(img4, threshold1, threshold2[,apertureSize[, L2gradient]]) function of the OpenCV open-source library. Here, img4 refers to the grayscale image after denoising processing, threshold1 sets the low threshold in the Canny edge detection, threshold2 is the high threshold in the Canny edge detection, apertureSize specifies the aperture size of the Sobel operator used when calculating the image gradient, and L2gradient is used to select the method for calculating the gradient amplitude; Step 2.6: Merge the foreground region result of the protection pressure plate obtained in Step 2.4 and the Canny edge detection result obtained in Step 2.5 through logical OR operation to obtain the complete foreground region of the protection pressure plate. The implementation method uses the bitwise_or(mask_HSV, mask_canny) function of the OpenCV open source library, where mask_HSV is the mask image of the foreground region of the protection pressure plate extracted, and mask_canny represents the mask image containing the edge information of the protection pressure plate obtained through the Canny edge detection algorithm; Step 2.7: Generate candidate boxes for the foreground region of the protection pressure plate merged in Step 2.6 by means of connected component analysis. The implementation method uses the findContours(mask, mode=cv2.RETR_EXTERNAL, method=cv2.CHAIN_APPROX_SIMPLE) and boundingRect(coutour) functions of the OpenCV open source library, where mask is the complete foreground region of the protection pressure plate in Step 2.6, mode=cv2.RETR_EXTERNAL means only detecting the outermost contours, method=cv2.CHAIN_APPROX_SIMPLE means only retaining the endpoint information of the contours, and coutour represents the information of a single contour detected and processed to a certain extent; Generate candidate boxes for the complete foreground region of the protection pressure plate in Step 2.6 by means of connected component analysis. The method for clearing redundant candidate boxes is as follows: Step 2.7.1: Sort the candidate boxes according to the area size; Step 2.7.2: Select the candidate box with the largest area and compare the intersection over union (IOU) with other candidate boxes respectively. If the IOU overlap is greater than or equal to 0.3, delete the smaller candidate box; Step 2.7.3: Starting from the second largest candidate box, repeat the method in Step 2.7.2 to delete the smaller candidate boxes until the end.

[0023] The comparison of the effects of clearing redundant candidate boxes is as Figure 4 shown; Step 2.8: Calculate the width-to-height ratio of the candidate box obtained in Step 2.7 as R1, and perform function mapping on R1 to obtain the candidate box score S1; For the candidate box, calculate the width-to-height ratio R1 of the candidate box, and perform function mapping on R1 to obtain the candidate box score S1. The candidate box score S1 represents the confidence of the pressure plate input calculated by the width-to-height ratio. The calculation formula of the candidate box score S1 is: ; Step 2.9: Calculate the color feature score S2 based on the foreground image of the protection pressure plate framed by the candidate bounding boxes obtained in Step 2.7; Calculate the center point coordinates of the foreground image of the protection pressure plate as (x, y), and the maximum coordinate value of the foreground image of the protection pressure plate in the vertical direction is y_max. Select the coordinate points (x, y), (x, y + 1).... (x, y_max) as a set, and calculate the average pixel of this set in the RGB space. The calculation formula is as follows: ; In the formula respectively represent the coordinate points The pixel values of the red, green, and blue channels.

[0024] Calculate the Euclidean distance D between the average pixel and the coordinate point (x, y) in the RGB space. The calculation formula is as follows: ; In the formula, , , are the red, green, and blue channel values of the average pixel respectively.

[0025] The Euclidean distance D is mapped through the function f(D) to obtain the color feature score S2. The calculation formula of the function f(D) is: ; Step 2.10: Extract the foreground image of the yellow label from the HSV image obtained in Step 2.3. Calculate the spatial feature score S3 based on the foreground image of the yellow label and the foreground image of the protection pressure plate framed by the candidate bounding boxes obtained in Step 2.7; Obtain the line L1 connecting the lower left coordinate and the lower right coordinate of the yellow label, obtain the line L2 connecting the upper left coordinate and the lower left coordinate of the foreground image of the protection pressure plate, calculate the angle α between L2 and L1, and perform a function transformation on the angle α to obtain the spatial feature score S3. The calculation formula of the spatial feature score S3 is: ; Step 2.11: Calculate the switching score S: S = k1*S1 + k2*S2 + (1 - k1 - k2)*S3, where k1 and k2 are the weights of S1 and S2 respectively.

[0026] Step 3: According to the switching score S obtained in Step 2, use the XGBoost model to obtain the threshold S of the switching score th , compare S with S th in size, and judge the switching state of the pressure plate according to the comparison result; Use the XGBoost model to calculate the threshold of the investment / withdrawal score S, which specifically includes the following steps: Step 3.1: Determine the parameters of the XGBoost model; Step 3.2: Train the XGBoost model with the training sample set; Step 3.3: Test the XGBoost model with the test sample set; Step 3.3.1: Set the threshold search range and initialize the threshold variable; Step 3.3.2: Use the trained XGBoost model to predict the test set, and judge the investment / withdrawal status by comparing the investment / withdrawal score according to the current threshold S th ; Step 3.3.3: Traverse all thresholds S th ; Step 3.4: Analyze the change curve of the index with the threshold S th and select the optimal threshold; The XGboost algorithm contains multiple decision trees, which combine a group of weak learners into a strong learner. Given a data set , is the input vector containing S1, S2, S3, is the sample label, the input state is recorded as 1, and the withdrawal state is recorded as 0, is the sample serial number, with a total of samples. Use to represent the th regression tree, is the parameter set for the regression tree; The predicted value of the XGboost algorithm is expressed as: ; In the formula, FK( ) represents the output function of the XGboost algorithm, is the number of regression trees; solve by minimizing the objective function , and the objective function is constructed according to the loss function of the actual value and the predicted value , ; ; In the formula, is the objective function; ( ) is the loss function; is the regularization function; , are both hyperparameters; is the number of leaves; is the weight of the leaf node; Train the XGBoost model with the training sample set, and rewrite the objective function as: ; In the formula, represents the objective function of the s-th round of training; represents the predicted value of the s-th round; n represents the number of samples; represents the predicted value of the (s - 1)-th round of training; For the loss function in the objective function, expand it using the Taylor formula to obtain the second derivative: ; In the formula , represent the first-order and second-order coefficients respectively; ; ; Set the threshold search range from 0 to 1, initialize the threshold variable, and the search step size is 0.01; Use the trained model for the test sample set to obtain the on / off score predicted by the model, and judge the on / off state according to the current threshold compared with the on / off score; According to the set search step size, change the value of the threshold in turn, repeat step 3.3.2, judge the on / off state on the test set for all threshold values, and record the accuracy of the model under each threshold; Analyze the accuracy change curve with the threshold, and select the optimal threshold; The accuracy The calculation formula is: ; Step 4: Use the OCR neural network to identify the name of the protection pressure plate; The method of using the OCR neural network to identify the name of the protection pressure plate is: Step 4.1: For each candidate box, according to the upper left and lower right coordinates of the candidate box, use the image cropping technology to crop the original protection pressure plate foreground image from the original image, calculate the angle between the line connecting the upper left and lower right coordinates of the original protection pressure plate foreground image and the vertical direction, and then obtain the angle through the arctangent function; Step 4.2: According to the angle obtained in Step 4.1, use the image rotation function provided by the Pillow library to adaptively rotate the corresponding candidate box so that the pressure plate is basically in the vertical direction; Step 4.3: Use the PaddleOCR tool to identify the name of the protection pressure plate for the rotated cropped image obtained in Step 4.1.

[0027] Step 5: Take the protection pressure plate's put-in and take-out status obtained in Step 3 and the protection pressure plate name obtained in Step 4 together as the final detection result of the protection pressure plate's put-in and take-out status; The detection result of the protection pressure plate's put-in and take-out status is as Figures 5-7 shown.

[0028] Example 2: Step 1: Obtain the protection pressure plate image, construct the sample set, and divide the sample set into a training sample set and a test sample set; The obtained protection pressure plate image is the original image, including the protection pressure plate and the yellow label on the upper part of the protection pressure plate. The yellow label contains the protection pressure plate number and name information. Use the monitoring video or the track inspection robot to collect the target protection pressure plate video image, transmit it to the server through the network, manually mark the sample input status, use the random number generation algorithm to shuffle the sample set, and allocate it to the training sample set and the test sample set according to the ratio of 8:2; Step 2: Perform image processing on the sample set obtained in Step 1 to generate the target candidate box, calculate the candidate box score value S1, color feature score S2, spatial feature score S3, and brightness feature score S4, and determine the put-in and take-out score S; Step 2 includes the following sub-steps: Step 2.1: Convert the protection pressure plate image obtained in Step 1 into a grayscale image. The implementation method uses the cvtColor (img1, COLOR_BGR2GRAY) function of the OpenCV open source library, where img1 represents the input image data, and COLOR_BGR2GRAY represents that the image is converted from the BGR color space to the grayscale color space; Step 2.2: Use the median filtering algorithm to denoise the grayscale image obtained in Step 2.1. The implementation method uses the medianBlur(img2, 5) function of the OpenCV open source library, where img2 represents the input grayscale image data; Step 2.3: Convert the protection pressure plate image obtained in Step 1 into an HSV image. The implementation method uses the cvtColor(img3, COLOR_BGR2HSV) function of the OpenCV open source library, where img3 represents the input image data, and COLOR_BGR2GRAY represents that the image is converted from the BGR color space to the HSV color space; Step 2.4: Use the HSV color feature of the protection pressure plate to extract the foreground area of the protection pressure plate. The implementation method uses the inRange(hsv_img, lower, upper) function of the OpenCV open source library, where hsv_img is the image that has been converted to the HSV color space, lower is used to set the lower limit value of the color range to be extracted, and upper is used to set the upper limit value of the color range to be extracted; When there are multiple colors of protection pressure plates, different HSV thresholds are set to extract the foreground multiple times, and the complete foreground area of the protection pressure plate is obtained by using the OR operation. When extracting Figure 3 the red protection pressure plate, the HSV values of the red protection pressure plate are collected multiple times, and the HSV range of the inRange function is set to: [0, 43, 46]-[10, 255, 255], and the extraction of the red pressure plate can be completed; Step 2.5: Use the Canny edge detection algorithm to extract the edge information from the grayscale image obtained after Step 2.2. The implementation method uses the Canny(img4, threshold1, threshold2[,apertureSize[, L2gradient]]) function of the OpenCV open source library, where img4 refers to the grayscale image after denoising processing, threshold1 sets the low threshold in the Canny edge detection, threshold2 is the high threshold in the Canny edge detection, apertureSize specifies the aperture size of the Sobel operator used when calculating the image gradient, and L2gradient is used to select the method for calculating the gradient amplitude; Step 2.6: Combine the foreground area result of the protection pressure plate obtained in Step 2.4 with the Canny edge detection result obtained in Step 2.5 through a logical OR operation to obtain the complete foreground area of the protection pressure plate. The implementation method uses the bitwise_or(mask_HSV, mask_canny) function of the OpenCV open source library, where mask_HSV is the mask image of the foreground area of the extracted protection pressure plate, and mask_canny represents the mask image containing the edge information of the protection pressure plate obtained through the Canny edge detection algorithm; Step 2.7: Generate candidate boxes for the foreground area of the protection pressure plate merged in Step 2.6 by using the connected component analysis method. The implementation method uses the findContours(mask, mode=cv2.RETR_EXTERNAL,method=cv2.CHAIN_APPROX_SIMPLE) and boundingRect(coutour) functions of the OpenCV open source library, where mask is the complete foreground area of the protection pressure plate in Step 2.6, mode=cv2.RETR_EXTERNAL means only detecting the outermost contour, method=cv2.CHAIN_APPROX_SIMPLE means only retaining the endpoint information of the contour, and coutour represents the single contour information detected and processed after a certain process; Generate candidate boxes for the complete foreground area of the protection pressure plate in step 2.6 using the connected component analysis method. The method for removing redundant candidate boxes is as follows: Step 2.7.1: Sort the candidate boxes according to their areas; Step 2.7.2: Select the candidate box with the largest area and compare the intersection over union (IOU) with other candidate boxes respectively. If the IOU overlap is greater than or equal to 0.3, delete the smaller candidate box; Step 2.7.3: Starting from the second largest candidate box, repeat the method in step 2.7.2 to delete the smaller candidate boxes until the end.

[0029] Step 2.8: Calculate the width-to-height ratio R1 of the candidate boxes obtained in step 2.7, and perform a function mapping on R1 to obtain the candidate box score S1; For the candidate boxes, calculate the width-to-height ratio R1 of the candidate boxes, and perform a function mapping on R1 to obtain the candidate box score S1. The candidate box score S1 represents the confidence level of the pressure plate being put in through the width-to-height ratio. The calculation formula for the candidate box score S1 is: ; Step 2.9: Calculate the color feature score S2 based on the foreground image of the protection pressure plate framed by the candidate boxes obtained in step 2.7; Calculate the center point coordinates of the foreground image of the protection pressure plate as (x, y), and the maximum coordinate value of the foreground image of the protection pressure plate in the vertical direction is y_max. Select the coordinate points (x, y), (x, y + 1).... (x, y_max) as a set, and calculate the average pixel of this set in the RGB space. The calculation formula is as follows: ; In the formula respectively represent the coordinate points The pixel values of the red, green, and blue channels.

[0030] Calculate the Euclidean distance D between the average pixel and the coordinate point (x, y) in the RGB space. The calculation formula is as follows: ; In the formula, , , are respectively the red, green, and blue channel values of the average pixel .

[0031] The Euclidean distance D is mapped through the function f(D) to obtain the color feature score S2. The calculation formula for the function f(D) is: ; Step 2.10: Extract the foreground image of the yellow label from the HSV image obtained in Step 2.3. Based on the foreground image of the yellow label and the foreground image of the protection pressure plate framed by the candidate boxes obtained in Step 2.7, calculate the spatial feature score S3; Obtain the line L1 connecting the lower left coordinate and the lower right coordinate of the yellow label, obtain the line L2 connecting the upper left coordinate and the lower left coordinate of the foreground image of the protection pressure plate, calculate the angle α between L2 and L1, perform a function transformation on the angle α to obtain the spatial feature score S3, and the calculation formula for the spatial feature score S3 is: ; Step 2.11: Traverse all pixel points in the foreground area of the protection pressure plate merged in Step 2.6, accumulate their brightness values, and then divide by the total number of pixel points to calculate the average value of the pixel values in the brightness channel within the foreground area of the pressure plate and further calculate to obtain the brightness feature score S4. ; In the formula, is the brightness reference value, is a set brightness tolerance range value; The brightness feature plays an indispensable role in the identification of the input state of the protection pressure plate. From the perspective of the feature dimension, it breaks through the limitations of only relying on the aspect ratio of the candidate box, color features, and spatial features. The newly added brightness feature score S4 enriches the feature system, can more comprehensively reflect the state of the protection pressure plate, and provides more information support for accurate judgment. When dealing with complex application scenarios, in the face of uneven illumination or darker situations, the brightness feature can sensitively capture the true situation of the pressure plate, effectively improving the robustness of the detection method under different conditions and ensuring accurate identification of the input state of the protection pressure plate in various environments.

[0032] Step 2.12: The calculation formula for the input / output score T is: T = v1*S1+v2*S2+(1-k1-k2)*S3+v4*S4, where k1, k2, and v4 are the weights of S1, S2, and S4 respectively.

[0033] Step 3: According to the input / output score S obtained in Step 2, use the XGBoost model to obtain the threshold S of the input / output score th , compare the sizes of S and S th and judge the input / output state of the pressure plate according to the comparison result; Using the XGBoost model to calculate the threshold of the input / output score S specifically includes the following steps. Step 3.1: Determine the parameters of the XGBoost model; Step 3.2: Train the XGBoost model with the training sample set; Step 3.3: Test the test sample set against the XGBoost model; Step 3.3.1: Set the threshold search range and initialize the threshold variable; Step 3.3.2: Use the trained XGBoost model to predict the test set, and judge the input / output status according to the current threshold S th by comparing the input / output scores; Step 3.3.3: Traverse all thresholds S th ; Step 3.4: Analyze the curve of the metric changing with the threshold S th and select the optimal threshold; The XGboost algorithm includes multiple decision trees, which combine a group of weak learners into a strong learner. Given a data set , is the input vector containing S1, S2, S3, S4, is the sample label, the input status is denoted as 1, and the output status is denoted as 0, is the sample serial number, with a total of samples, and represents the th regression tree, is the parameter set for the regression tree; The predicted value of the XGboost algorithm is expressed as: ; In the formula, FK( ) represents the output function of the XGboost algorithm, is the number of regression trees; solve by minimizing the objective function, and the objective function is constructed based on the loss function of the actual value and the predicted value , ; ; In the formula, is the objective function; ( ) is the loss function; is the regularization function; , are both hyperparameters; is the number of leaves; is the weight of the leaf node; Train the training sample set against the XGBoost model, and rewrite the objective function as: ; In the formula, denotes the objective function for the $s$-th round of training; denotes the predicted value for the $s$-th round; $n$ denotes the number of samples; denotes the predicted value for the $(s - 1)$-th round of training; For the loss function in the objective function, expand it using Taylor's formula to obtain the second derivative: ; In the formula and denote the first-order and second-order coefficients respectively; ; ; Set the threshold search range from 0 to 1, initialize the threshold variable, and the search step size is 0.01; Use the trained model for the test sample set to obtain the on / off scores predicted by the model, and judge the on / off state according to the current threshold compared with the on / off scores; According to the set search step size, change the value of the threshold in turn, repeat step 3.3.2, judge the on / off state on the test set for all threshold values, and record the accuracy of the model under each threshold; Analyze the curve of accuracy changing with the threshold, and select the optimal threshold; The said accuracy The calculation formula is: ; Step 4: Use the OCR neural network to recognize the name of the protection pressure plate; The method of using the OCR neural network to recognize the name of the protection pressure plate is: Step 4.1: For each candidate box, according to the coordinates of the upper left corner and the lower right corner of the candidate box, use the image cropping technology to extract the original foreground image of the protection pressure plate from the original image, calculate the angle between the line connecting the coordinates of the upper left corner and the lower right corner of the original foreground image of the protection pressure plate and the vertical direction, and then obtain the angle through the arctangent function; Step 4.2: According to the angle obtained in Step 4.1, use the image rotation function provided by the Pillow library to adaptively rotate the corresponding candidate box so that the pressure plate is basically in the vertical direction; Step 4.3: Use the PaddleOCR tool to recognize the name of the protection pressure plate for the rotated cropped image obtained in Step 4.1.

[0034] Step 5: Take the on / off state of the protection pressure plate obtained in Step 3 and the name of the protection pressure plate obtained in Step 4 together as the final detection result of the on / off state of the protection pressure plate.

Claims

1. A method for detecting the insertion and withdrawal status of a protective pressure plate based on the OCR model to identify the pressure plate name, characterized in that: The following steps are involved: Step 1: Obtain a protective platen image, construct a sample set, and divide the sample set into a training sample set and a test sample set; Step 2: Perform image processing on the sample set obtained in step 1 to generate a target candidate frame, calculate the candidate frame score S1, color feature score S2 and spatial feature score S3, and determine the rejection score S; Step 3: Based on the investment and withdrawal score S obtained in step 2, the XGBoost model is used to obtain the investment and withdrawal score threshold S th , compare S with S th Size, according to the comparison result, the state of the pressing plate can be determined; Step 4: Use the OCR neural network to identify the name of the protective pressure plate; Step 5: The protection pressure plate throw-in / out status obtained in step 3 and the protection pressure plate name obtained in step 4 are taken together as the final protection pressure plate throw-in / out status detection result.

2. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 1 is characterized in that: In step 1, the protective pressure plate image is obtained, and the obtained protective pressure plate image is an original image, including the protective pressure plate and a yellow label above the protective pressure plate, and the yellow label contains the number and name information of the protective pressure plate.

3. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 2 is characterized in that: The step 2 includes the following sub-steps: Step 2.1: Convert the protection platen image obtained in step 1 into a grayscale image; Step 2.2: Use the median filter algorithm to denoise the grayscale image obtained in step 2.1; Step 2.3: Convert the protection plate image obtained in step 1 into an HSV image; Step 2.4: extracting the foreground area of ​​the protective plate according to the HSV color features of the protective plate; Step 2.5: Use the Canny edge detection algorithm to extract edge information from the grayscale image obtained by step 2.2; Step 2.6: Combine the protection plate foreground area result obtained in step 2.4 and the Canny edge detection result obtained in step 2.5 through a logical OR operation to obtain a complete protection plate foreground area; Step 2.7: Generate a target candidate frame by using a connected domain analysis method for the protection plate foreground area obtained in step 2.6; Step 2.8: Calculate the aspect ratio R1 of the candidate box obtained in step 2.7, perform function mapping on R1, and obtain the candidate box score S1; Step 2.9: Calculate the color feature score S2 based on the protective plate foreground image framed by the candidate frame obtained in step 2.7; Step 2.10: extracting a yellow label foreground image from the HSV image obtained in step 2.3, and calculating a spatial feature score S3 based on the yellow label foreground image and the protective plate foreground image framed by the candidate frame obtained in step 2.7; Step 2.11: Calculate the investment and withdrawal score S: S = k1*S1+k2*S2+(1-k1-k2)*S3, where k1 and k2 are the weights of S1 and S2 respectively.

4. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 3 is characterized in that: The step 2.7 includes the following sub-steps: Step 2.7.1: Sort the candidate boxes according to their area sizes; Step 2.7.2: Select the candidate box with the largest area and compare the IOU with other candidate boxes. If the IOU is greater than or equal to 0.3, delete the smaller candidate box. Step 2.7.3: Starting from the second largest candidate box, repeat the method in step 2.7.2 to delete smaller candidate boxes until the end.

5. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 3 is characterized in that: In step 2.8, for the candidate frame, the aspect ratio R1 of the candidate frame is calculated, and a function mapping is performed on R1 to obtain a candidate frame score S1. The candidate frame score S1 represents the confidence of the platen input calculated by the aspect ratio. The calculation formula of the candidate frame score S1 is: 。 6. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 3 is characterized in that: In the step 2.9, the coordinates of the center point of the protective pressure plate foreground image are calculated as (x, y), the maximum coordinate value of the protective pressure plate foreground image in the vertical direction is y_max, the coordinate points (x, y), (x, y+1) .... (x, y_max) are taken as a set, the mean pixel of the set in the RGB space is calculated, the Euclidean distance D between the mean pixel and the coordinate point (x, y) in the RGB space is calculated, and the Euclidean distance D is mapped by the function f(D) to obtain the color feature score S2. The calculation formula of the function f(D) is: 。 7. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 3 is characterized in that: In step 2.10, the calculation to obtain the spatial feature score S3 specifically includes: Get the line L1 between the lower left corner coordinates and the lower right corner coordinates of the yellow label, get the line L2 between the upper left corner coordinates and the lower left corner coordinates of the protection platen foreground image, calculate the angle α between L2 and L1, perform function transformation on the angle α to obtain the spatial feature score S3, and the calculation formula of the spatial feature score S3 is: 。 8. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 1, 2 or 3, characterized in that: In step 3, the method for determining the threshold of the investment and withdrawal score S according to the feature data extracted in step 2 uses the XGBoost model to calculate the threshold of the investment and withdrawal score S, and specifically includes the following steps: Step 3.1: Determine the parameters of the XGBoost model; Step 3.2: Use the training sample set to train the XGBoost model; Step 3.3: Test the XGBoost model with the test sample set; Step 3.3.1: Set the threshold search range and initialize the threshold variable; Step 3.3.2: Use the trained XGBoost model to predict the test set according to the current threshold S th Compare the investment and withdrawal scores to determine the investment and withdrawal status; Step 3.3.3: Traverse all thresholds S th ; Step 3.4: Analyze the indicator with the threshold S th Change curve and select the best threshold.

9. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 8 is characterized in that: The XGboost algorithm contains multiple decision trees, which combines a group of weak learners into a strong learner. , is the input vector containing S1, S2, S3, is the sample label, the input state is recorded as 1, and the exit state is recorded as 0. is the sample number, samples, using Indicates A regression tree, is the parameter set for the regression tree; The predicted value of the XGboost algorithm The expression is: ； Where FK( ) ​​represents the output function of the XGboost algorithm, is the number of regression trees; solve by minimizing the objective function , the objective function is based on the actual value With the predicted value The loss function is constructed. ； ； In the formula, is the objective function; ( ) is the loss function; is the regularization function; , All are hyperparameters; is the number of leaves; is the weight of the leaf node; The XGBoost model is trained with the training sample set, and the objective function is rewritten as: ； In the formula, Represents the objective function of the sth round of training; represents the predicted value of the sth round; n represents the number of samples; Represents the predicted value of the s-1th round of training; For the loss function in the objective function, use Taylor's formula to expand and get the second-order derivative: ； In the formula , denote the first-order and second-order coefficients respectively; ； ； Set the threshold search range to 0 to 1, initialize the threshold variable, and the search step size to 0.01; Use the trained model to test the sample set to obtain the investment and withdrawal score predicted by the model, and compare the investment and withdrawal score with the current threshold to determine the investment and withdrawal status; According to the set search step length, change the threshold value in turn, repeat step 3.3.2, perform a test set investment and withdrawal status judgment for all threshold values, and record the accuracy of the model under each threshold; Analyze the curve of accuracy changing with threshold and select the best threshold; The accuracy The calculation formula is: 。 10. The method for detecting the protective pressure plate insertion and withdrawal status based on the OCR model recognition of the pressure plate name according to claim 1, 2 or 3, characterized in that: In step 4, the method of using the OCR neural network to identify the name of the protective pressure plate is: Step 4.1: For each candidate frame, crop the original image portion corresponding to the candidate frame, the cropped original image portion is the original protective plate foreground image, and calculate the angle between the vertical direction and the line connecting the upper left corner coordinates and the lower right corner coordinates of the original protective plate foreground image; Step 4.2: According to the angle obtained in step 4.1, the corresponding candidate frame is adaptively rotated until the pressure plate is substantially in a vertical direction; Step 4.3: Use the PaddleOCR tool to identify the name of the protective plate on the rotated cropped image obtained in step 4.2.

Images