An intelligent inspection arrester instrument reading recognition method based on deep learning

Abstract

The present invention discloses a method for intelligent patrol lightning arrester instrument reading recognition based on deep learning. The method comprises the following steps: a patrol robot collects the lightning arrester instrument image, and then detects the lightning arrester instrument image to obtain the dial area image. The dial area image is then separately recognized and processed to obtain the action number value and the range value. The lightning arrester instrument dial image is then recognized to obtain the pointer reading. Finally, the action number value and the pointer reading are returned to the lightning arrester instrument image to obtain the reading of the lightning arrester instrument image, thereby realizing real-time intelligent reading operation of the lightning arrester instrument. In the process of recognizing the lightning arrester instrument dial image to obtain the pointer reading, the method solves the problem of poor reading accuracy when the lightning arrester instrument is tilted and the dial is blurred through operations such as deep learning semantic segmentation algorithm and perspective transformation, and supports the reading of the lightning arrester action number, thereby improving the lightning arrester monitoring efficiency and enhancing the robustness of the reading.

Description

Technical Field

[0001] The present invention relates to the field of intelligent inspection, and in particular to a method for identifying meter readings of intelligent inspection lightning arresters based on deep learning. Background Art

[0002] Lightning arresters are widely used in the fields of electricity, communications, chemical industry, etc., and have become one of the indispensable equipment in modern industrial production and life.

[0003] Traditional methods rely on manual collection and recording of arrester instrument information, which consumes significant manpower and material resources. Furthermore, manual work efficiency decreases when the workload is high, and it is prone to misreading. Furthermore, some arresters operate in locations such as substations, communication towers, and conductors, making manual work a safety risk.

[0004] With the rapid development of digital image processing technology, especially deep learning methods, more and more image processing technologies and deep learning methods are being applied to lightning arrester instrument reading recognition.

[0005] In recent years, researchers have used deep learning algorithms to extract the position of lightning arrester meters in images. They also employ image processing techniques to determine the ROI (Region of Interest) for the meter's dial scale. A filtering algorithm is then used to retrieve the dial scale. Finally, the meter's reading is calculated using the needle's scale position to determine the meter's angle. This method largely addresses the challenges of traditional manual operations, such as safety risks, low efficiency, and potential for misreading. However, poor meter reading accuracy persists when the meter is tilted or the dial is blurry. Furthermore, many meter pointers on the market lack bilateral symmetry, and the aforementioned method relies on parallelism with the scale to detect the pointer's position. This makes the method ineffective for these types of meters. Finally, in a lightning arrester meter, not only the dial current needs to be read, but also the number of arrester actuations, as this is a crucial indicator for analyzing the arrester's operating status. However, existing methods often focus solely on current readings and fail to address the number of arrester actuations. Summary of the Invention

[0006] The purpose of the present invention is to provide an intelligent patrol lightning arrester instrument reading recognition method based on deep learning, which solves the above-mentioned technical problems pointed out in the prior art.

[0007] The present invention proposes a method for intelligent inspection of lightning arrester meter reading recognition based on deep learning, which includes the following steps:

[0008] Collect and obtain images of lightning arrester instruments;

[0009] Detecting the arrester instrument image to obtain a dial area image;

[0010] The dial area image includes an overall image of the arrester instrument, an image of the arrester instrument dial, and an image of the number of times the arrester instrument operates;

[0011] Identify the dial area image to obtain the action times and range values; and identify and analyze the arrester instrument dial image based on the range values to obtain the pointer reading value corresponding to the arrester instrument dial image;

[0012] The action times value and the pointer reading value are returned to the arrester instrument image to obtain the reading of the arrester instrument image.

[0013] Preferably, detecting the arrester instrument image to obtain the dial area image includes the following steps:

[0014] Preprocessing the arrester instrument image to obtain a preprocessed arrester instrument image;

[0015] Based on the pre-processed arrester instrument image, a detection is performed using a trained dial area image detection model to obtain a detection result;

[0016] The test results include the arrester instrument test results, the arrester instrument dial test results, and the arrester instrument action times test results;

[0017] A dial area image is captured from the pre-processed arrester instrument image according to the detection result.

[0018] Preferably, identifying the dial area image to obtain the action count value and the range value includes the following steps:

[0019] Based on the arrester instrument dial image, a range image is obtained by performing detection using a range detection model;

[0020] The range image is a digital image above the dial scale image;

[0021] Based on the range image and the arrester instrument action times image, a pre-trained character recognition model is used to perform detection to obtain the range value and the action times value.

[0022] Preferably, the acquiring of the range value and the action number value by detecting based on the range image and the arrester instrument action number image through a pre-trained character recognition model comprises the following steps:

[0023] Traversing the range image and the arrester instrument action times image, randomly selecting any image as a target image; performing a feature extraction operation on the target image to obtain a feature sequence;

[0024] Acquiring image association relationship information in the target image through the feature sequence;

[0025] The feature sequence is converted into a numerical value based on the image association relationship information.

[0026] Preferably, the step of identifying and analyzing the arrester instrument dial image based on the range value to obtain a pointer reading value corresponding to the arrester instrument dial image comprises the following steps:

[0027] Segmenting the arrester instrument dial image to obtain a dial pointer image and a dial scale image; and adjusting the arrester instrument dial image according to the dial scale image to obtain a horizontal arrester instrument dial image;

[0028] The dial scale image includes a common scale image and a key scale image;

[0029] Acquire the scale range value corresponding to the key scale image by matching the coordinates corresponding to the scale value with the coordinates of the dial scale image;

[0030] Acquire a key area image and an important pointer image based on the dial pointer image and the dial scale image; calculate and acquire the centroid abscissa of the important pointer image and the centroid abscissa of all the key scale images based on the key scale image, the important pointer image and the arrester instrument dial image;

[0031] The pointer reading value is calculated based on the horizontal coordinate of the center of gravity of the important pointer and the horizontal coordinates of the center of gravity of all the key scales.

[0032] Preferably, the step of segmenting the arrester instrument dial image to obtain a dial pointer image and a dial scale image comprises the following steps:

[0033] Preprocessing the arrester instrument dial image to obtain a preprocessed arrester instrument dial image;

[0034] The pre-processed arrester instrument dial image is detected by a trained semantic segmentation model to obtain a dial pointer image and a dial scale image.

[0035] Preferably, the step of adjusting the arrester instrument dial image according to the dial scale image to obtain a horizontal arrester instrument dial image comprises the following steps:

[0036] Obtaining the midpoint of the common scale image; and obtaining a scale ellipse image by calculation based on the midpoint of the common scale image;

[0037] Obtaining a rotation angle by calculating the scale ellipse image;

[0038] An affine transformation is performed on the arrester instrument dial image based on the rotation angle to obtain a horizontal arrester instrument dial image.

[0039] Preferably, the acquiring of the key area image and the important pointer image based on the dial pointer image and the dial scale image comprises the following steps:

[0040] Obtaining all the key scale image vertices and the key scale image bottom points;

[0041] Fitting is performed according to all the key scale image vertices and the key scale image bottom points to obtain a key area image;

[0042] An important pointer image is acquired according to the dial pointer image and the key area image.

[0043] Preferably, the step of calculating and obtaining the horizontal coordinates of the center of gravity of the important pointer image and the horizontal coordinates of the center of gravity of all the key scale images based on the key scale image, the important pointer image, and the arrester instrument dial image comprises the following steps:

[0044] Performing a contour extraction operation on the key scale images to obtain key scale vertex coordinates and key scale bottom point coordinates corresponding to each key scale image;

[0045] Acquire the original point and the original point coordinates corresponding to the original point according to all the key scale vertex coordinates and the key scale bottom point coordinates;

[0046] The original points include the coordinates of the vertices of the leftmost key scale image, the coordinates of the bottom point of the leftmost key scale image, the coordinates of the vertices of the rightmost key scale image, and the coordinates of the bottom point of the rightmost key scale image;

[0047] Fitting the upper arc function formula according to the coordinates of all the key scale vertices; and fitting the lower arc function formula according to the coordinates of all the key scale bottom points;

[0048] Obtain the highest point coordinates and the lowest point coordinates in all the key scale images; obtain the target point and the target point coordinates corresponding to each target point through the upper arc function and the lower arc function according to the highest point coordinates, the lowest point coordinates and all the original point coordinates;

[0049] Matching the original point with the target point respectively to obtain a perspective transformation matrix M; performing a perspective transformation operation on the arrester instrument dial image according to the perspective transformation matrix M to obtain a perspective transformed dial image; and obtaining a transformed important pointer image and coordinate information of the transformed important pointer image, a transformed key scale image and coordinate information of the transformed key scale image in the perspective transformed dial image;

[0050] The barycenter abscissa of the transformed important pointer and the barycenter abscissa of all the transformed key scales are calculated and obtained according to the transformed important pointer image and its coordinate information, the transformed key scale image and its coordinate information.

[0051] Preferably, the step of calculating and obtaining the pointer reading value based on the abscissa of the center of gravity of the important pointer and the abscissa of the center of gravity of all the key scales includes the following steps:

[0052] Obtain the barycenter abscissa key_left of the left key scale corresponding to the barycenter abscissa pl of the important pointer, the range value key_left_value corresponding to the left key scale, and the barycenter abscissa key_right of the right key scale, the range value key_right_value corresponding to the right key scale;

[0053] Calculate the pointer reading value based on the center of gravity abscissa pl of the important pointer, the center of gravity abscissa key_left of the left key scale, the range value key_left_value corresponding to the left key scale, the center of gravity abscissa key_right of the right key scale, and the range value key_right_value corresponding to the right key scale;

[0054] The pointer reading value is calculated as follows:

[0055]

[0056] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0057] An analysis of the above-mentioned intelligent patrol arrester instrument reading recognition method based on deep learning provided by the present invention shows that in specific applications, the arrester instrument image is collected by the patrol robot, and then the arrester instrument image is detected to obtain the dial area image, and the dial area image is further separately identified and processed, that is, the arrester instrument action number image and the range image in the arrester instrument dial image are identified to obtain the action number value and the range value, the arrester instrument dial image is identified to obtain the pointer reading, and finally the action number value and the pointer reading are returned to the arrester instrument image to obtain the reading of the arrester instrument image, realizing real-time intelligent reading operation of the arrester instrument based on intelligent detection of the patrol robot, which can replace manual operation, protect the safety of workers, reduce production costs, and avoid losses caused by human operational errors; and in the process of identifying the arrester instrument dial image to obtain the pointer reading, the deep learning semantic segmentation algorithm and perspective transformation and other operations are used to solve the problem of poor reading accuracy when the arrester instrument is tilted and the dial is blurred, and support the reading of the arrester action number, improve the arrester monitoring efficiency, and enhance the robustness of the reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 A schematic diagram of the overall operation steps of a method for identifying meter readings of intelligent patrol arresters based on deep learning provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a simulated image of a lightning arrester instrument in a method for intelligent inspection and lightning arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the operating steps for obtaining a dial area image in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of a simulation of obtaining detection results by using a trained dial area image detection model in a deep learning-based intelligent inspection arrester meter reading recognition method provided by an embodiment of the present invention;

[0063] Figure 5A schematic diagram simulating the detection results of a lightning arrester instrument in a method for intelligent inspection and recognition of lightning arrester instrument readings based on deep learning provided by an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the overall image simulation of a lightning arrester instrument in a method for intelligent inspection and lightning arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of a lightning arrester instrument dial image simulation in a method for intelligent inspection and lightning arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0066] Figure 8 A schematic diagram of an image simulation of the number of arrester meter actions in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0067] Figure 9 A schematic diagram of the operating steps for obtaining the number of actions and the range value in a method for identifying meter readings of an intelligent patrol arrester based on deep learning provided by an embodiment of the present invention;

[0068] Figure 10 A schematic diagram of a range image simulation in a method for intelligent inspection of lightning arrester meter readings based on deep learning provided by an embodiment of the present invention;

[0069] Figure 11 A schematic diagram of the operating steps for obtaining the range value and the number of actions in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0070] Figure 12 A schematic diagram of RNN network operation simulation in a deep learning-based intelligent inspection lightning arrester meter reading recognition method provided by an embodiment of the present invention;

[0071] Figure 13 A schematic diagram of the operational steps for obtaining a pointer reading value corresponding to an arrester instrument dial image in a method for intelligent inspection arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0072] Figure 14 A schematic diagram of a segmentation simulation of a semantic segmentation model in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0073] Figure 15 A schematic diagram of the steps for obtaining a dial pointer image and a dial scale image in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0074] Figure 16 A schematic diagram simulating the segmentation results of the lightning arrester instrument dial in a method for intelligent inspection lightning arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0075] Figure 17 A schematic diagram of the operating steps for adjusting and obtaining a horizontal lightning arrester instrument dial image in a method for intelligent inspection lightning arrester instrument reading recognition based on deep learning provided by an embodiment of the present invention;

[0076] Figure 18 A schematic diagram of the operating steps for obtaining key area images and important pointer images in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0077] Figure 19 A schematic diagram of the operating steps for obtaining the horizontal coordinates of the center of gravity of an important pointer image and the horizontal coordinates of the center of gravity of a key scale image in a method for intelligent inspection arrester meter reading recognition based on deep learning provided by an embodiment of the present invention;

[0078] Figure 20 A schematic diagram of a simulated dial image after perspective transformation in a method for recognizing meter readings of an intelligent patrol arrester based on deep learning provided by an embodiment of the present invention;

[0079] Figure 21 A schematic diagram of the operational steps for calculating and obtaining a pointer reading value in a method for identifying meter readings of intelligent patrol lightning arresters based on deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0081] The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings.

[0082] like Figure 1 As shown, the present invention proposes a method for identifying meter readings of intelligent patrol arresters based on deep learning, which includes the following steps:

[0083] Step S10: using the inspection robot to collect and obtain the arrester instrument image and point parameters corresponding to the target arrester instrument;

[0084] The target arrester instrument includes a pointer-type instrument panel; correspondingly, the arrester instrument image includes a pointer-type instrument panel image;

[0085] It should be noted that the technical solution adopted in the above embodiment of the present application is to scan the surrounding environment through the inspection robot, and after scanning the target lightning arrester instrument, use the positioning and navigation algorithm (laser SLAM positioning and navigation algorithm) to reach the vicinity of the target lightning arrester instrument (i.e., the target point), and then obtain the point parameters (of the target point), adjust the camera posture of the inspection robot according to the point parameters, and then trigger the shooting command to collect and obtain the lightning arrester instrument image (as shown below Figure 2 shown);

[0086] Before implementing the above embodiment of the present application, it is necessary to first configure the matching relationship between the lightning arrester instrument position information and the lightning arrester instrument range information;

[0087] The above-mentioned point parameters include the position parameter information of the target point of the lightning arrester instrument. Through the position parameter information, the current range information of the lightning arrester instrument can be known.

[0088] Step S20: detecting the arrester instrument image to obtain a dial area image;

[0089] The dial area image includes an overall image of the arrester instrument, an image of the arrester instrument dial, and an image of the number of times the arrester instrument operates;

[0090] Step S30: Identify the dial area image to obtain the action count value and the range value (i.e., identify the arrester instrument action count image to obtain the action count value; identify the arrester instrument dial image to obtain the range value); and identify and analyze the arrester instrument dial image based on the range value to obtain the pointer reading value corresponding to the arrester instrument dial image;

[0091] Step S40: returning the action times value and the pointer reading value to the arrester instrument image to obtain the reading of the arrester instrument image.

[0092] It should be noted that the above-mentioned embodiment of the present application collects the lightning arrester instrument image through the inspection robot, and then detects the lightning arrester instrument image to obtain the dial area image, and further identifies and processes the dial area image separately, that is, the lightning arrester instrument action count image and the range image in the lightning arrester instrument dial image are identified to obtain the action count value and the range value, the lightning arrester instrument dial image is identified to obtain the pointer reading, and finally the action count value and the pointer reading are returned to the lightning arrester instrument image to obtain the reading of the lightning arrester instrument image, realizing real-time lightning arrester instrument intelligent reading operation based on the inspection robot intelligent detection, which can replace manual operation, protect the safety of workers, reduce production costs, and avoid losses caused by human operational errors; and in the process of identifying the lightning arrester instrument dial image to obtain the pointer reading, the problem of poor reading accuracy when the lightning arrester instrument is tilted and the dial is blurred is solved through deep learning semantic segmentation algorithm and perspective transformation and other operations, and supports the reading of the lightning arrester action count, improves the lightning arrester monitoring efficiency, and enhances the robustness of the reading.

[0093] Specifically, if Figure 3 As shown, in step S20, the arrester instrument image is detected to obtain a dial area image, including the following steps:

[0094] Step S21: pre-processing the arrester instrument image to obtain a pre-processed arrester instrument image;

[0095] It should be noted that the above preprocessing, including image resizing, normalization, and denoising, helps reduce noise and inconsistency in image data, improves the robustness and generalization ability of the model, and thus better captures important features in the image and achieves better detection results.

[0096] Step S22: performing detection based on the pre-processed arrester instrument image using a trained dial area image detection model to obtain a detection result;

[0097] The test results include the arrester instrument test results, the arrester instrument dial test results, and the arrester instrument action times test results;

[0098] It should be noted that the above-mentioned dial area image detection model is constructed based on the open source YOLOX_S model of the deep learning target detection algorithm (the YOLOX_S model has fewer parameters and faster inference speed); the above-mentioned dial area image detection model needs to be trained to obtain a trained dial area image detection model; the above-mentioned training method is an existing technology, including: (1) obtaining a large amount of original lightning arrester instrument image (represented by image) data through an inspection robot; (2) manually annotating the lightning arrester instrument rectangular frame on the original image data; (3) dividing the labeled data into a training set and a test set; (4) constructing a dial area image detection model (i.e., a YOLOX detection model), and training it through the training set and the test set to obtain a trained dial area image detection model (i.e., a trained lightning arrester instrument detection YOLOX model);

[0099] as follows Figure 4 As shown, the input image is 640*640*3 in size, and its output is modified to 8400*8 (originally 8400*85). This is because the dial area image detection model trained in the above steps in this embodiment of the application only has 3 detection categories, and the original structure supports 80 categories.

[0100] as follows Figure 5 As shown, the above-mentioned lightning arrester instrument detection result (i.e., the circular target in the lightning arrester instrument image) is represented by (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the upper left corner point of the lightning arrester instrument image as a whole, and (x2, y2) represents the coordinates of the lower right corner point of the lightning arrester instrument image as a whole; the above-mentioned lightning arrester instrument dial detection result (i.e., the sector-shaped target in the lightning arrester instrument image) is represented by (x'1, y'1, x'2, y'2), where (x'1, y' 1) represents the coordinates of the upper left corner of the lightning arrester instrument dial image, and (x'2, y'2) represents the coordinates of the lower right corner of the lightning arrester instrument dial image; the above-mentioned lightning arrester instrument operation times detection result (i.e., the digital target in the lightning arrester instrument image) is represented by (x'1, y'1, x'2, y'2), where (x'1, y'1) represents the coordinates of the upper left corner of the lightning arrester instrument operation times image, and (x'2, y'2) represents the coordinates of the lower right corner of the lightning arrester instrument operation times image;

[0101] Step S23: obtaining a dial area image from the pre-processed arrester instrument image according to the detection result;

[0102] It should be noted that, in the above embodiment of the present application, according to the above detection results, the whole image of the arrester instrument in the pre-processed arrester instrument image is cut (as shown below). Figure 6 As shown), the lightning arrester instrument dial image (as shown below Figure 7 As shown) and the arrester instrument action times image (as shown below Figure 8 shown);

[0103] The above-mentioned embodiment of the present application uses the open source YOLOX_S model of YOLOX of the deep learning target detection algorithm to detect the dial area image, which can extract the dial area image in the lightning arrester instrument image more quickly and efficiently, thereby improving the detection efficiency.

[0104] Specifically, if Figure 9 As shown, in step S30, the dial area image is identified to obtain the action number value and the range value, including the following steps:

[0105] Step S301: acquiring a range image by performing detection based on the arrester instrument dial image through a range detection model;

[0106] The range image is a digital image above the dial scale image (such as Figure 10 shown);

[0107] It should be noted that the above-mentioned range detection model is constructed based on the YOLOX algorithm. The construction and training methods of the range detection model are existing technologies and will not be described in detail in this application.

[0108] The range detection model differs from the dial area image detection model trained in step S22 in that the range detection model in the embodiment of the present application detects only one category (i.e., the range image), while the trained dial area image detection model detects three categories (i.e., the overall image of the arrester instrument, the image of the arrester instrument dial, and the image of the number of times the arrester instrument operates). The range detection model and the trained dial area image detection model use the same target detection algorithm, i.e., they are two models with the same structural system trained using different data but with different specific functions.

[0109] The above-mentioned process of acquiring a range image by performing detection based on the arrester instrument dial image through a range detection model includes the following steps:

[0110] The arrester instrument dial image is detected by a range detection model to obtain range coordinates;

[0111] A range image is obtained by intercepting the arrester instrument dial image based on the range coordinates.

[0112] Step S302: Based on the range image and the arrester instrument action times image, a pre-trained character recognition model is used to perform detection to obtain a range value and an action times value.

[0113] It should be noted that the above-mentioned trained character recognition model is constructed and trained based on the CRNN optical character recognition algorithm. The CRNN (Convolutional Recurrent Neural Network) optical character recognition algorithm is a deep learning algorithm that combines convolutional neural networks (CNN) and recurrent neural networks (RNN). It can automatically recognize image characters more efficiently and accurately, thereby improving detection efficiency.

[0114] Specifically, Figure 11 In step S302, based on the range image and the arrester instrument action times image, a pre-trained character recognition model is used to detect and obtain the range value and the action times value, which includes the following steps:

[0115] Step S3021: traverse the range image and the arrester instrument action times image, randomly select any image as a target image; perform a feature extraction operation on the target image to obtain a feature sequence;

[0116] It should be noted that the above-mentioned embodiment of the present application first performs feature extraction by inputting the target image into a trained character recognition model, thereby reducing the data dimension and removing redundant information, thereby improving the effect of detecting numerical values.

[0117] Step S3022: obtaining image association relationship information in the target image through the feature sequence;

[0118] It should be noted that the above embodiment of the present application uses an RNN network to learn the association information of each feature information in the feature sequence, that is, the association relationship information in the target image; the above RNN network is a neural network structure with a recurrent connection, which learns the image association relationship information (that is, the context information of the image) in the feature sequence through hidden states;

[0119] Step S3023: converting the feature sequence based on the image association relationship information and outputting a numerical value (the numerical value is the above-mentioned range value or action number value);

[0120] It should be noted that if Figure 12 As shown, the above-mentioned embodiment of the present application sends the image association relationship information output by the RNN network to the transcription layer, and then converts the feature sequence into a series of output symbols through the transcription layer based on the above-mentioned image association relationship information. These symbols can be characters or words, etc. The symbols output here are the digital value characters corresponding to the range image and the lightning arrester instrument action number image (i.e., the range value and the action number value), which can more accurately and quickly identify the values in the image and improve the detection efficiency.

[0121] Specifically, if Figure 13As shown, in step S30, the image of the lightning arrester instrument dial is identified and analyzed based on the range value to obtain the pointer reading value corresponding to the image of the lightning arrester instrument dial, including the following steps:

[0122] Step S31: Segmenting the arrester instrument dial image using a pre-trained semantic segmentation model to obtain a dial pointer image and a dial scale image; and adjusting the arrester instrument dial image according to the dial scale image to obtain a horizontal arrester instrument dial image (to avoid reading errors caused by the tilt of the arrester instrument);

[0123] The dial scale image includes a common scale image and a key scale image;

[0124] Step S32: obtaining the scale range value corresponding to the key scale image by matching the coordinates corresponding to the range value with the coordinates of the dial scale image (the scale range value is obtained by matching the range value with the key scale image by position);

[0125] It should be noted that according to the range image, range value and the position of the key scale image, the range corresponding to each key scale can be matched according to the distance relationship, and the range value is known, so the range value of each key scale (that is, the scale range value) is known.

[0126] Step S33: acquiring a key area image and an important pointer image based on the dial pointer image and the dial scale image; calculating and acquiring the horizontal coordinates of the center of gravity of the important pointer image and the horizontal coordinates of the center of gravity of all the key scale images based on the key scale image, the important pointer image, and the arrester instrument dial image;

[0127] Step S34: Calculate and obtain the pointer reading value according to the horizontal coordinate of the center of gravity of the important pointer and the horizontal coordinates of the center of gravity of all the key scales.

[0128] It should be noted that the semantic segmentation model used in the above embodiment of the present application can quickly and efficiently identify the dial pointer image and the dial scale image (such as Figure 14 As shown), and adjusting the arrester instrument image can avoid reading errors caused by the tilt of the arrester instrument; at the same time, the scale range value and the horizontal coordinates of the center of gravity of the important pointer image and the horizontal coordinates of the center of gravity of the key scale image are used to prepare for the following identification of the pointer reading.

[0129] Specifically, if Figure 15 As shown, in step S31, the arrester instrument dial image is segmented using a pre-trained semantic segmentation model to obtain a dial pointer image and a dial scale image, including the following steps:

[0130] Step S311: pre-processing the arrester instrument dial image to obtain a pre-processed arrester instrument dial image;

[0131] It should be noted that the above-mentioned preprocessing includes image scaling and image filling processing operations. The preprocessing operation is first performed on the lightning arrester instrument dial image because the size of the lightning arrester instrument dial image obtained in the above-mentioned step S23 is irregular, and the irregular image size will cause a large error after being input into the semantic segmentation model. Therefore, the lightning arrester instrument dial image needs to be preprocessed to obtain a preprocessed lightning arrester instrument dial image, so that the dial pointer image and dial scale image outputted after the preprocessed lightning arrester instrument dial image is input into the trained semantic segmentation model for detection are more accurate;

[0132] The above preprocessing operation includes first obtaining the width and height of the lightning arrester instrument dial image and taking the maximum value (that is, the maximum width of the lightning arrester instrument dial image and the maximum height of the lightning arrester instrument dial image); then scaling the image proportionally according to the maximum value to an image with a maximum width and height of 416; finally, filling the edges less than 416 with a pixel value of 114 to make the image 416*416 in size, thereby obtaining the preprocessed lightning arrester instrument dial image.

[0133] Step S312: The pre-processed arrester instrument dial image is detected by a trained semantic segmentation model to obtain a dial pointer image and a dial scale image.

[0134] It should be noted that if Figure 16 As shown, through the technical solution adopted in the embodiment of the present application, the dial pointer image, the key scale image, and the ordinary scale image are detected and attached to the pre-processed arrester instrument dial image input by the trained semantic segmentation model;

[0135] Normally, due to reasons such as light and wear, some ordinary scale images of the lightning arrester instrument dial image will be blurred, but the key scale image is still easy to identify. Therefore, in order to utilize the high recognition characteristics of the key scale image, the embodiment of the present application distinguishes the key scale image from the ordinary scale image during segmentation. Because in real scenarios, ordinary scales are very easy to be difficult to identify due to reasons such as light, shooting angle, wear, etc., the present invention greatly reduces the dependence on ordinary scales, and has better robustness than the existing method of relying on ordinary scales to read instrument dials.

[0136] Specifically, if Figure 17 As shown, in step S31, the arrester instrument dial image is adjusted according to the dial scale image to obtain a horizontal arrester instrument dial image, including the following steps:

[0137] Step S3101: obtaining the midpoint of the common scale image; and obtaining a scale ellipse image by calculation based on the midpoint of the common scale image;

[0138] It should be noted that the above embodiment of the present application is to calculate the scale ellipse image by using the least square method through the midpoint of the ordinary scale;

[0139] Step S3102: Calculating and obtaining a rotation angle through the scale ellipse image;

[0140] Step S3103: performing an affine transformation on the arrester instrument dial image based on the rotation angle to obtain a horizontal arrester instrument dial image.

[0141] It should be noted that in the above-mentioned embodiment of the present application, the midpoint of the segmented ordinary scale image is used to directly calculate the scale ellipse image (based on the least squares method), and the rotation angle of the calculated scale ellipse image is used to perform an affine transformation on the lightning arrester instrument dial image, so that the lightning arrester instrument dial image remains as horizontal as possible, thereby reducing the interference caused by the tilt angle of the lightning arrester instrument;

[0142] The above-mentioned affine transformation refers to the affine transformation in which the arrester instrument dial image is reversely rotated by angle a according to the calculated rotation angle a of the scale ellipse image. This ensures that the instrument dial (sector) in the arrester instrument dial image after reverse rotation is horizontal, so that the subsequent steps can produce more accurate results.

[0143] Specifically, if Figure 18 As shown, in step S33, the key area image and the important pointer image are acquired based on the dial pointer image and the dial scale image, including the following steps:

[0144] Step S331: Acquire all the key scale image vertices and the key scale image bottom points;

[0145] Step S332: performing fitting based on all the key scale image vertices and the key scale image bottom points to obtain a key area image;

[0146] It should be noted that the technical solution adopted in the above-mentioned embodiment of the present application, as shown in Figure xx, is to use the vertices and bottom points of all key scale images to fit to obtain two arcs (denoted as the upper arc and the lower arc), and the area image between these two arcs is the key area image.

[0147] Step S333: Acquire an important pointer image according to the dial pointer image and the key area image.

[0148] It should be noted that the technical solution adopted in the above-mentioned embodiment of the present application is to use the part of the dial pointer image within the key area image as the important pointer image.

[0149] Specifically, if Figure 19 As shown, in step S33, the horizontal coordinates of the center of gravity of the important pointer image and the horizontal coordinates of the center of gravity of all the key scale images are calculated based on the key scale image, the important pointer image and the arrester instrument dial image, including the following steps:

[0150] Step S3301: performing a contour extraction operation on the key scale image to obtain the key scale vertex coordinates and key scale bottom point coordinates corresponding to each key scale image;

[0151] Step S3302: obtaining the original point and the original point coordinates corresponding to the original point according to the coordinates of all the key scale vertices and the coordinates of the key scale bottom points;

[0152] The original points include the coordinates of the vertices of the leftmost key scale image, the coordinates of the bottom point of the leftmost key scale image, the coordinates of the vertices of the rightmost key scale image, and the coordinates of the bottom point of the rightmost key scale image;

[0153] Step S3303: fitting the upper arc function according to the coordinates of all the key scale vertices; and fitting the lower arc function according to the coordinates of all the key scale bottom points;

[0154] Step S3304: obtaining the highest point coordinates and the lowest point coordinates in all the key scale images; obtaining the target point and the target point coordinates corresponding to each target point through the upper arc function and the lower arc function according to the highest point coordinates, the lowest point coordinates and all the original point coordinates;

[0155] It should be noted that the above-mentioned target points include target point 1, target point 2, target point 3, and target point 4;

[0156] In the above embodiment of the present application, the contour coordinate information of each key scale image is solved by extracting the contour, thereby obtaining the vertex and bottom coordinates of each key scale image, and then sorting them to obtain the original points and their coordinates (i.e., the vertex coordinates of the leftmost key scale image, the bottom coordinates of the leftmost key scale image, the vertex coordinates of the rightmost key scale image, and the bottom coordinates of the rightmost key scale image).

[0157] The upper and lower arcs are further fitted using the vertex and bottom coordinates of each keyscale image (the function formula of the upper arc and the function formula of the lower arc are obtained); then the coordinates of the highest point and the lowest point in all keyscale images are obtained, the horizontal coordinate of the highest point in the keyscale is substituted into the upper arc function formula to obtain the vertical coordinate of the highest point of the upper arc, and the horizontal coordinate of the lowest point in the keyscale is substituted into the lower arc function formula to obtain the vertical coordinate of the lowest point of the lower arc. Finally, the coordinates of the target points are obtained as follows: the coordinates of target point 1 are (the horizontal coordinates of the vertex coordinates of the leftmost keyscale image, the vertical coordinates of the highest point of the upper arc); the coordinates of target point 2 are (the horizontal coordinates of the vertex coordinates of the leftmost keyscale image, the vertical coordinates of the lowest point of the lower arc); the coordinates of target point 3 are (the horizontal coordinates of the vertex coordinates of the rightmost keyscale image, the vertical coordinates of the highest point of the upper arc); and the coordinates of target point 4 are (the horizontal coordinates of the vertex coordinates of the rightmost keyscale image, the vertical coordinates of the lowest point of the lower arc).

[0158] Step S3305: Match the original point with the target point (pairwise matching) to obtain a perspective transformation matrix M; perform a perspective transformation operation on the arrester instrument dial image according to the perspective transformation matrix M to obtain a perspective transformed dial image; and obtain a transformed important pointer image and its coordinate information, a transformed key scale image and its coordinate information in the perspective transformed dial image;

[0159] It should be noted that the above embodiment of the present application is to perform perspective transformation on all pixels in the arrester instrument dial image according to the perspective transformation matrix M to obtain the dial image after perspective transformation (such as Figure 20 shown).

[0160] Step S3306: Calculating and obtaining the barycenter abscissa of the transformed important pointer (i.e., the barycenter abscissa of the important pointer image) and the barycenter abscissa of all the transformed key scales (i.e., the barycenter abscissa of the key scale image) based on the transformed important pointer image and its coordinate information, the transformed key scale image and its coordinate information;

[0161] It should be noted that the above-mentioned embodiment of the present application calculates the horizontal coordinates of the center of mass (center of mass or center of gravity) of the transformed important pointer image and the horizontal coordinates of the center of gravity of the transformed key scale image based on the coordinate information of the transformed important pointer image and the coordinate information of the transformed key scale image in the dial image after the above-mentioned perspective transformation (the transformed important pointer image and the transformed key scale image are both vertical images) through the contour zero-order moment and first-order moment, wherein the horizontal coordinates of the center of gravity of the transformed key scale image are an array containing multiple horizontal coordinates (ascending horizontal coordinates). For example: when the width of the instrument dial image is 416, the array of the horizontal coordinates of the center of gravity of the transformed key scale image finally obtained is [10, 70, 130, 200, 270, 340, 410], and the 7 values represent the horizontal coordinates of each key scale from left to right, and the corresponding range values are [0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0].

[0162] The existing method determines the pointer position by the parallel relationship between the pointer contour and all scale contours. However, considering that the root of the lightning arrester instrument pointer is not necessarily at the horizontal center of the instrument (that is, the pointer root is not necessarily symmetrical on the left and right, as shown in the lightning arrester instrument image in this embodiment), this embodiment processes the lightning arrester instrument dial image through perspective transformation, and then determines the pointer position based on the horizontal coordinate position relationship between the centroid of all key scales and the centroid of the pointer in the processed instrument dial image. This method supports more lightning arrester instruments and has better robustness.

[0163] Specifically, if Figure 21 As shown, in step S34, the pointer reading value is calculated based on the horizontal coordinate of the center of gravity of the important pointer and the horizontal coordinates of the center of gravity of all the key scales, including the following steps:

[0164] Step S341: Obtain the barycenter abscissa key_left of the left key scale corresponding to the barycenter abscissa pl of the important pointer, the range value key_left_value corresponding to the left key scale, and the barycenter abscissa key_right of the right key scale, the range value key_right_value corresponding to the right key scale;

[0165] It should be noted that in the above step S32, the scale range value corresponding to the key scale image is obtained by matching. In the steps of this embodiment, the scale value key_left_value corresponding to the left key scale and the scale value key_right_value corresponding to the right key scale can be obtained according to the horizontal coordinate pl of the center of gravity of the important pointer corresponding to the left key scale and the right key scale.

[0166] Step S342: Calculate and obtain the pointer reading value based on the horizontal coordinate of the center of gravity of the important pointer, the horizontal coordinate of the center of gravity of the left key scale, the horizontal coordinate of the center of gravity of the left key scale, and the range value of the left key scale (obtained in step S32). The horizontal coordinate of the center of gravity of the right key scale, key_right, and the range value of the right key scale (obtained in step S32) are calculated.

[0167] The pointer reading value is calculated as follows:

[0168]

[0169] For example: In the above example of transforming the horizontal coordinate of the center of gravity of the key scale image into an array containing multiple horizontal coordinates, if the horizontal coordinate of the center of gravity of the important pointer image is 150, then the horizontal coordinate of the center of gravity of the left key scale key_left and the horizontal coordinate of the center of gravity of the right key scale key_right are 130 and 200 respectively; then, after calculating based on the key_left_value and key_right_value being 1.0 and 1.5 respectively, the value is 1.14.

[0170] In summary, the present invention proposes a method for intelligent inspection of lightning arrester meter reading recognition based on deep learning. The method uses an inspection robot to collect lightning arrester meter images, and then uses the open source YOLOX_S model of YOLOX based on the deep learning target detection algorithm to detect the lightning arrester meter image to obtain the dial area image, thereby improving detection efficiency.

[0171] The dial area image is further identified and processed separately, that is, the range image in the arrester instrument action count image and the arrester instrument dial image is identified by the Crnn optical character recognition algorithm and the RNN network to identify the range value and the action count value, which can automatically recognize the image characters more efficiently and accurately, thereby improving the detection efficiency;

[0172] Then, the semantic segmentation model is used to segment the lightning arrester instrument dial image to obtain the dial pointer image and the dial scale image, and the lightning arrester instrument dial image is adjusted to a horizontal state through affine transformation to avoid reading errors caused by the tilt of the lightning arrester instrument. At the same time, the scale range value is obtained by matching the range value and the key scale image. The high recognition feature of the key scale image is utilized to avoid reading errors caused by light, shooting angle, wear and tear, thereby improving robustness; the lightning arrester instrument dial image is further processed by perspective transformation, and the pointer position is determined based on the horizontal coordinate position relationship between the centroid of all key scales and the centroid of the pointer in the processed instrument dial image, so that the pointer reading is calculated based on the pointer position and the range value of the key scale images on the left and right sides of the pointer;

[0173] Finally, the action times value and pointer reading are returned to the arrester instrument image to obtain the reading of the arrester instrument image, realizing real-time intelligent reading operation of the arrester instrument based on intelligent detection of the inspection robot, which can replace manual operation, protect the safety of workers, reduce production costs, and avoid losses caused by human operational errors; and in the process of identifying the arrester instrument dial image to obtain the pointer reading, the deep learning semantic segmentation algorithm and perspective transformation and other operations solve the problem of poor reading accuracy when the arrester instrument is tilted and the dial is blurred, and supports the reading of the arrester action times, improves the arrester monitoring efficiency, and enhances the robustness of the reading.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. A person skilled in the art may modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying meter readings of intelligent patrol arresters based on deep learning, characterized in that: The steps are as follows: Collect and obtain images of lightning arrester instruments; Detecting the arrester instrument image to obtain a dial area image; The dial area image includes an overall image of the arrester instrument, an image of the arrester instrument dial, and an image of the number of times the arrester instrument operates; Identify the dial area image to obtain the action number value and the range value; and identify and analyze the lightning arrester instrument dial image based on the range value to obtain the pointer reading value corresponding to the lightning arrester instrument dial image; Returning the action times value and the pointer reading value to the arrester instrument image to obtain the reading of the arrester instrument image; The detecting the arrester instrument image to obtain the dial area image includes the following steps: Preprocessing the arrester instrument image to obtain a preprocessed arrester instrument image; Based on the pre-processed arrester instrument image, a detection is performed using a trained dial area image detection model to obtain a detection result; The test results include the arrester instrument test results, the arrester instrument dial test results, and the arrester instrument action times test results; intercepting a dial area image from the pre-processed arrester instrument image according to the detection result; Recognizing the dial area image to obtain the action count value and the range value includes the following steps: Based on the arrester instrument dial image, a range image is obtained by performing detection using a range detection model; The range image is a digital image above the dial scale image; Based on the range image and the arrester instrument action times image, a pre-trained character recognition model is used to detect and obtain the range value and the action times value; The step of identifying and analyzing the arrester instrument dial image based on the range value to obtain a pointer reading value corresponding to the arrester instrument dial image includes the following steps: Segmenting the arrester instrument dial image to obtain a dial pointer image and a dial scale image; and adjusting the arrester instrument dial image according to the dial scale image to obtain a horizontal arrester instrument dial image; The dial scale image includes a common scale image and a key scale image; Acquire the scale range value corresponding to the key scale image by matching the coordinates corresponding to the scale value with the coordinates of the dial scale image; Acquire a key area image and an important pointer image based on the dial pointer image and the dial scale image; calculate and acquire the centroid abscissa of the important pointer image and the centroid abscissa of all the key scale images based on the key scale image, the important pointer image and the arrester instrument dial image; The pointer reading value is calculated based on the horizontal coordinate of the center of gravity of the important pointer and the horizontal coordinates of the center of gravity of all the key scales.

2. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 1 is characterized in that: The method of acquiring the range value and the action count value by detecting based on the range image and the arrester instrument action count image through a pre-trained character recognition model includes the following steps: Traversing the range image and the arrester instrument action times image, randomly selecting any image as a target image; performing a feature extraction operation on the target image to obtain a feature sequence; Acquiring image association relationship information in the target image through the feature sequence; The feature sequence is converted into a numerical value based on the image association relationship information.

3. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 2 is characterized in that: The step of segmenting the arrester instrument dial image to obtain a dial pointer image and a dial scale image includes the following steps: Preprocessing the arrester instrument dial image to obtain a preprocessed arrester instrument dial image; The pre-processed arrester instrument dial image is detected by a trained semantic segmentation model to obtain a dial pointer image and a dial scale image.

4. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 3 is characterized in that: The step of adjusting the arrester instrument dial image according to the dial scale image to obtain a horizontal arrester instrument dial image includes the following steps: Obtaining the midpoint of the common scale image; and obtaining a scale ellipse image by calculation based on the midpoint of the common scale image; Obtaining a rotation angle by calculating the scale ellipse image; An affine transformation is performed on the arrester instrument dial image based on the rotation angle to obtain a horizontal arrester instrument dial image.

5. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 4 is characterized in that: The step of acquiring the key area image and the important pointer image based on the dial pointer image and the dial scale image includes the following steps: Obtaining all the key scale image vertices and the key scale image bottom points; Fitting is performed according to all the key scale image vertices and the key scale image bottom points to obtain a key area image; An important pointer image is acquired according to the dial pointer image and the key area image.

6. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 5 is characterized in that: The step of calculating and obtaining the horizontal coordinates of the center of gravity of the important pointer image and the horizontal coordinates of the center of gravity of all the key scale images based on the key scale image, the important pointer image, and the arrester instrument dial image includes the following steps: Performing a contour extraction operation on the key scale images to obtain key scale vertex coordinates and key scale bottom point coordinates corresponding to each key scale image; Acquire the original point and the original point coordinates corresponding to the original point according to all the key scale vertex coordinates and the key scale bottom point coordinates; The original points include the coordinates of the vertices of the leftmost key scale image, the coordinates of the bottom point of the leftmost key scale image, the coordinates of the vertices of the rightmost key scale image, and the coordinates of the bottom point of the rightmost key scale image; Fitting the upper arc function formula according to the coordinates of all the key scale vertices; and fitting the lower arc function formula according to the coordinates of all the key scale bottom points; Obtain the highest point coordinates and the lowest point coordinates in all the key scale images; obtain the target point and the target point coordinates corresponding to each target point through the upper arc function and the lower arc function according to the highest point coordinates, the lowest point coordinates and all the original point coordinates; Match the original point with the target point respectively to obtain the perspective transformation matrix M; Performing a perspective transformation operation on the arrester instrument dial image according to the perspective transformation matrix M to obtain a perspective transformed dial image; and obtaining a transformed important pointer image and its coordinate information, a transformed key scale image and its coordinate information in the perspective transformed dial image; The barycenter abscissa of the transformed important pointer and the barycenter abscissa of all the transformed key scales are calculated and obtained according to the transformed important pointer image and its coordinate information, the transformed key scale image and its coordinate information.

7. The method for identifying meter readings of an intelligent patrol arrester based on deep learning according to claim 6 is characterized in that: The step of calculating and obtaining the pointer reading value based on the abscissa of the center of gravity of the important pointer and the abscissa of the center of gravity of all the key scales includes the following steps: Obtain the barycenter abscissa key_left of the left key scale corresponding to the barycenter abscissa pl of the important pointer, the range value key_left_value corresponding to the left key scale, and the barycenter abscissa key_right of the right key scale, the range value key_right_value corresponding to the right key scale; Calculate the pointer reading value based on the center of gravity abscissa pl of the important pointer, the center of gravity abscissa key_left of the left key scale, the range value key_left_value corresponding to the left key scale, the center of gravity abscissa key_right of the right key scale, and the range value key_right_value corresponding to the right key scale; The pointer reading value is calculated as follows: 。

Images