A method and device for identifying the knob state of a power switch cabinet based on deep learning

Abstract

The present invention discloses a method and device for identifying the state of a power switchgear knob based on deep learning. The method includes the following steps: obtaining an original image of a power switchgear knob, pre-processing it, and inputting it into a pre-designed and trained target detection network to determine the position of the knob in the image and obtain a knob detection frame; cropping the image to obtain a knob image containing only the knob; using a pre-designed knob pointer key point and word position detection network to perform knob pointer key point detection and word position detection on the knob image; calculating the angle between all words and the pointer direction based on the words and the pointer direction and sorting them; identifying the words corresponding to the angle, performing a validity check on the identified words, finding the first valid word, determining it as the current state of the knob, and outputting it. The present invention has high accuracy and good versatility and is applicable to various types of knob state recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence application technology, and in particular to a method and device for identifying the knob status of a power switch cabinet based on deep learning. Background Art

[0002] Power switchgear features a variety of knobs that control power equipment settings. Accurate, real-time monitoring of knob status is crucial to the safe and reliable operation of the power system. The wide variety of knobs in power switchgear, each capable of different settings, poses a challenge to developing a unified, universal monitoring technology for knob status.

[0003] Currently, the status of power switchgear knobs is often identified by patrol robots taking regular photos, followed by manual observation in the background. This method has high recognition accuracy but requires high labor costs. Another approach is to have the patrol robot take a pre-image of each knob to be identified and record its status. During the robot's inspection, it can compare the photos with the pre-image to determine the knob's status. The advantage is that no human intervention is required during the inspection. The disadvantage is that a set of adaptive algorithm logic must be developed for each knob type, the pre-image creation process is time-consuming, and the number of knobs that can be covered is limited. Summary of the Invention

[0004] Technical purpose: In response to the above technical problems, the present invention proposes a method and device for identifying the knob status of a power switch cabinet based on deep learning. It adopts deep learning technology, designs a multi-task network model system, and uses different models to process and analyze the collected photos. It has high accuracy and good versatility, and can be applied to various types of knob status recognition.

[0005] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] A method for identifying the knob state of a power switch cabinet based on deep learning, characterized by comprising the steps of:

[0007] (1) Obtain the original image of the power switch cabinet knob taken by the inspection robot;

[0008] (2) Preprocess the original image and then input it into a pre-designed and trained target detection network. The target detection network determines the location of the knob in the preprocessed image and obtains the knob detection frame.

[0009] (3) Expand the knob detection frame, enlarge the width and height of the knob detection frame by a preset multiple, and then crop the knob detection frame after the expansion process to obtain a knob image containing only the knob;

[0010] (4) Using a pre-designed knob pointer key point and word position detection network, perform knob pointer key point detection and word position detection on the knob image; wherein, knob key point detection is to detect two key points, the pointer support and the pointer head, in the knob image, and determine the pointer direction through the pointer support and the pointer head; word position detection is to detect all words in the knob image, and each word represents a state of the knob;

[0011] (5) Calculate the angles between all words and pointer directions based on the words and pointer directions obtained in step (4), and then arrange all the calculated angles in ascending order;

[0012] (6) According to the sorting result of step (5), the words corresponding to the angle are identified, the legitimacy of the identified words is judged, and the first valid word is found;

[0013] (7) The first valid word found in step (6) is determined as the current state of the knob and output.

[0014] Preferably, in step (2), the preprocessing of the original image includes: scaling the size of the original image to 640*640, the target detection network adopts the yolov5s knob detection model, and the obtained knob detection frame is a rectangular frame containing the knob image.

[0015] Preferably, in step (3), the width and height of the knob detection frame are expanded by 1.25 times, and the center point of the frame remains unchanged.

[0016] Preferably, in step (3), the knob pointer key point and word position detection network includes a backbone network and a first detection head and a second detection head connected to an output end of the backbone network;

[0017] The backbone network is used to extract features from the input image, i.e., the knob image, to obtain a knob feature map;

[0018] The first detection head is a key point branch that outputs two-channel probability maps. The probability map of the first channel represents the probability map of the pointer pillar, and the probability map of the second channel represents the probability map of the pointer head. The position with the largest array in each probability map represents the corresponding key point position.

[0019] The second detection head and the word detection branch output a word segmentation probability map, perform contour and minimum circumscribed quadrilateral operations on the word segmentation probability map, and the obtained quadrilateral box is the location of the word, that is, the word box.

[0020] Preferably, the backbone network processes the input image including:

[0021] The input image is passed through a convolutional layer with a stride of 4, and the input image is calculated into a feature map with 64 channels and a size of 96*72;

[0022] Entering the first stage, stage 1, four bottleneck operations are performed to extract features, and then downsampled to generate feature maps of different resolutions;

[0023] Enter the second stage stage2, the third stage stage3 and the fourth stage stage4 in sequence, where each stage performs the following operations: perform 4 bottleneck operations to extract features, then perform downsample operations to generate new feature maps of different resolutions. While performing the downsample operation, the feature maps with resolutions less than the preset threshold are enlarged through bilinear interpolation operations and fused with the corresponding feature maps before resolution enlargement. The size of the fused feature maps is 96*72;

[0024] The fused feature map is used as the knob feature map and input into the first detection head and the second detection head.

[0025] Preferably, the step (5) specifically includes:

[0026] Obtain the center point positions of all word frames obtained in step (4), and calculate the angle formed by the pointer support as the starting point, the word frame center point and the pointer head point;

[0027] All calculated angles are sorted from large to small, and the size of the angle represents the probability that the pointer points to the corresponding word box.

[0028] Preferably, the step (6) specifically includes:

[0029] Use the CRNN network model to complete word recognition and identify the words corresponding to the angle;

[0030] The legitimacy of the recognized words is judged, that is, the words are compared with the legal status identification text counted in advance. If they are legal status identification texts, the corresponding words are determined as the current state of the knob. Otherwise, continue to recognize the next word.

[0031] A device for identifying the knob state of a power switch cabinet based on deep learning, characterized by comprising:

[0032] The original image acquisition module is used to obtain the original image of the power switch cabinet knob taken by the inspection robot;

[0033] Preprocessing module, used to preprocess the original image;

[0034] The object detection network is used to identify the location of the knob in the preprocessed image and obtain the knob detection frame;

[0035] The image cropping module is used to expand the knob detection frame, enlarge the width and height of the knob detection frame by a preset multiple, and then crop the knob detection frame after the expansion process to obtain a knob image containing only the knob;

[0036] The knob pointer key point and word position detection network is used to detect knob pointer key points and word positions on the knob image. Knob key point detection detects two key points in the knob image: the pointer support and the pointer head. The pointer direction is determined by the pointer support and the pointer head. Word position detection detects all words in the knob image, each word representing a state of the knob.

[0037] Angle calculation and sorting module, which is used to calculate the angles between all words and pointer directions based on the knob pointer key points and the words and pointer directions obtained by the word position detection network, and then arrange all the calculated angles in order from small to large;

[0038] A word recognition module is used to recognize words corresponding to the angle according to the sorting results of the angle calculation and sorting module;

[0039] The word legitimacy judgment module is used to judge the legitimacy of the recognized words and find the first valid word;

[0040] The knob state determination module is used to determine the first valid word found by the word legitimacy judgment module as the current state of the knob and output it.

[0041] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following beneficial effects:

[0042] The present invention first performs knob target detection on the photo to accurately locate the knob. It then crops the knob area to create a smaller image containing only the knob. This image is then fed into a knob pointer keypoint and word position detection network to accurately identify the knob pointer keypoints and all word positions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the power switch cabinet knob state recognition method based on deep learning proposed by the present invention;

[0044] Figure 2 It is a schematic diagram of the structure of the knob key point and word position detection network in the present invention;

[0045] Figure 3Schematic diagram of the structure of the Bottleneck module in the present invention;

[0046] Figure 4 Schematic diagram of the structure of the downsample module in the present invention;

[0047] Figure 5 This is a picture taken by the inspection robot. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0049] The present invention aims to provide a method for identifying the status of a power switchgear knob based on deep learning technology. The method utilizes deep learning technology to analyze captured photos. First, a target detection network determines the knob's location. The detection frame is then expanded 1.25 times for cropping. The cropped image is then fed into a key point and word position detection network. This network detects the direction of the knob pointer and all words. The angle between the knob pointer and the words is then calculated and sorted from smallest to largest. Finally, the words are sequentially recognized, and the word with the smallest angle with the pointer is determined as the knob status.

[0050] like Figure 1 As shown, the implementation is carried out according to the following steps:

[0051] Step 1: Knob Target Detection Module. After the inspection robot captures an image at a fixed point, the Knob Target Detection Module determines the specific location of the knob. After obtaining the knob's target frame, it is expanded by 1.25 times in width and height. After expansion, the frame is cropped to produce a new image containing only the knob.

[0052] Step 2: Pointer direction and word position detection module. Pointer direction detection primarily detects key points on the knob to determine the pointer direction. Knob key points include the position of the knob support and the knob head. The position of the support and knob head can be used to determine the pointer's direction. Word position detection primarily detects the position of all words on the knob. Generally speaking, each word on the knob represents a state of the knob.

[0053] Step 3: Knob state recognition module. This module processes the words and pointer directions obtained in step 2. It first calculates the angles between all words and the pointer, then sorts these angles from smallest to largest. It then begins recognizing the sorted words. The first valid word found is labeled as the current state of the knob.

[0054] Among them, step 1 knob target detection module:

[0055] Step 1.1: Current inspection robots typically patrol various locations by taking fixed-point photos. When a knob target is photographed, a visible light image is obtained. To accurately locate the knob, the image is processed using the YoLoV5s network. Before performing inference on the image, a YoLoV5s knob detection model must be trained. The inference process is as follows: First, the image is pre-processed and resized to 640*640 pixels. Then, the YoLoV5s model is used for inference to determine the knob frame's location.

[0056] Step 1.2: Obtain the knob frame from step 1.1. If there are multiple knobs, multiple knob frames will be included. This frame is a rectangular frame that only contains the knobs. Expand the rectangular frame by increasing its width and height by 1.25 times, while keeping its center point unchanged. Crop the expanded rectangular frame to obtain an image containing only the rectangular frame.

[0057] Step 2: Pointer direction and word position detection module

[0058] Step 2.1: The pointer direction is determined by detecting two key points on the pointer, namely the pointer support and the pointer head. Word detection mainly detects the position of words on the knob. The two detections share the backbone network to extract features, and then use different detection heads to detect key points and word positions respectively. Figure 2 As shown, first build the backbone network. The backbone network is a high-resolution network. The network first passes the input image through a convolutional layer with a stride of 4, and calculates the image into a feature map with 64 channels and a size of 96*72. It then extracts features through 4 bottleneck operations, and then generates feature maps of different resolutions through downsample operations. After the same 4 bottleneck operations, a new resolution feature map is generated through a downsample operation. At the same time, the small feature map needs to be expanded through a bilinear interpolation operation to expand the feature map resolution and fused with the original resolution feature map. After 4 consecutive stages. The feature maps of each resolution are fused into a feature map of size 96*72. The backbone network is followed by two different detection heads, which output different probability maps to represent key points and word position information respectively.

[0059] Bottleneck is the bottleneck operation, such as Figure 3 This layer uses a residual structure, which can avoid network degradation while deepening the network. Bottleneck first uses 1x1 convolution to increase the dimension, then uses depth-separable convolution for convolution operation, and then uses 1x1 convolution for dimensionality reduction, and finally performs a shortcut operation. The activation function of Bottleneck is Relu6, and its expression is as follows:

[0060] y=Relu6(x)=min(max(x,0),6)

[0061] The depthwise separable convolution used in the Bottleneck operation can greatly reduce the parameters of the model, thereby saving computing power and improving the model prediction speed. Figure 4 This is a schematic diagram of the downsample operation.

[0062] Step 2.2: Preprocess the image obtained in Step 1.2 and resize it to 384 x 288 pixels. Then normalize the image data by subtracting the mean values ​​of 0.485, 0.456, and 0.406 from the R, G, and B channels, respectively, and then dividing them by the mean values ​​of 0.229, 0.224, and 0.225.

[0063] Step 2.3: Input the processed data from Step 2.2 into the network. After inference, two output branches are generated: keypoints and words. The keypoint branch outputs a two-channel probability map: the first channel represents the pointer support position probability map, and the second channel represents the pointer head probability map. The position with the largest array in each probability map represents the keypoint location. The word detection branch outputs the word segmentation probability map. First, all values ​​within the probability map with a value less than 0.3 are filtered out. Then, the contour and minimum circumscribed quadrilateral operations are performed on the probability map. The resulting quadrilateral represents the word location.

[0064] Step 3: The knob status recognition module is implemented as follows:

[0065] Step 3.1: Get the center points of all word boxes obtained in step 2.3, and start calculating the angle between the center point of the word box and the pointer head point with the pointer support as the starting point.

[0066] Step 3.2: Sort the angles obtained in step 3.1 from small to large to obtain the relative position of the word box and the pointer. The smaller the angle, the greater the possibility that the pointer points to the word box.

[0067] Step 3.3: The word information contained in each knob frame may not all be valid knob status identifiers. Therefore, it is necessary to filter the word validity. Count all possible legal status identifiers in advance, such as Figure 5The two words "far away" and "local" in it are legal status identifiers. The words obtained in step 3.2 are identified in the order of the included angle. The identification technology also uses deep learning technology. Here, the CRNN (Convolutional Recurrent Neural Network) network is adopted. The CRNN network is a relatively mature open-source text recognition model. After the word recognition is completed, it is judged whether it is a legal status identifier. If it is a legal identifier, this identifier is the setting state of the entire knob, otherwise, continue to identify the next word.

[0068] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form. Any technical solutions obtained by using equivalent replacements or equivalent transformations fall within the protection scope of the present invention.

Claims

1. A method for identifying the knob state of a power switch cabinet based on deep learning, characterized in that: Including steps: (1) Obtain the original image of the power switch cabinet knob taken by the inspection robot; (2) Preprocess the original image and then input it into the pre-designed and trained target detection network. The target detection network determines the location of the knob in the preprocessed image and obtains the knob detection frame. (3) Expand the knob detection frame, enlarge the width and height of the knob detection frame by a preset multiple, and then crop the knob detection frame after the expansion process to obtain a knob image containing only the knob; (4) Using the pre-designed knob pointer key point and word position detection network, perform knob pointer key point detection and word position detection on the knob image; wherein, knob key point detection is to detect the two key points of the pointer support and pointer head in the knob image, and determine the pointer direction through the pointer support and pointer head; word position detection is to detect all words in the knob image, and each word represents a state of the knob; (5) Based on the words and pointer directions obtained in step (4), calculate the angles between all words and pointer directions, and then arrange all the calculated angles in order from small to large; (6) According to the sorting results of step (5), identify the words corresponding to the angle, judge the legitimacy of the identified words, and find the first valid word; (7) The first valid word found in step (6) is determined as the current state of the knob and output; In step (4), the knob pointer key point and word position detection network includes a backbone network and a first detection head and a second detection head connected to the output end of the backbone network; the backbone network is used to extract features from the input image, i.e., the knob image, to obtain a knob feature map; The first detection head is a key point branch that outputs two-channel probability maps. The probability map of the first channel represents the probability map of the pointer pillar, and the probability map of the second channel represents the probability map of the pointer head. The position with the largest array in each probability map represents the corresponding key point position. The second detection head is a word detection branch, which outputs a word segmentation probability map, performs contour and minimum circumscribed quadrilateral operations on the word segmentation probability map, and the resulting quadrilateral box is the location of the word, that is, the word box.

2. The method for identifying the state of a power switch cabinet knob based on deep learning according to claim 1, characterized in that: In the step (2), the preprocessing of the original image includes: scaling the size of the original image to 640*640, the target detection network adopts the yolov5s knob detection model, and the obtained knob detection frame is a rectangular frame containing the knob image.

3. The method for identifying the knob status of a power switch cabinet based on deep learning according to claim 1, characterized in that: In step (3), the width and height of the knob detection frame are expanded by 1.25 times, and the center point of the frame remains unchanged.

4. The method for identifying the knob status of a power switch cabinet based on deep learning according to claim 1, characterized in that: The backbone network processes the input image as follows: The input image is passed through a convolutional layer with a stride of 4, and the input image is calculated into a feature map with 64 channels and a size of 96*72; Entering the first stage, stage 1, four bottleneck operations are performed to extract features, and then downsampled to generate feature maps of different resolutions; Enter the second stage stage2, the third stage stage3 and the fourth stage stage4 in sequence, where each stage performs the following operations: perform 4 bottleneck operations to extract features, then perform downsample operations to generate new feature maps of different resolutions. While performing the downsample operation, the feature maps with resolutions less than the preset threshold are enlarged through bilinear interpolation operations and fused with the corresponding feature maps before resolution enlargement. The size of the fused feature maps is 96*72; The fused feature map is used as the knob feature map and input into the first detection head and the second detection head.

5. The method for identifying the knob status of a power switch cabinet based on deep learning according to claim 4, characterized in that: Said step (5) specifically comprises: Obtain the center point positions of all word frames obtained in step (4), and calculate the angle formed by the pointer support as the starting point, the word frame center point and the pointer head point; All calculated angles are sorted from large to small, and the size of the angle represents the probability that the pointer points to the corresponding word box.

6. The method for identifying the state of a power switch cabinet knob based on deep learning according to claim 4, characterized in that: The step (6) specifically includes: Use the CRNN network model to complete word recognition and identify the words corresponding to the angle; The legitimacy of the recognized words is judged, that is, the words are compared with the legal status identification text counted in advance. If they are legal status identification texts, the corresponding words are determined as the current state of the knob. Otherwise, continue to recognize the next word.

7. A power switch cabinet knob state recognition device based on deep learning, characterized in that: include: The original image acquisition module is used to obtain the original image of the power switch cabinet knob taken by the inspection robot; Preprocessing module, used to preprocess the original image; The object detection network is used to identify the location of the knob in the preprocessed image and obtain the knob detection frame; The image cropping module is used to expand the knob detection frame, enlarge the width and height of the knob detection frame by a preset multiple, and then crop the knob detection frame after the expansion process to obtain a knob image containing only the knob; The knob pointer key point and word position detection network is used to detect knob pointer key points and word positions on the knob image. Knob key point detection detects two key points in the knob image: the pointer support and the pointer head. The pointer direction is determined by the pointer support and the pointer head. Word position detection detects all words in the knob image, each word representing a state of the knob. Angle calculation and sorting module, which is used to calculate the angles between all words and pointer directions based on the knob pointer key points and the words and pointer directions obtained by the word position detection network, and then arrange all the calculated angles in order from small to large; A word recognition module is used to recognize words corresponding to the angle according to the sorting results of the angle calculation and sorting module; The word legitimacy judgment module is used to judge the legitimacy of the recognized words and find the first valid word; The knob state determination module is used to determine the first valid word found by the word legitimacy judgment module as the current state of the knob and output it; The knob pointer key point and word position detection network includes a backbone network and a first detection head and a second detection head connected to the output end of the backbone network; The backbone network is used to extract features from the input image, i.e., the knob image, to obtain a knob feature map; The first detection head is a key point branch that outputs two-channel probability maps. The probability map of the first channel represents the probability map of the pointer pillar, and the probability map of the second channel represents the probability map of the pointer head. The position with the largest array in each probability map represents the corresponding key point position. The second detection head is a word detection branch, which outputs a word segmentation probability map, performs contour and minimum circumscribed quadrilateral operations on the word segmentation probability map, and the resulting quadrilateral box is the location of the word, that is, the word box.

Images