A method, device and storage medium for identifying one-button sequential control state of a knife switch

Abstract

The present application discloses a method, device and storage medium for identifying the one-touch sequential control state of a knife switch. The method comprises: preparing reference images, which are images that mark the knife switch action arm area when the knife switch is in the open, closed, nearly open and nearly closed positions; using a key point detection model to extract key points from each image to be identified and each reference image one by one, and obtaining matching key point pairs between the image to be identified and the reference image; removing incorrect key point pairs to obtain correct matching key point pairs; using the number of correct matching key point pairs for each image to be identified as the matching score between the image to be identified and the reference image, and combining the judgment conditions to obtain the knife switch state recognition result; the recognition results of each image to be identified are output as the one-touch sequential control recognition result of the image group to be identified. The present application uses a template library containing four images of open, closed, nearly open and nearly closed, combined with key point extraction and matching methods to make a knife switch state judgment, with high recognition accuracy and sensitivity.

Description

Technical Field

[0001] The present invention belongs to the field of electric power inspection and computer vision, and relates to a method, device and storage medium for identifying the one-button sequential control state of a knife switch. Background Art

[0002] In substations, operating knife switches is one of the most critical O&M operations. Remote control of knife switches is called "one-touch sequential control," and the switch's state changes from open to closed, or from closed to open. Common types of knife switches used in one-touch sequential control include horizontal center rotation, dual-arm horizontal motion, dual-arm horizontal motion, horizontal telescopic motion, and vertical telescopic motion, requiring versatile recognition algorithms. Current recognition technologies for monitoring one-touch sequential control status include those based on reference images or videos, deep learning-based image recognition, and electronic pressure sensor-based recognition.

[0003] Image recognition methods based on deep learning require hundreds or even thousands of training data for a single task. However, collecting abnormal sample data is costly and often insufficient, limiting the model's sensitivity in identifying abnormal conditions. Methods based on electronic pressure sensors require adding sensors to each switch, increasing deployment costs. However, methods that use comparisons with baseline images or videos offer lower implementation costs and greater practical value.

[0004] The current monitoring method based on a reference image requires preparing the outline and straight line features of the knife switch in the template of the reference image. For example, the straight line features of the knife switch are obtained and then compared with the straight line outline of the knife switch in the closed state (for example, the action arm in the closed state forms a straight line) to determine the open or closed state of the knife switch. These features are easily affected by the type of knife switch, shooting conditions, etc. Since site deployment inevitably involves outdoor shooting, outdoor shooting has difficulties such as large changes in lighting and shooting resolution, the influence of rain and snow, backlight and dimness, and overlapping and occlusion of on-site equipment. Therefore, the current monitoring method based on a reference image has shortcomings such as complex template preparation, low recognition accuracy, and a single type of knife switch supported. Summary of the Invention

[0005] To solve the above problems, the present application provides a method for identifying the one-button sequential control state of a knife switch, comprising:

[0006] Construct a template library, which contains a template sub-library for each knife switch. Each template sub-library contains four reference images of the knife switch. The four reference images are images of the knife switch operating arm area when the knife switch is open, closed, nearly open, and nearly closed.

[0007] Among them, the "close to open" state means that the actuating arm moves toward the closed state relative to the open knife gate, and the knife gate is still in the open state; the "close to closed" state means that the actuating arm moves toward the open state relative to the closed knife gate, and the knife gate is still in the closed state.

[0008] Receive a one-key sequence control command, obtain images of the knife switch to be identified to form an image group to be identified, search the template sub-library corresponding to the knife switch to be identified, extract key points from each image to be identified in the image group and match them with each reference image one by one using a key point detection model, and obtain multiple matching key point pairs between the image to be identified and the reference image, wherein there are one or two images to be identified in the image group;

[0009] Use distance filtering strategy to remove incorrect matching key point pairs and obtain correct matching key point pairs;

[0010] For each image to be identified in the image group to be identified, the number of correctly matched key point pairs is used as the matching score between the image to be identified and the reference image, and the matching score of the image to be identified is recorded as S according to the annotation of the reference image. 开 ,S 合 ,S 接近开 ,S 接近合 , set S M =max{S 开 ,S 合 ,S 接近开 ,S 接近合}, then the state recognition result of the image to be recognized is:

[0011]

[0012] T2 and T3 are thresholds, and both are positive numbers;

[0013] According to the recognition results of each image to be recognized in the image group to be recognized, the one-key sequential control recognition result of the image group to be recognized is output.

[0014] Optionally, the one-key sequential recognition of the image group to be recognized is divided into the following cases:

[0015] (1) The image group to be identified contains only a single image to be identified, and its identification result is one of open, closed, and abnormal;

[0016] (2) The image group to be identified includes two images to be identified, corresponding to the images of the switch before and after the one-button sequence control instruction respectively;

[0017] (3) The image group to be identified includes a video to be identified, which is converted into an image frame sequence, and then the image frames before and after the one-key sequence control command are extracted to form the image group to be identified;

[0018] For cases 2 and 3, the final one-key sequence control recognition result is obtained based on the status recognition results of the two images to be recognized and their associations before and after the one-key sequence control instruction.

[0019] Optionally, the degree of approaching open or approaching closed is measured by the moving distance L or the rotation angle D. The image corresponding to L = NL0 or D = ND0 is selected as the disconnector image of approaching open or approaching closed. L0 or D0 is a unit movement of the action arm, where 0 < N ≤ 2. For the disconnector with horizontal movement of both arms and the disconnector with horizontal center rotation, the rotation angle D of the action arm is used for measurement, and for the telescopic movement disconnector, the moving distance L of the action arm is used for measurement.

[0020] Optionally, the key point detection model LF-Net is used to extract key points and match the corresponding key points to form the matched key point pairs.

[0021] Optionally, the use of the distance filtering strategy to remove incorrect matched key point pairs and obtain correct matched key point pairs includes: the key points in the image to be recognized are represented by and the corresponding key points of the images in the template sub-library are represented by . The key point pairs that satisfy ||x′ i - H(x i )|| < T1 are retained,

[0022] where N represents a total of N key points;

[0023] x i represents the i-th key point in the image to be recognized, and x i ′ represents the i-th key point corresponding to the image in the template sub-library;

[0024] The homography transformation y = H(x) from the key point coordinates of the image to be recognized to the key point coordinates of the reference image is calculated using the findHomography method in the software package pydegensac;

[0025] T1 is the threshold.

[0026] Optionally, it further includes: where the reference images are all normalized to a fixed resolution. For each image to be recognized in the group of images to be recognized, before extracting the key points, it is first normalized to the same fixed resolution as the reference image. During the normalization process, the scaling ratio is calculated and scaled according to the width. If the height value of the image obtained after scaling is greater than the fixed resolution, it is cropped, and if it is less than the fixed resolution, the image size is filled by padding with 0 pixel values.

[0027] Optionally, the conversion of the video to be recognized into an image frame sequence, and then the extraction of the image frames before and after the one-key sequence control instruction to form the group of images to be recognized includes:

[0028] Blank frames are also detected. The color image is converted into a grayscale image through OpenCV, and then the difference between the maximum and minimum grayscale values is calculated. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame. Once it is identified as a blank frame, an image is selected from the adjacent frames.

[0029] Optionally, in the image where the knife switch actuating arm regions when the knife switch is in the open, closed, nearly open and nearly closed positions are marked, the regions outside the marked regions are removed.

[0030] The present application also provides an electronic device, comprising:

[0031] at least one processor; and,

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the one-button sequential control state identification method of the knife switch as described above.

[0034] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the one-button sequential control state identification method for a knife switch as described above.

[0035] The present invention has the following beneficial effects:

[0036] (1) A template library containing four images of open, closed, nearly open, and nearly closed is used in combination with key point extraction and matching methods to determine the state of the knife switch. Images of nearly open and nearly closed, which reflect the abnormal state of the knife switch, are used to assist in judging the opening and closing state of the knife switch, and minor opening and closing faults can be identified.

[0037] (2) Through pre-alignment, images with large resolution differences are normalized to similar resolutions, and then further aligned through key point detection to solve the problem of resolution differences; when problems such as blur and occlusion occur, key points in clear areas can be used for matching to reduce the impact of blur and occlusion on recognition;

[0038] (3) The matching score is obtained by extracting and matching key points. There is no need to identify the outer contour and straight line features of the knife switch. Therefore, even if the straight lines in the image to be identified are mixed with the straight lines of the knife switch action arm, it will not affect the row judgment result. This improves the adaptability and stability of the algorithm. The algorithm has high recognition accuracy and sensitivity, and the measured accuracy is above 99% to 99.9%.

[0039] (4) A template library key point model solution containing four images of open, closed, nearly open, and nearly closed is adopted to meet the technical characteristics of small sample learning. It does not require a large number of training images and supports minute-level deployment and update. Compared with the solution of a single deep learning model, the deployment and update speed is increased by orders of magnitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the method for identifying the one-button sequential control state of a knife switch according to the present invention;

[0041] Figure 2 This is a schematic diagram of using a polygonal frame to mark the outer contour of the knife switch operating arm according to the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] The one-button sequential control state identification method of the knife switch of this embodiment includes an action arm, a joint point and a contact. The knife switch realizes opening and closing of the switch by rotating around the joint point. The method includes the following steps:

[0044] Step S1: construct a template library, which contains a template sub-library for each knife switch. Each template sub-library contains four reference images of the knife switch. The four reference images are images of the knife switch action arm area when the knife switch is open, closed, nearly open, and nearly closed.

[0045] Baseline image data can be captured and collected using devices such as power inspection cameras, robotic cameras, and mobile phones. The data type can be either images or videos, and videos can be processed into image formats using software. The preferred image resolution is 720P, 1080P, or higher.

[0046] Among them, in the process of preparing the template library, if the acquisition type is image, four reference images need to be collected, namely the images when the knife switch is in "open", "closed", "nearly open", and "nearly closed"; if the input type is video, the OpenCV video to image frame sequence method can be used to obtain the image frame sequence, and then select the reference images corresponding to the four states when the knife switch is in "open", "closed", "nearly open", and "nearly closed".

[0047] Among them, the knife gate that is close to opening means that the action arm has moved slightly toward the closed state relative to the open knife gate, and the knife gate is still in the open state; the knife gate that is close to closing means that the action arm has moved slightly toward the open state relative to the closed knife gate, and the knife gate is in the closed state. It can be measured by the moving distance L or the rotation angle D. According to the actual required recognition sensitivity L0 or D0, the image corresponding to L=NL0 or D=ND0 can be selected as the knife gate image that is close to opening or close. The so-called recognition sensitivity L0 or D0 is a unit movement of the action arm, 0 <N≤2。

[0048] For example, a two-arm horizontal motion knife switch has two actuator arms that rotate around their respective joints. The ends of the actuator arms have contacts. Rotating until the contacts of the two actuator arms touch indicates the knife switch is closed, and rotating until the two contacts separate indicates the knife switch is open. For example, a horizontal center rotation knife switch has two actuator arms that rotate around their midpoint joints, causing the moving contacts at each end to contact the fixed contacts for a closed state. Rotating around their midpoint joints causes the moving contacts at each end to separate from the fixed contacts for an open state. The rotation angle D of the actuator arm can be used to measure this.

[0049] For example, in a telescopic knife switch, the folded operating arm extends by rotating around its center joint, causing the moving contact at the end of the operating arm to contact the fixed contact (the closed state); the folded operating arm retracts by rotating around its center joint, causing the moving contact at the end of the operating arm to separate from the fixed contact (the open state). The distance L traveled by the operating arm can be used as a measurement.

[0050] After obtaining the four-state reference images, refer to the attached Figure 2 A polygonal box is used to mark the outer contour of the knife switch action arm. The polygon should be as close to the outer contour of the action arm as possible, and the fixed area should be excluded. If other fixed areas are circled in the polygonal box when marking the outer contour of the action arm, the score of the non-matching image will increase, which may easily lead to incorrect recognition.

[0051] In addition, we can further extract the annotated area and remove the area outside the annotated area (foreground), for example, setting the pixel values outside the annotated area to 0. Since all image information outside the foreground is removed, the matching key points fall on the action arm, which reduces interference from the background image, speeds up matching, and improves matching accuracy.

[0052] In addition, you can also normalize the reference image to a fixed resolution, that is, normalize the reference image to the same aspect ratio. For example, if the input image aspect ratio is 16:9, you can calculate the ratio of the target width to the current width and use it as the scaling ratio for the entire image to obtain the resulting image. If the input image aspect ratio is not a standard value, you can still calculate the scaling ratio based on the width and scale it. If the height value of the image after scaling is too large, it needs to be cropped. If it is too small, the image size can be filled with zero pixels. You can appropriately choose a smaller fixed resolution, such as 640×360 pixels, to reduce the amount of calculation in subsequent processes.

[0053] Each template sub-library has a corresponding number. Subsequently, according to the number of the input image group to be identified (corresponding to the number of the template sub-library), the four reference images in the corresponding template sub-library can be obtained.

[0054] Step S2: Receive a one-button sequence control command, obtain images of the switch to be identified, and form an image group to be identified. Search the corresponding template sub-library based on the number of the image group to be identified. Use the key point detection model to extract key points from each image in the image group to be identified, and then use a matching algorithm to obtain matching key point pairs between the image to be identified and the reference image. The image group to be identified may include one or two images to be identified.

[0055] Specifically, the keypoint detection model LF-Net can be used to extract keypoints and match corresponding keypoints. Refer to the methods in ("Learning Local Features from Images," Advances in Neural Information Processing Systems, 2018, 31, Ono, Yuki and Trulls, Eduard and Fua, Pascal and Yi, Kwang Moo). The keypoints of the image to be identified and the corresponding keypoints of the reference image can form multiple matching keypoint pairs.

[0056] In addition, for each image to be identified in the image group to be identified, it is first normalized to the same fixed resolution as the reference image before key points are extracted.

[0057] Step S3: Use distance filtering strategy to remove wrong matching key point pairs and obtain correct matching key point pairs. To represent, the corresponding key points of the image of the template sub-library are represented by To bring representatives, retain the satisfaction of ||x′-H(x i)||Pairs of key points of T1, where T1 is a threshold parameter, such as 10 pixels. By fine-tuning T1 on the validation set and verifying whether the matching key point pairs are correct, a better threshold setting can be obtained.

[0058] Among them, the findHomography method in the software package pydegensac can be used to calculate the homography transformation y = H(x) from the key point coordinates of the image to be recognized to the key point coordinates of the reference image.

[0059] Step S4: For each image to be recognized in the group of images to be recognized, use the number of correctly matched key point pairs as the matching score between the image to be recognized and the reference image. Record the matching scores of the images to be recognized according to the annotations of the reference image as S 开 , S 合 , S 接近开 , S 接近合 , that is, if the annotation of the reference image in step S1 is open, the matching score of the corresponding image to be recognized is recorded as S 开 , if the annotation of the reference image in step S1 is close to open, the matching score of the corresponding image to be recognized is recorded as S 接近开 . Record S M = max{S 开 , S 合 , S 接近开 , S 接近合}, then the state recognition result of the image to be recognized is:

[0060]

[0061] Among them, close to open and close to closed states both belong to the abnormal states of the disconnecting switch, while open and closed belong to the normal states of the disconnecting switch. When the scores of the normal state and the abnormal states such as close to open and close to closed are close, for example, if only S 开 ≈ S 接近开 , the determination result is still abnormal. The purpose is to reduce the missed recognition of abnormal states. Therefore, in this embodiment, by setting thresholds T2 and T3, a more accurate recognition purpose is achieved. Only when the score of the normal state exceeds a certain margin T2 of the abnormal state, the corresponding normal state is output. For an image that randomly matches to form correct key point pairs, due to accidental factors, it may also obtain a positive matching score. Therefore, a threshold T3 is set, and only when it exceeds T3, it is considered a valid match.

[0062] Among them, T2 and T3 are thresholds, and the thresholds T2 and T3 are both positive numbers. For example, the reference value of T2 is 5 pixels and the reference value of T3 is 10 pixels. The specific values can be enumerated through the validation set and verify whether the state recognition result is correct, so as to test and obtain the optimal values of T2 and T3.

[0063] Step S5 , outputting a one-key sequence recognition result of the image group to be recognized according to the recognition result of each image to be recognized in the image group to be recognized.

[0064] Specifically, the one-key sequential control recognition of the image group to be recognized is divided into the following cases:

[0065] (1) The image group to be identified contains only a single image to be identified, and its recognition results are divided into three types: "open", "closed", and "abnormal";

[0066] (2) The image group to be identified includes two images to be identified, corresponding to the images of the switch before and after the one-button sequence control instruction respectively;

[0067] (3) The image group to be identified contains a video to be identified. The video to be identified is converted into an image frame sequence, and then the images before and after the one-key sequence control command are extracted to form the image group to be identified. Then, the image group is processed according to Case 2. Among them, the video to be identified may have blank frames, so it is necessary to detect blank frames. Once a blank frame is identified, an image is selected from the adjacent frame. To detect a blank frame, OpenCV can be used to convert the color image into a grayscale image, and then calculate the difference between the maximum and minimum grayscale values. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame.

[0068] The recognition results for Cases 2 and 3 are categorized as "Open," "Closed," "Open Abnormal," and "Close Abnormal." For Cases 2 and 3, the final one-touch sequence recognition result can be derived based on the state recognition results of the two images to be recognized and their association with the state before and after the one-touch sequence instruction. As shown in Table 1 below, if the recognition result of the image to be recognized before the one-touch sequence instruction is "Open," and the recognition result of the image to be recognized after the one-touch sequence instruction is "Closed," indicating that the switch is changing from open to closed, then the one-touch sequence recognition result is "Closed."

[0069] Table 1:

[0070]

[0071]

[0072] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications shall fall within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying the one-button sequential control state of a knife switch, characterized in that: Comprising: Construct a template image library, which contains a template sub-library for each disconnecting switch. Each template sub-library contains four reference images of the disconnecting switch. The four reference images are images that mark the action arm area of the disconnecting switch when it is in the open, closed, near-open, and near-closed states respectively. Among them, near-open means that the action arm has moved towards the closed state relative to the open disconnecting switch, and the disconnecting switch is still in the open state; the near-closed disconnecting switch means that the action arm has moved towards the open state relative to the closed disconnecting switch, and the disconnecting switch is still in the closed state. Receive a one-key sequence control instruction, obtain the images of the disconnecting switch to be recognized to form a group of images to be recognized, find the corresponding template sub-library of the disconnecting switch to be recognized, and for each image to be recognized in the group of images to be recognized, extract key points and match them one by one with each reference image using a key point detection model, obtaining multiple pairs of matching key points between the image to be recognized and the reference image. Among them, there is one or two images to be recognized in the group of images to be recognized. Adopt a distance filtering strategy to remove the wrong pairs of matching key points and obtain the correct pairs of matching key points. For each image to be identified in the image group to be identified, the number of correctly matched key point pairs is used as the matching score between the image to be identified and the reference image, and the matching score of the image to be identified is recorded as S according to the annotation of the reference image. 开 ,S 合 ,S 接近开 ,S 接近合 , set S M =max{S 开 ,S 合 ,S 接近开 ,S 接近合 }, then the state recognition result of the image to be recognized is: T2 and T3 are thresholds and are both positive numbers. According to the recognition results of each image to be recognized in the group of images to be recognized, output the one-key sequence control recognition result of the group of images to be recognized. The one-key sequence control recognition of the group of images to be recognized is divided into the following situations: (1) The group of images to be recognized only contains a single image to be recognized, and its recognition result is one of open, closed, and abnormal. (2) The group of images to be recognized contains two images to be recognized, corresponding to the images of the disconnecting switch before and after the one-key sequence control instruction respectively. (3) The group of images to be recognized contains a video to be recognized. By converting the video to be recognized into a sequence of image frames, and then extracting the image frames before and after the one-key sequence control instruction to form the group of images to be recognized. For situations 2 and 3, obtain the final one-key sequence control recognition result according to the state recognition results of the two images to be recognized and their association with before and after the one-key sequence control instruction.

2. The method for recognizing the one-key sequence control state of a disconnecting switch according to claim 1, wherein The degree of near-open and near-closed is measured by the moving distance L or the rotation angle D. Select the image corresponding to L = NL0 or D = ND0 as the image of the near-open or near-closed disconnecting switch. L0 or D0 is a unit action of the action arm, and 0 < N ≤ 2. Among them, for the disconnecting switch with horizontal movement of both arms and the disconnecting switch with horizontal center rotation, the rotation angle D of the action arm is used for measurement, and for the telescopic movement disconnecting switch, the moving distance L of the action arm is used for measurement.

3. The method for recognizing the one-key sequence control state of a disconnecting switch according to claim 1, wherein Use the key point detection model LF-Net to extract key points and match the corresponding key points to form the pairs of matching key points.

4. The method for recognizing the one-key sequence control state of a disconnecting switch according to claim 1, wherein The method of using a distance filtering strategy to remove incorrect matching key point pairs and obtain correct matching key point pairs includes: the key points in the image to be recognized are represented by and the corresponding key points of the images in the template sub-library are represented by . Key point pairs that satisfy ∥x i ′ -H(x i )∥ < T1 are retained. Among them, N represents a total of N key points; x i represents the i-th key point in the image to be identified, x i ′ represents the i-th key point corresponding to the image of the template sub-library; Use the findHomography method in the software package pydegensac to calculate the homography transformation y = H(x) from the key point coordinates of the image to be recognized to the key point coordinates of the reference image. T1 is a threshold.

5. The method for identifying the one-button sequential control state of a knife switch according to claim 3, characterized in that: It further comprises: The reference images are all normalized to a fixed resolution. For each image in the group of images to be identified, before extracting key points, it is first normalized to the same fixed resolution as the reference image. During the normalization process, the scaling ratio is calculated according to the width and the image is scaled. If the height value of the image after scaling is greater than the fixed resolution, it is cropped. If it is less than the fixed resolution, the image size is padded with zero pixels.

6. The method for identifying the one-button sequential control state of a knife switch according to claim 1, characterized in that: The process of converting the video to be identified into an image frame sequence and then extracting the image frames before and after the one-key sequence control command to form an image group to be identified includes: Blank frames are also detected. The color image is converted into a grayscale image through OpenCV, and then the difference between the maximum and minimum grayscale values is calculated. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame. Once it is identified as a blank frame, an image is selected from the adjacent frames.

7. The method for identifying the one-button sequential control state of a knife switch according to claim 3, characterized in that: In the image where the knife switch operating arm regions when the knife switch is in the open, closed, nearly open and nearly closed positions are marked, the regions outside the marked regions are removed.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the one-button sequential control state identification method for a knife switch as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying the one-button sequential control state of a knife switch as claimed in any one of claims 1 to 7 is implemented.

Images