Knife switch one-key sequence control state identification method and device and storage medium

Abstract

The invention discloses a disconnecting link one-key sequence control state identification method and device and a storage medium, and the method comprises the steps: preparing a reference image which is an image marked with a disconnecting link action arm region when a disconnecting link is in an opening state, a closing state, a proximity opening state and a proximity closing state; key points are extracted from each to-be-recognized image and each reference image one by one by using a key point detection model, and matched key point pairs of the to-be-recognized images and the reference images are obtained; removing wrong key point pairs to obtain correct matching key point pairs; taking the correct matching key point pair number of each to-be-recognized image as a matching score of the to-be-recognized image and the reference image, and obtaining a disconnecting link state recognition result in combination with a judgment condition; and according to the recognition result of each to-be-recognized image, outputting a one-key sequential control recognition result of the to-be-recognized image group. According to the method, the state of the disconnecting link is judged by adopting the template image library comprising the opening image, the closing image, the proximity opening image and the proximity closing image in combination with the key point extraction and matching method, and the recognition accuracy and sensitivity are high.

Description

Technical Field

[0001] The invention belongs to the field of electric power inspection and computer vision, and relates to a one-button sequential control state recognition method, device and storage medium for a knife switch. Background Art

[0002] In substations, knife switch operation is one of the most important operation and maintenance operations. The remote control of knife switches is called "one-button sequential control", which is reflected in the change of the knife switch state from open to closed, or from closed to open. There are many types of knife switches commonly used in one-button sequential control, such as horizontal center rotation, double-arm horizontal movement, double-arm horizontal movement, horizontal telescopic movement, vertical telescopic movement, etc., which puts forward requirements for the versatility of recognition algorithms. In order to monitor the state of one-button sequential control, current recognition technologies include recognition technology based on reference images or reference videos, image recognition technology based on deep learning, and recognition technology based on electronic pressure sensors.

[0003] Among them, the image recognition method based on deep learning requires hundreds or thousands of training data for a single task, but the cost of collecting sample data in abnormal state is high and the quantity is usually insufficient, which limits the sensitivity of the model to identify abnormal state. The method based on electronic pressure sensor needs to add sensors to each knife switch, which increases the cost of deployment. The method of comparing benchmark images or videos has low implementation cost and greater practical value.

[0004] The current monitoring method against the reference image requires the preparation of the outline and straight line features of the knife switch in the template of the reference image. For example, the straight line features of the knife switch are obtained and then compared with the straight line outline of the knife switch in the closed state (for example, the action arm in the closed state forms a straight line) to determine the opening and closing state of the knife switch. These features are easily affected by the type of knife switch, shooting conditions, etc. Since outdoor shooting is inevitable during site deployment, outdoor shooting has difficulties such as large changes in lighting, large changes in shooting resolution, rainy and snowy weather, backlight and dimness, and overlapping and occlusion of on-site equipment. Therefore, the current monitoring method against the reference image has the disadvantages of complex template preparation, low recognition accuracy, and single type of knife switches supported. Summary of the invention

[0005] In order to solve the above problems, the present application provides a method for identifying a switch one-button sequential control state, comprising:

[0006] A template library is constructed, wherein the template library contains a template sub-library of each knife switch, and each template sub-library contains four reference images of the knife switch, wherein the four reference images are images of the knife switch action arm area when the knife switch is in the open, closed, nearly open, and nearly closed states, respectively.

[0007] Among them, "close to open" means that the action arm moves towards the closed state relative to the open knife gate, and the knife gate is still in the open state; "close to closed knife gate" means that the action arm moves towards the open state relative to the closed knife gate, and the knife gate is still in the closed state;

[0008] Receive a one-key sequential control instruction, obtain images of the knife switch to be identified to form an image group to be identified, search for a template sub-library corresponding to the knife switch to be identified, extract key points from each image to be identified in the image group and match them with each reference image one by one using a key point detection model, and obtain multiple matching key point pairs between the image to be identified and the reference image, wherein there are one or two images to be identified in the image group;

[0009] Use distance filtering strategy to remove wrong matching key point pairs and obtain correct matching key point pairs;

[0010] For each image to be identified in the image group to be identified, the number of correctly matched key point pairs is used as the matching score between the image to be identified and the reference image, and the matching score of the image to be identified is recorded as S according to the annotation correspondence of the reference image. 开 ,S 合 ,S 接近开 ,S 接近合 , set S M =max{S 开 ,S 合 ,S 接近开 ,S 接近合}, then the state recognition result of the image to be recognized is:

[0011]

[0012] T 2 ,T 3 is the threshold value, and they are all positive numbers;

[0013] According to the recognition result of each image to be recognized in the image group to be recognized, the one-key sequential control recognition result of the image group to be recognized is output.

[0014] Optionally, the one-key sequential control recognition of the image group to be recognized is divided into the following cases:

[0015] (1) The image group to be identified contains only a single image to be identified, and its identification result is one of open, closed, and abnormal;

[0016] (2) The image group to be identified includes two images to be identified, which correspond to the images of the switch before and after the one-key sequential control instruction respectively;

[0017] (3) The image group to be identified includes a video to be identified, and the video to be identified is converted into an image frame sequence, and then the image frames before and after the one-key sequence control command are extracted to form the image group to be identified;

[0018] For cases 2 and 3, based on the status recognition results of the two images to be recognized and their associations with the one-key sequence control instructions before and after, the final one-key sequence control recognition result is obtained.

[0019] Optionally, the degree of approaching open or approaching closed is measured by the moving distance L or the rotation angle D, and L = NL is selected. 0 Or D = ND 0 The corresponding image is used as the disconnector image of approaching open or approaching closed, and L 0 Or D 0 Is a unit movement of the action arm, 0 < N ≤ 2. Among them, for the disconnector with horizontal movement of both arms and the disconnector with horizontal center rotation, the rotation angle D of the action arm is used for measurement, and for the telescopic movement disconnector, the moving distance L of the action arm is used for measurement.

[0020] Optionally, the key point detection model LF-Net is used to extract key points and match the corresponding key points to form the matching key point pairs.

[0021] Optionally, the distance filtering strategy is used to remove the incorrect matching key point pairs to obtain the correct matching key point pairs, including: the key points in the image to be recognized are represented by And the corresponding key points of the images in the template sub-library are represented by Retain the key point pairs that satisfy ||x′ i - H(x i )|| < T 1 Where N represents a total of N key points;

[0022] Among them, N represents a total of N key points;

[0023] x i Represents the i-th key point in the image to be recognized, and x i ′ represents the i-th key point corresponding to the image in the template sub-library;

[0024] The findHomography method in the software package pydegensac is used to calculate the homography transformation y = H(x) from the key point coordinates of the image to be recognized to the key point coordinates of the reference image;

[0025] T 1 Is the threshold.

[0026] Optionally, it further includes: where the reference images are all normalized to a fixed resolution. For each image to be recognized in the group of images to be recognized, before extracting the key points, it is first normalized to the same fixed resolution as the reference image. During the normalization process, the scaling ratio is calculated according to the width and scaled. If the height value of the obtained image after scaling is greater than the fixed resolution, it is cropped, and if it is less than the fixed resolution, the image size is filled by padding 0 pixel values.

[0027] Optionally, converting the video to be identified into an image frame sequence, and then extracting the image frames before and after the one-key sequence control command to form an image group to be identified, includes:

[0028] Blank frames are also detected. The color image is converted into a grayscale image through OpenCV, and then the difference between the maximum and minimum grayscale values ​​is calculated. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame. Once it is identified as a blank frame, the image is selected in the adjacent frame.

[0029] Optionally, in the image where the knife switch actuating arm regions when the knife switch is in the open, closed, nearly open and nearly closed positions are marked, the region outside the marked region is removed.

[0030] The present application also provides an electronic device, comprising:

[0031] at least one processor; and,

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the one-button sequential control state identification method of the knife switch as described above.

[0034] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the one-button sequential control state identification method for a knife switch as described above.

[0035] The present invention has the following beneficial effects:

[0036] (1) A template library containing four images of open, closed, nearly open, and nearly closed is used in combination with key point extraction and matching methods to determine the state of the knife switch. Images of nearly open and nearly closed, which reflect the abnormal state of the knife switch, are used to assist in determining the opening and closing state of the knife switch, and minor opening and closing faults can be identified.

[0037] (2) Through pre-alignment, images with large resolution differences are normalized to similar resolutions, and then further aligned through key point detection to solve the problem of resolution differences. When problems such as blur and occlusion occur, key points in clear areas can be used for matching to reduce the impact of blur and occlusion on recognition.

[0038] (3) The matching score is obtained by extracting key points and matching methods. There is no need to identify the outer contour and straight line features of the knife switch. Therefore, even if the straight lines in the image to be identified are mixed with the straight lines of the knife switch action arm, it will not affect the row judgment result, which improves the adaptability and stability of the algorithm. The algorithm has high recognition accuracy and sensitivity, and the measured accuracy is above 99% to 99.9%;

[0039] (4) A template library key point model solution containing four images: open, closed, nearly open, and nearly closed is adopted to meet the technical characteristics of small sample learning. It does not require the collection of a large number of training images and supports minute-level deployment and update. Compared with the solution of a single deep learning model, the deployment and update speed is increased by orders of magnitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the one-button sequential control state identification method of the knife switch of the present invention;

[0041] Figure 2 This is a schematic diagram of using a polygonal frame to mark the outer contour of the knife switch actuating arm according to the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The one-button sequential control state identification method of the knife switch of this embodiment, the knife switch includes an action arm, a joint point and a contact, and the knife switch realizes the opening and closing of the switch by rotating around the joint point. The method includes the following steps:

[0044] Step S1, constructing a template library, wherein the template library contains a template sub-library for each knife switch, and each template sub-library contains four reference images of the knife switch, wherein the four reference images are images of the knife switch action arm area when the knife switch is open, closed, nearly open, and nearly closed.

[0045] The benchmark image data can be captured and collected by power inspection fixed-point cameras, robot cameras, mobile phones and other devices. The data type can be images or videos, and the videos can be processed into image formats by software. The preferred image resolution is 720P, 1080P and above.

[0046] Among them, in the process of preparing the template library, if the acquisition type is image, four reference images need to be collected, namely, images when the knife switch is in "open", "closed", "nearly open", and "nearly closed"; if the input type is video, the OpenCV video to image frame sequence method can be used to obtain the image frame sequence, and then select the reference images corresponding to the four states of the knife switch in "open", "closed", "nearly open", and "nearly closed".

[0047] Among them, the knife gate close to opening means that the action arm has moved slightly towards the closed state relative to the open knife gate, and the knife gate is still in the open state; the knife gate close to closing means that the action arm has moved slightly towards the open state relative to the closed knife gate, and the knife gate is in the closed state. It can be measured by the moving distance L or the rotation angle D, according to the actual recognition sensitivity L required 0 or D 0 , we can choose L = NL 0 Or D=ND 0 The corresponding image is used as the image of the knife switch that is close to opening or closing. 0 or D 0 is a unit action of the action arm, 0 <N≤2。

[0048] For example, a double-arm horizontal motion knife switch has two action arms that rotate around their respective joints. The ends of the action arms have contacts. When the contacts of the two action arms are rotated to touch, it indicates that the knife switch is in a closed state. When the contacts are rotated to separate, it indicates that the knife switch is in an open state. For example, a horizontal center rotation knife switch has two action arms that rotate around the joint at its midpoint, so that the moving contacts at both ends are in contact with the fixed contacts respectively, which is a closed state; the action arm rotates around the joint at its midpoint, so that the moving contacts at both ends are separated from the fixed contacts respectively, which is an open state. The rotation angle D of the action arm can be used for measurement.

[0049] For example, for a telescopic knife switch, the folded action arm is extended by rotating around its middle joint point, so that the moving contact at the end of the action arm is in contact with the fixed contact, which is the closed state; the folded action arm is contracted by rotating around its middle joint point, so that the moving contact at the end of the action arm is separated from the fixed contact, which is the open state. The moving distance L of the action arm can be used as a measurement.

[0050] After obtaining the four-state baseline images, refer to the attached Figure 2 , use a polygonal box to mark the outer contour of the knife switch action arm. The polygon should be as close to the outer contour of the action arm as possible, and the fixed area should be excluded. If other fixed areas are circled in the polygonal box when marking the outer contour of the action arm, the score of the non-matching image will increase, which may easily lead to misrecognition.

[0051] In addition, the marked area can be further extracted, and the area outside the marked area (foreground) can be removed, for example, the pixel value of the area outside the marked area is set to 0. Since the image information outside the foreground is removed, the key points of the match fall on the action arm, which reduces the interference in the background image, speeds up the matching speed, and improves the matching accuracy.

[0052] In addition, the reference image can be normalized to a fixed resolution, that is, the reference image can be normalized to the same aspect ratio. For example, if the input image aspect ratio is 16:9, the ratio of the target width to the current width can be calculated and used as the scaling ratio of the entire image to obtain the result image; if the input image aspect ratio is not a standard value, the scaling ratio can still be calculated according to the width and scaled. If the image height value obtained after scaling is too large, it needs to be cropped. If it is too small, the image size can be filled by filling the area with 0 pixel values. A smaller fixed resolution can be appropriately selected, such as 640×360 pixels, to reduce the amount of calculation in the subsequent process.

[0053] Each template sub-library has a corresponding number. Subsequently, according to the number of the input image group to be identified (corresponding to the number of the template sub-library), four reference images in the corresponding template sub-library can be obtained.

[0054] Step S2, receiving a one-key sequential control instruction, obtaining images of the knife switch to be identified to form an image group to be identified, searching the corresponding template sub-library by the number of the image group to be identified, extracting key points from each image to be identified in the image group to be identified by using a key point detection model to compare each reference image one by one, and obtaining matching key point pairs between the image to be identified and the reference image by a matching algorithm. The image group to be identified includes one or two images to be identified.

[0055] Specifically, the key point detection model LF-Net can be used to extract key points and match corresponding key points. The method in ("Learning Local Features from Images", Advances in Neural Information Processing Systems, 2018, 31, Ono, Yuki and Trulls, Eduard and Fua, Pascal and Yi, Kwang Moo) can be referred to. Thus, the key points of the image to be identified and the corresponding key points of the reference image can form multiple matching key point pairs.

[0056] In addition, for each image to be identified in the group of images to be identified, it is first normalized to the same fixed resolution as the reference image before extracting key points.

[0057] Step S3, using distance filtering strategy to remove wrong matching key point pairs, obtain correct matching key point pairs, and use the key points in the image to be identified. To represent, the corresponding key points of the image of the template sub-library are represented by To bring representatives, retain the satisfaction of ||x′-H(x i )|| <T 1 The key point pair, T 1is a threshold parameter, such as 10 pixels, which can be fine-tuned on the validation set. 1 And verify whether the matching point pairs are correct, so as to obtain a better threshold setting.

[0058] The findHomography method in the software package pydegensac may be used to calculate the homography transformation y=H(x) from the coordinates of the key points of the image to be identified to the coordinates of the key points of the reference image.

[0059] Step S4: for each image to be identified in the image group to be identified, the number of correctly matched key point pairs is used as the matching score between the image to be identified and the reference image, and the matching score of the image to be identified is recorded as S and S according to the annotation of the reference image. 开 ,S 合 ,S 接近开 ,S 接近合 , that is, if the reference image is marked as on in step S1, the matching score of the image to be identified is recorded as S 开 , in step S1, the reference image is marked as close to open, and the matching score of the image to be identified is recorded as S 接近开 . Remember S M =max{S 开 ,S 合 ,S 接近开 ,S 接近合}, then the state recognition result of the image to be recognized is:

[0060]

[0061] Among them, the state of close to open and close to close are both abnormal states of the knife switch, while open and close are normal states of the knife switch. When the scores of the normal state and the abnormal state of close to open and close are close, for example, if only S 开 ≈S 接近开 , the result is still judged as abnormal, the purpose is to reduce the missed recognition of abnormal state. Therefore, this embodiment sets the threshold T 2 ,T 3 To achieve more accurate identification. Only when the score of the normal state exceeds the abnormal state by a certain margin T 2 , the corresponding normal state is output. Random matching forms the correct key point pair of images. Due to accidental factors, a positive matching score may be obtained. Therefore, the threshold T is set 3 , only when it exceeds T 3 , is considered a valid match.

[0062] Among them, T 2 ,T 3 is the threshold value, threshold value T 2 ,T 3are all positive numbers, such as T 2 The reference value is 5 pixels, T 3 The reference value is 10 pixels. The specific value can be enumerated by the verification set and the state recognition result can be verified to be correct, so as to test and obtain T 2 ,T 3 The optimal value of .

[0063] Step S5: outputting the one-key sequence control recognition result of the image group to be recognized according to the recognition result of each image to be recognized in the image group to be recognized.

[0064] Specifically, the one-key sequential control recognition of the image group to be recognized is divided into the following cases:

[0065] (1) The image group to be identified contains only a single image to be identified, and its identification results are divided into three types: "open", "closed", and "abnormal";

[0066] (2) The image group to be identified includes two images to be identified, which correspond to the images of the switch before and after the one-key sequential control instruction respectively;

[0067] (3) The image group to be identified includes a video to be identified. The video to be identified is converted into an image frame sequence, and then the images before and after the one-key sequence control command are extracted to form an image group to be identified, and then processed according to situation 2. Among them, the video to be identified may have blank frames, so it is necessary to detect blank frames. Once identified as blank frames, images are selected from adjacent frames. To detect blank frames, OpenCV can be used to convert color images into grayscale images, and then the difference between the maximum and minimum grayscale values ​​is calculated. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame.

[0068] The recognition results of cases 2 and 3 are divided into four results: "open", "closed", "open abnormal", and "close abnormal". For cases 2 and 3, the final one-key sequential control recognition result can be obtained based on the state recognition results of the two images to be recognized and the association before and after the one-key sequential control instruction. As shown in Table 1 below, if the recognition result of the image to be recognized before the one-key sequential control instruction is open, and the recognition result of the image to be recognized after the one-key sequential control instruction is closed, it means that the knife switch changes from open to closed state, and the recognition result of the one-key sequential control is closed.

[0069] Table 1:

[0070]

[0071]

[0072] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for identifying the one-button sequential control state of a knife switch, characterized in that: Including: Construct a template image library, where each template sub-library corresponding to a switch-disconnector is included in the template image library, and each template sub-library contains four reference images of the switch-disconnector. The four reference images are images marked with the action arm area of the switch-disconnector when it is in the open, closed, near-open, and near-closed states respectively. Among them, near-open means that the action arm has moved towards the closed state relative to the open switch-disconnector, and the switch-disconnector is still in the open state; the near-closed switch-disconnector means that the action arm has moved towards the open state relative to the closed switch-disconnector, and the switch-disconnector is still in the closed state. Receive a one-key sequence control instruction, obtain the images of the switch-disconnector to be recognized to form a group of images to be recognized, search for the corresponding template sub-library of the switch-disconnector to be recognized, and for each image to be recognized in the group of images to be recognized, extract key points and match them one by one with each reference image using a key point detection model, to obtain multiple pairs of matching key points between the image to be recognized and the reference image. Among them, there is one or two images to be recognized in the group of images to be recognized. Adopt a distance filtering strategy to remove incorrect pairs of matching key points, and obtain correct pairs of matching key points. For each image to be identified in the image group to be identified, the number of correctly matched key point pairs is used as the matching score between the image to be identified and the reference image, and the matching score of the image to be identified is recorded as S according to the annotation correspondence of the reference image. 开 ,S 合 ,S 接近开 ,S 接近合 , set S M =max{S 开 ,S 合 , S 接近开 ,S 接近合 }, then the state recognition result of the image to be recognized is: T2 and T3 are thresholds, and both are positive numbers. According to the recognition results of each image to be recognized in the group of images to be recognized, output the one-key sequence control recognition result of the group of images to be recognized.

2. The method for recognizing the one-key sequence control state of a switch-disconnector according to claim 1, characterized in that The one-key sequence control recognition of the group of images to be recognized is divided into the following situations: (1) The group of images to be recognized only contains a single image to be recognized, and its recognition result is one of open, closed, and abnormal. (2) The group of images to be recognized contains two images to be recognized, corresponding to the images of the switch-disconnector before and after the one-key sequence control instruction respectively. (3) The group of images to be recognized contains a video to be recognized. By converting the video to be recognized into a sequence of image frames, and then extracting the image frames before and after the one-key sequence control instruction to form the group of images to be recognized. For situations 2 and 3, obtain the final one-key sequence control recognition result according to the state recognition results of the two images to be recognized and their association with before and after the one-key sequence control instruction.

3. The method for recognizing the one-key sequence control state of a switch-disconnector according to claim 1, characterized in that The degree of near-open and near-closed is measured by the moving distance L or the rotation angle D. Select the image corresponding to L = NL0 or D = ND0 as the image of the near-open or near-closed switch-disconnector. L0 or D0 is a unit movement of the action arm, 0 < N ≤ 2. Among them, for the switch-disconnector with horizontal movement of both arms and the switch-disconnector with horizontal center rotation, the rotation angle D of the action arm is used for measurement, and for the telescopic movement switch-disconnector, the moving distance L of the action arm is used for measurement.

4. The method for recognizing the one-key sequence control state of a switch-disconnector according to claim 1, characterized in that Adopt the key point detection model LF-Net to extract key points and match the corresponding key points to form the pairs of matching key points.

5. The method for recognizing the one-key sequence control state of a switch-disconnector according to claim 1, characterized in that The distance filtering strategy is used to remove the wrong matching key point pairs to obtain the correct matching key point pairs, including: the key points in the image to be identified are used To represent, the corresponding key points of the image of the template sub-library are represented by To represent, retain the satisfaction ||x′ i -H(x i )||<T1 key point pair, Among them, N represents a total of N key points; x i represents the i-th key point in the image to be identified, x i ′ represents the i-th key point corresponding to the image of the template sub-library; Adopt the findHomography method in the software package pydegensac to calculate the homography transformation y = H(x) from the key point coordinates of the image to be recognized to the key point coordinates of the reference image. T1 is a threshold.

6. The method for identifying the one-button sequential control state of a knife switch according to claim 4 is characterized in that: Also including: The reference images are all normalized to a fixed resolution. For each image to be identified in the group of images to be identified, before extracting the key points, it is first normalized to the same fixed resolution as the reference image. During the normalization process, the scaling ratio is calculated according to the width and scaled. If the image height value after scaling is greater than the fixed resolution, it is cropped. If it is less than the fixed resolution, the image size is padded with 0 pixels.

7. The method for identifying the one-button sequential control state of a knife switch according to claim 2, characterized in that: The step of converting the video to be identified into an image frame sequence, and then extracting the image frames before and after the one-key sequence control command to form an image group to be identified includes: Blank frames are also detected. The color image is converted into a grayscale image through OpenCV, and then the difference between the maximum and minimum grayscale values ​​is calculated. If the difference is zero, it is identified as a blank frame, otherwise it is a non-blank frame. Once it is identified as a blank frame, the image is selected in the adjacent frame.

8. The method for identifying the one-button sequential control state of a knife switch according to claim 4, characterized in that: In the image where the knife switch actuating arm regions when the knife switch is open, closed, nearly open, and nearly closed are marked, the regions outside the marked regions are removed.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the one-button sequential control state identification method of a knife switch as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for identifying the one-button sequential control state of a knife switch as described in any one of claims 1 to 8 is implemented.

Images