An opening and closing state recognition method, device and storage medium for an opening and closing component

Abstract

The present application discloses a method, device, and storage medium for identifying the opening and closing state of an opening and closing member. The method includes: collecting and annotating the image data of the action arm, joint points, and contact points in the opening and closing state, and training the target detection model until the loss function output of the target detection model is less than a set threshold; using the trained target detection model to identify the image to be recognized, and outputting the target detection frames of the action arm, joint points, and contact points in the opening and closing state; constructing a geometric determination rule using the geometric relationship between the action arm, joint points, and contact points, and using the geometric determination rule to determine the opening and closing state of the opening and closing member. The present application can identify the opening and closing state of the opening and closing member only by combining the target detection frame with the geometric determination rule, with good versatility and high recognition accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition. Specifically, it relates to a method, device, and storage medium for recognizing the opening and closing state of an opening and closing member. Background Art

[0002] The traditional single inspection method in substations is manual inspection, which has problems such as high labor intensity, low inspection efficiency, incomplete inspection, and difficult visual discrimination during inspection under harsh conditions such as rain and snow. For knife-switch type electrical switches, currently, inspection can also be carried out using inspection point cameras, inspection robots, or identification based on electronic pressure sensors.

[0003] Among them, for the identification based on an electronic pressure sensor, a pressure sensor is set at the position where the knife-switch moves to open and close to sense the opening and closing state of the knife-switch. Although it can solve the problem of state identification to a certain extent, additional sensing devices need to be added and maintained for each knife-switch, which are easily damaged in the outdoor environment, increasing the deployment and maintenance costs.

[0004] Therefore, considering costs, using inspection point cameras and inspection robots for inspection has a greater application prospect. Inspection point cameras and inspection robots are based on image recognition. By recognizing the state of the knife-switch in the captured image, for example, in the article "Three-dimensional State Recognition Method of Power Knife-Switch Based on Point Cloud Feature Extraction" (Journal of Electric Power Science and Technology, Vol. 37, No. 3, May 2022), a recognition method based on 3D point cloud is adopted. However, this method is not applicable to the input of a single planar image and requires adding a depth camera device or estimating depth information. The equipment cost is still high, and estimating depth information will also introduce estimation errors, resulting in a decrease in the recognition accuracy.

[0005] Therefore, there is currently a method of segmenting the outer contours of the operating arm, joint points, and contacts of the disconnect switch from the captured two-dimensional images, and then judging its opening and closing states based on the outer contours. However, this involves image segmentation algorithms, which have a large amount of calculation, and the segmentation accuracy will also affect the subsequent opening and closing judgment. Therefore, there is also a method of judging through a simplified recognition method. For example, in the article "The Automatic Identification Method of Switch State" (International Journal of Simulation: Systems Science and Technology, Vol. 17, pp. 211-214, 2016), an identification method based on line detection is adopted to detect the linear patterns existing on the operating arm of the disconnect switch. When the disconnect switch is in the open or closed state, the operating arm and the lines on the operating arm are located in different regions of the picture, and the state of the disconnect switch is judged accordingly. This method is not universal and cannot judge the state of the disconnect switch for those with a curved outer contour, so it cannot be applied to various types of disconnect switches. In addition, if there are many other linear elements in the picture background, such as wires, it will cause great interference to the recognition result.

[0006] There is also a method of identifying by using the detection frame containing the operating arm and the detection frame of the insulator. For example, in the article "Research on the Identification Method of Disconnect Switch State Based on Improved Deep Learning" (Electric Power and Instrumentation, Vol. 55, No. 5, March 10, 2018), an identification method of detecting the connectivity of the detection frame containing the operating arm of the disconnect switch and the detection frame of the insulator in the image is adopted. This method is also not universal. Due to detecting connectivity, if other devices are captured in the connected area, it will affect the judgment of connectivity and thus affect the recognition result. Summary of the Invention

[0007] To solve the above problems, the present application provides a method for identifying the opening and closing state of an opening and closing component, including:

[0008] Collect and label the image data of the operating arm, joint points, and contacts of the opening and closing component in the open state and the closed state, and use the labeled image data as training data to train the target detection model until the loss function output of the target detection model is less than the set threshold to obtain the trained target detection model.

[0009] Among them, during labeling, for the double-arm movement opening and closing component, the two operating arms in the closed state share one operating arm detection frame; for the central rotation opening and closing component, the fixed contact and the moving contact in the closed state are in contact and combined into a closed contact, sharing one contact detection frame; for the telescopic movement opening and closing component, the fixed contact and the moving contact in the closed state are combined into a closed contact, sharing one contact detection frame.

[0010] Use the trained object detection model to identify the image to be recognized, and output the object detection frames of the action arm, joint points, and contact points in the open and closed states;

[0011] Construct geometric determination rules using the geometric relationships of the action arm, joint points, and contact points, and use the geometric determination rules to determine the open and closed states of the opening and closing components.

[0012] Optionally, the loss function of the object detection model is as follows:

[0013] loss = loss d +γ·loss GHM

[0014] where loss d is the original loss function of the object detection model, γ is the weight parameter, and loss GHM is the GHM loss function.

[0015] Optionally, the geometric determination rules include the first determination rule applicable to the opening and closing components of the double-arm movement:

[0016]

[0017] where jcd i represents the i-th contact point detection frame, 1 ≤ i ≤ 2;

[0018] jgd i represents the i-th joint point detection frame;

[0019] e i =(jgd i , jcd i ) represents the vector from the center point of jgd i to the center point of jcd i ;

[0020] dist(jcd1, jcd2) represents the Euclidean distance between jcd1 and jcd2, and dist(jgd1, jgd2) represents the Euclidean distance between jgd1 and jgd2, satisfying 0 < T7 < T8 < 1;

[0021] T1, T2, T7, and T8 are set thresholds.

[0022] Optionally, the geometric determination rules include the second determination rule applicable to the opening and closing components of the double-arm movement,

[0023] Second determination rule:

[0024]

[0025] Among them, jh represents the detection frame of the action arm in the closed state, and jf represents the detection frame of the action arm in the open state.

[0026] Optionally, the middle region M is formed by taking all the vertices of the action arm detection frame as a set of sample points, and the principal component direction is calculated for the sample points through principal component analysis. For each vertex, the vertex coordinates are added and subtracted from to achieve contraction towards the central region. L is the projected distance from the vertex to the center of the action arm detection frame, γ ∈ (0, 1) is the contraction ratio, and all the contracted vertices demarcate the middle region M.

[0027] Optionally, the geometric determination rule includes a third determination rule applicable to the double-arm movement opening and closing component:

[0028]

[0029] Among them, the preset frame is a polygon frame including the left and right joint points and the connection area of the opening and closing component in the pre-shot image.

[0030] Optionally, the geometric determination rule includes a fourth determination rule applicable to the central rotation opening and closing component:

[0031]

[0032] Among them, xhcd represents the closed contact detection frame, xfcdg represents the fixed contact detection frame, and xfcdy represents the moving contact detection frame.

[0033] Among them, it is judged whether the contact detection frame is within the action arm detection frame by whether the center point of the contact detection frame is within the action arm detection frame.

[0034] Optionally, if the number of contacts of the same contact type in the action arm detection frame exceeds two, the redundant contacts are excluded by enumerating the necessary condition that the center points of the contact detection frames of this contact type and the center points of the joint detection frames should be collinear.

[0035] Optionally, the geometric determination rule includes a fifth determination rule applicable to the central rotation opening and closing component.

[0036] Fifth determination rule:

[0037]

[0038] Among them, xh represents the detection frame of the action arm in the closed state, and xf represents the detection frame of the action arm in the open state.

[0039] Optionally, the geometric determination rule includes a sixth determination rule applicable to the telescopic opening and closing component:

[0040]

[0041] Among them, when the contact type is yh, it means that the fixed contact and the moving contact are in contact, which is called the closed contact detection frame;

[0042] ygd m represents the intermediate joint point detection frame, ygd e represents the end-side joint point detection frame;

[0043] When the contact type is yf, it represents the fixed contact detection frame;

[0044] e yf，e =(yf, ygd e ) represents the vector pointing from the center point of yf to the center point of ygd e ; e m，e =(ygd m , ygd e ) represents the vector pointing from the center point of ygd m to the center point of ygd e ;

[0045] yfcd represents the moving contact detection frame;

[0046] dist(yfcd, ygd e ) represents the distance from the center point of the moving contact detection frame to the center point of the end-side joint point detection frame;

[0047] dist(yf, ygd e ) represents the distance from the center point of the fixed contact detection frame to the center point of the end-side joint point detection frame;

[0048] T 11 is the set threshold.

[0049] Optionally, the geometric determination rule includes a seventh determination rule applicable to the telescopic opening and closing member:

[0050]

[0051] Among them, ybd represents the action arm detection frame.

[0052] Optionally, the geometric determination rule includes an eighth determination rule applicable to the telescopic opening and closing member:

[0053]

[0054] Among them, the preset frame is a polygon frame including the end-side joint point, the fixed contact, and the connection area in the pre-shot image.

[0055] Optionally, image data under various shooting conditions are generated by data augmentation, including converting the image to the HSV color format and simulating different lighting changes by adjusting the saturation and / or brightness values.

[0056] Optionally, image data under various shooting conditions are generated by data augmentation, including generating random noises with different densities using OpenCV to simulate rain and snow of different sizes and superimposing them on the original image to simulate rainy and snowy weather.

[0057] Optionally, image data under various shooting conditions are generated by data augmentation, including simulating rainy and snowy weather using deep learning-based style transfer.

[0058] Optionally, the object detection model is YOLO or Transformer, and the center point of the object detection box is determined by the vertex mean, or the object detection model is mask-rcnn, and the center point of the object detection box is determined by equally spacing sampling points inside the mask and calculating the average value of all sampling points.

[0059] Optionally, in the first to third determination rules, if the detection box of the open-state action arm is consistent with the outer bounding box of the corresponding contact detection box and joint point detection box on the same side, it is considered that the detection box of the open-state action arm is accurately recognized;

[0060] If the detection box of the closed-state action arm is consistent with the outer bounding box of the two joint point detection boxes, it is considered that the detection box of the closed-state action arm is accurately recognized;

[0061] If the outer bounding box of the two contact detection boxes is located in the middle of the action arm detection box, it is considered that the detection box of the closed-state action arm is accurately recognized.

[0062] This application also provides an electronic device, which includes:

[0063] At least one processor; and,

[0064] A memory communicatively connected to the at least one processor; wherein,

[0065] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for identifying the opening and closing states of the opening and closing member as described above.

[0066] This application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it implements the method for identifying the opening and closing states of the opening and closing member as described above.

[0067] This application has the following beneficial effects:

[0068] (1) Since it is not necessary to segment the outline of the disconnecting switch, nor to identify the straight lines representing the outline of the disconnecting switch, but only to obtain the detection frame containing the target, its versatility is good, and it can be applied to various common types of disconnecting switches and is easy to deploy.

[0069] (2) Since it is not necessary to segment the outline of the disconnecting switch, nor to identify the straight lines representing the outline of the disconnecting switch, and the state of the disconnecting switch can be identified only by combining the target detection frame with some geometric determination rules, the response time is short, and the result can be returned within seconds.

[0070] (3) Through a variety of geometric determination rules, as long as any one of them is met, the opening and closing state can be determined, thereby making up for the influence of different shooting angles, image blurring, and occlusion on the recognition accuracy.

[0071] (4) Since it is not necessary to segment the outline of the disconnecting switch, the error of image segmentation is reduced, nor is it necessary to identify the straight lines representing the outline of the disconnecting switch, which can avoid confusion with the straight lines in the background, and the recognition accuracy is high. The measured accuracy rate reaches more than 99% - 99.9%.

[0072] (5) By using the method of target detection to identify the target detection frame including the contacts, joints and action arms of the disconnecting switch to be detected, it can support the detection of multiple targets in a single image, which is beneficial to reducing the camera shooting points and lowering the cost.

[0073] (6) By simulating training images with different light changes and rain and snow effects, the problem of the influence of light changes and rain and snow environments on the recognition accuracy can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flowchart for determining the opening and closing state of the opening and closing member in the embodiment of the present invention for a double-arm horizontal movement disconnecting switch.

[0075] Figure 2 It is a schematic diagram of the target detection frame of the open state of the double-arm horizontal movement disconnecting switch in the embodiment of the present invention.

[0076] Figure 3 It is a schematic diagram of the target detection frame of the closed state of the double-arm horizontal movement disconnecting switch in the embodiment of the present invention.

[0077] Figure 4 It is a schematic diagram of the preset frame of the double-arm horizontal movement disconnecting switch in the embodiment of the present invention Figure 1 。

[0078] Figure 5 It is a schematic diagram of the preset frame of the double-arm horizontal movement disconnecting switch in the embodiment of the present invention Figure 2 。

[0079] Figure 6 It is a schematic diagram of the target detection frame of the open state of the center-rotating disconnecting switch in the embodiment of the present invention.

[0080] Figure 7 Schematic diagram of the target detection frame in the closed state of the center-rotating disconnecting switch in the embodiment of the present invention.

[0081] Figure 8 Schematic diagram of the target detection frame in the open state of the telescopic disconnecting switch in the embodiment of the present invention.

[0082] Figure 9 Schematic diagram of the target detection frame in the closed state of the telescopic disconnecting switch in the embodiment of the present invention. Detailed implementation manners

[0083] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0084] This embodiment provides a method for identifying the opening and closing states of an opening and closing member, which can be used to identify the opening and closing states of all opening and closing members that rotate around a joint point to contact the contact points. This embodiment takes the identification of the opening and closing states of the disconnecting switch in electrical equipment as an example for illustration, but is not limited thereto. As long as the moving arm rotates around the joint point to contact the contact points, the method of this application can be adopted. For example, it can also be the identification of the opening and closing states of the barrier gate railing rotating around the joint point, or the identification of the states of the track rotating around the joint point to be engaged or disconnected. Only the disconnecting switch is taken as an example for illustration below. The disconnecting switch includes a moving arm, a joint point, and a contact point. The disconnecting switch realizes the opening and closing of the switch through mechanical movement, where the moving arm rotates around the joint point. For example, for a double-arm horizontally moving disconnecting switch, the two moving arms rotate around their respective joint points, and the end of the moving arm has a contact point. When the contact points of the two moving arms rotate to contact, it means the disconnecting switch is in the closed state; when the two contact points rotate to separate, it means the disconnecting switch is in the open state. For example, for a horizontal center-rotating disconnecting switch, the moving arm rotates around the joint point at its midpoint, and when the moving contact points at both ends contact the fixed contact points respectively, it is in the closed state; when the moving arm rotates around the joint point at its midpoint and the moving contact points at both ends are separated from the fixed contact points respectively, it is in the open state. For example, for a telescopic disconnecting switch, the folded moving arm rotates and extends around the middle joint point, and when the moving contact point at the end of the moving arm contacts the fixed contact point, it is in the closed state; when the folded moving arm rotates and contracts around the middle joint point, and the moving contact point at the end of the moving arm is separated from the fixed contact point, it is in the open state.

[0085] This method is as shown in Figure 1 the flowchart shown below and includes the following steps:

[0086] Step S1, collect and label the action arms, joint points, and contact point image data of the opening and closing components in the open state and the closed state, and use the labeled image data as training data to train the object detection model until the loss function output of the object detection model is less than the set threshold to obtain the trained object detection model.

[0087] By adding data of various knife switch types to the image data, the object detection model can identify various common knife switch types. Among them, during labeling, for the double-arm horizontal movement knife switch, the two action arms in the closed state are unified into one action arm and share one action arm detection frame. For the central rotation knife switch, the fixed contact and the moving contact in the closed state are in contact and merged into the closed contact, sharing one contact detection frame. For the telescopic movement knife switch, the fixed contact and the moving contact in the closed state are merged into the closed contact, sharing one contact detection frame, and thus various required object detection frames can be obtained.

[0088] Specifically, the knife switch types in the image data can cover all knife switch types that need to be identified in the deployment scenario, and the collection angles are diverse, which is beneficial to improving the stability of the object detection model. The image data can be collected by devices such as fixed-point cameras, robot cameras, and mobile phones in power inspections. The data type can be photos or videos, and the videos can be processed into picture formats by software. The typical picture resolution is 720P, 1080P and above. If the resolution is less than 720P, the target size should be greater than 100x200 pixels.

[0089] Specifically, there is no fixed requirement for the collected lighting, weather, and clarity, and image data under various shooting conditions can be generated by data augmentation. For example, the image can be converted to the HSV color format, and different lighting changes can be simulated by adjusting the saturation and brightness values.

[0090] For example, the image under rain and snow environment can also be simulated by image processing. For example, different densities of random noise can be generated in OpenCV to simulate different sizes of rain and snow, and they can be superimposed on the original image to achieve the rain and snow effect, or the rain and snow weather can be simulated based on deep learning style transfer, which can refer to the method in the article "WeatherGAN: Unsupervised multi-weather image-to-image translation via single content-preserving UResNet generator" (Multimedia Tools and Applications, Vol. 81, 2022). By simulating different lighting changes, the image data with rain and snow effects can solve the problem of the influence of lighting changes and rain and snow environment on the recognition accuracy. Taking the double-arm horizontal movement knife switch as an example, inFigure 2 The contacts, joint points, and moving arms of the double - arm horizontal - movement - type disconnecting switch marked with the open state are, in Figure 3 The contacts, joint points, and moving arms of the double - arm horizontal - movement - type disconnecting switch marked with the closed state, and its two moving arms are framed by the same moving - arm detection frame. Different - thickness detection frames are used to distinguish the moving arms, joint points, and contacts, and the lines of the detection frames for the moving arms, joint points, and contacts gradually thicken. The defined contacts, joint points, and moving arms have certain redundant information. For example, for a single moving arm, the circumscribed rectangle of the detection frames of the contacts and joint points can also be used as the moving - arm detection frame. At this time, identifying the moving - arm detection frame is redundant, but this redundant information can be used to improve the recognition accuracy (described below).

[0091] There are already many general networks in the object - detection model based on deep learning. For example, the effectiveness of YOLO and Transformer in object detection has been widely verified in fields such as face recognition and unmanned retail. It has been verified that the recognition effect of general networks can already reach excellent levels. For example, mAP_0.5 (mAP: mean Average Precision) can reach a relatively high level of 0.8 - 0.9, and the landing stability of using these verified networks is good. Although the recognition effect of these general networks is already relatively accurate, the samples in the data of this embodiment are unbalanced, and the proportion of samples with different recognition difficulties is also unbalanced. Therefore, the object - recognition model of this embodiment uses the following loss function loss, that is, the GHM (Gradient Harmonizing Mechanism) loss function is added to the loss function of the original object - detection model to improve the stability of the object - detection model on data with different difficulties.

[0092] loss = loss d +γ·loss GHM

[0093] where loss d is the loss function of the original object - detection model, γ is the weight parameter, and loss GHM is the GHM loss function. γ can be determined by enumeration within a certain range (such as 0 - 10).

[0094] Step S2: Use the trained object - detection model to identify the image to be recognized, and output the moving arms, joint points, and contacts with object - detection frames in the open and closed states. The types of detection frames recognized by the object - detection model are as follows:

[0095] Table 1: Types of detection frames recognized by the object - detection model

[0096]

[0097] Step S3, use the following geometric determination rules to determine the open / closed state one by one until the open / closed state can be determined, and then output the open / closed state result. Among them, the first to third determination rules apply to double-arm horizontal movement disconnecting switches.

[0098] The first determination rule:

[0099]

[0100] This determination rule uses the cosine value of the angle between the vectors pointing from the center point of the joint point detection box to the center point of the corresponding contact detection box to determine the open / closed state. Among them, when the disconnecting switch is in the closed state, the cosine value of this angle tends to -1, and when the disconnecting switch is open, the cosine value of this angle tends to 1. The threshold setting satisfies -1 < T1 < T2 < 1.

[0101] Where e i =(jgd i , jcd i ) represents the vector pointing from the center point of jgd i to the center point of jcd i . The center point of the detection box can be represented by the mean value of all vertices of the detection box; cos(e1, e2) represents the cosine value of the angle between the two vectors.

[0102] dist(jcd1, jcd2)(D1) represents the Euclidean distance between jcd1 and jcd2, and dist(jgd1, jgd2)(D2) represents the Euclidean distance between jgd1 and jgd2. When the disconnecting switch is closed, D1 is much smaller than D2; when the disconnecting switch is open, D1 and D2 are close. The threshold setting satisfies 0 < T7 < T8 < 1.

[0103] Among them, the determination method of T1 and T2 is to set the initial value of T1 to -1 and increase it step by step, and take the maximum value that will not cause false alarms for the closed state as T1; set the initial value of T2 to 1 and decrease it step by step, and take the minimum value that will not cause false alarms for the open state and is greater than T1 as T2.

[0104] Set the initial value of T7 to 0 and increase it step by step, and take the maximum value that will not cause false alarms for the closed state as T7; set the initial value of T8 to 1 and decrease it step by step, and take the minimum value that will not cause false alarms for the open state and is greater than T7 as T8.

[0105] The second determination rule:

[0106]

[0107] Although the type of the action arm detection box is jh and it can determine that the action arm is currently in the closed position, it is still necessary to further accurately determine the opening and closing state of the entire disconnecting switch in combination with the position of the contact point. That is to say, for the action arm detection box in the closed state, obtain the middle area of the action arm detection box. If there is a contact point detection box located in the middle area M of the action arm detection box, as Figure 3 shown, it is determined to be closed; for the action arm detection box in the open state, if there is no contact point detection box located in the middle area M of the action arm detection box, it is determined to be open.

[0108] Among them, the middle area M can be calculated from the vertices of the action arm detection box. Specifically, all the vertices of the action arm detection box form a set of sample points, and the principal component direction is calculated from the sample points through principal component analysis For each vertex, add and subtract the value of the vertex coordinates with to shrink towards the center point. L is the projection distance of the vertex from the center, and γ ∈ (0, 1) is the shrinkage ratio. The shrunk vertex demarcates the middle area M.

[0109] The third determination rule:

[0110]

[0111] That is to say, first determine whether the type of the action arm detection box is in the closed state or the open state. For the action arm detection box in the closed state, obtain its confidence score. If the confidence score exceeds the threshold T3, it is determined to be closed; for the action arm detection box in the open state, if the confidence score exceeds the threshold T4, it is determined to be open.

[0112] Set the initial value of T3 to 1 and gradually decrease it, and take the minimum value that will not cause false alarms for the closed state as T3; set the initial value of T4 to 1 and gradually decrease it, and take the minimum value that will not cause false alarms for the open state as T4.

[0113] In addition, a preset box can be set in advance. The preset box is a polygon box including the left and right joint points and the connection area of the disconnecting switch in the pre-shot image, such as a quadrilateral box, and four points are marked in a clockwise direction starting from the upper left corner. Common cameras include spherical cameras and cylindrical cameras. Generally, spherical cameras can rotate 360° and zoom, support shooting targets in a larger range, and can center and shoot a single target; the shooting angle of cylindrical cameras is fixed, and the target may appear in different areas of the picture, and there may also be a situation where a single picture contains multiple targets of the same type. In actual use, for cylindrical cameras, if there are multiple targets, the specific target to be recognized can be specified by marking the preset box. The preset box has no requirements for the opening and closing state, and both opening and closing are acceptable. If there is no preset box, all the target detection boxes recognized in the image are returned by default.

[0114] For the detection frame of the action arm in the closed state, if the confidence score does not exceed the threshold T3, then determine whether the area ratio of the region enclosed by the preset frame in the jh region (detection frame of the action arm in the closed state) exceeds the threshold T5. If it exceeds the threshold T5, it is determined as closed; for the detection frame of the action arm in the open state, if the confidence score does not exceed the threshold T4, then determine whether the area ratio of the region enclosed by the preset frame in the jf region (detection frame of the action arm in the open state) is less than the threshold T6. If it is less, it is determined as open.

[0115] Among them, the initial value of T5 is set to 1 and gradually decreased, and the minimum value that does not cause false alarms for the closed state is used as T5; the initial value of T6 is set to 0 and gradually increased, and the maximum value that does not cause false alarms for the open state and is less than T5 is used as T6.

[0116] In addition, the accuracy of the detection frame can also be verified by whether the outer bounding boxes of the joint points and contact points are consistent with the detection frame of the action arm. For example, for the recognition of a single action arm, it can be verified through the outer bounding boxes of jcd i and jgd i to check whether the recognition is correct. If the detection frame of the single action arm output by the model is consistent with the outer bounding boxes of jcd i and jgd i , it is considered that the recognition is accurate.

[0117] For example, for the recognition of the double action arm in the closed state, it can be verified through the outer bounding boxes of two jgd i to check whether the recognition is correct. If the detection frame of the double action arm closed output by the model is consistent with the outer bounding boxes of two jgd i , it is considered that the detection frame is accurate. At the same time, the outer bounding boxes of two jcd i can be further used to verify the recognition. If the outer bounding boxes of two jcd i are located in the middle of the detection frame of the action arm, it is considered that the detection frame of the action arm is accurate.

[0118] Among them, the fourth and fifth determination rules are applicable to the opening and closing determination of the horizontal center rotating disconnecting switch.

[0119] Fourth determination rule:

[0120]

[0121] Among them, as shown in Table 1, the types of the detection frames of the action arm include xh and xf. If the detection frame of the action arm in the closed state contains (closed contact) xhcd, but does not contain (fixed contact) xfcdg or (moving contact) xfcdy, as Figure 7 shown, it is determined as closed; if the detection frame of the action arm in the open state contains xfcdg or xfcdy, but does not contain xhcd, as Figure 6 shown, it is determined as open.

[0122] Determine whether the detection frames (xhcd, xfcdg, xfcdy) of the contact type are within the detection frame of the moving arm. The center point of the contact detection frame can be calculated, and it is determined by whether the center point is within the detection frame of the moving arm. If the number of contacts of the same contact type within the detection frame of the moving arm exceeds two, the unnecessary contacts can be excluded by enumerating the necessary conditions that the center points of the two contact detection frames and the center point of the joint detection frame need to satisfy the collinear relationship. This can prevent the model from misidentifying contacts and prevent the contacts of other devices from moving into the detection frame of the moving arm and being misidentified.

[0123] The fifth determination rule:

[0124]

[0125] Set the initial value of T9 to 1 and gradually decrease it to the minimum value that will not cause false alarms for the closed state; set the initial value of T 10 to 1 and gradually decrease it to the minimum value that will not cause false alarms for the open state.

[0126] Among them, the sixth to eighth determination rules are applicable to the opening and closing determination of horizontal and vertical telescopic disconnect switches.

[0127] The sixth determination rule:

[0128] Judge through contacts and joints:

[0129]

[0130] Among them, ygd m , ygd e are two recognized joint detection frames. m represents the middle joint detection frame, and e represents the end-side joint detection frame. After detecting two ygd, two orders of m - e and e - m can be assumed respectively for inference. As long as there is a recognition result in one case, the corresponding result is returned. When judging the collinearity of three points and the logic of being at the midpoint of two detection frames, the center point of the detection frame is used to represent the detection frame for calculation.

[0131] e yf，e =(yf, ygd e ) represents the vector pointing from the center point of yf to the center point of ygd e . e m，e =(ygd m , ygd e ) represents the vector pointing from the center point of ygd m to the center point of ygd e .

[0132] The ratio of the distance from the moving contact to the end-side joint to the distance from the fixed contact to the end-side joint < T11 , it is determined as divided.

[0133] Set the initial value of T 11 to 0 and increase it step by step, with the maximum value being appropriate so as not to cause false alarms for the open state.

[0134] Seventh determination rule:

[0135]

[0136] Among them, for the middle area M, the calculation method is the same as that in the second determination rule.

[0137] Set the initial value of T 12 to 1 and decrease it step by step, with the minimum value being appropriate so as not to cause false alarms for the closed state; set the initial value of T 13 to 1 and decrease it step by step, with the minimum value being appropriate so as not to cause false alarms for the open state.

[0138] Eighth determination rule:

[0139]

[0140] Among them, the preset box is a polygon box including the end-side joint points, fixed contacts, and connection areas in the pre-shot image, such as a quadrilateral box.

[0141] Set the initial value of T 14 to 1 and decrease it step by step, with the minimum value being appropriate so as not to cause false alarms for the closed state; set the initial value of T 15 to 0 and increase it step by step, with the maximum value being appropriate so as not to cause false alarms for the open state, or increase T 15 to the value of T 14 .

[0142] In addition, the form of the target detection box can be a rectangular box or a polygon box, and the above geometric determination rules can also be applied for determination. Specifically, when calculating the center point, the average value of several vertices can be calculated to obtain it. Using a polygon box can better fit the detection target and improve the recognition accuracy.

[0143] In addition, the form of the target detection box can also adopt instance segmentation. The target detection model can adopt, for example, mask-rcnn, and the detected target is represented by one or more masks; the target representation method obtained by instance segmentation also applies the above geometric determination rules. Specifically, when calculating the center point, sampling points can be set equidistantly inside the mask, and the average value of all sampling points can be calculated to obtain it. Similarly, instance segmentation can better fit the detection target and improve the recognition accuracy.

[0144] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention. However, these corresponding changes and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A method for identifying the opening and closing state of an opening and closing component, characterized in that, Including: Collect and label the action arm, joint point, and contact point image data of the opening and closing component in the open state and the closed state, and use the labeled image data as training data to train the object detection model until the loss function output of the object detection model is less than the set threshold, and obtain the trained object detection model. Among them, during labeling, for the double-arm movement opening and closing component, the two action arms in the closed state share one action arm detection frame; for the central rotation opening and closing component, the fixed contact and the moving contact in the closed state are in contact and combined into a closed contact, sharing one contact detection frame; for the telescopic movement opening and closing component, the fixed contact and the moving contact in the closed state are combined into a closed contact, sharing one contact detection frame. Use the trained object detection model to identify the image to be recognized, and output the object detection frames of the action arm, joint point, and contact point in the opening and closing states. Construct geometric determination rules using the geometric relationships of the action arm, joint point, and contact point, and use the geometric determination rules to determine the opening and closing states of the opening and closing component. The geometric determination rules include a second determination rule applicable to the double-arm movement opening and closing component. Second determination rule: Among them, jh represents the action arm detection frame in the closed state, and jf represents the action arm detection frame in the open state. The middle region M forms a set of sample points from all vertices of the action arm detection frame, and the principal component direction is calculated from the sample points through principal component analysis. For each vertex, the vertex coordinates are added and subtracted from to achieve contraction towards the central region. L is the projected distance from the vertex to the center of the action arm detection frame, γ ∈ (0, 1) is the contraction ratio, and all the contracted vertices demarcate the middle region M. Among them, jcd i represents the i-th contact detection frame, where 1 ≤ i ≤ 2.

2. The method for identifying the opening and closing state of the opening and closing component according to claim 1, wherein The loss function of the object detection model is as follows: loss = loss d + γ·loss GHM Among them, loss d is the original loss function of the object detection model, γ is the weight parameter, and loss GHM is the GHM loss function.

3. The method for identifying the opening and closing state of the opening and closing component according to claim 1, wherein The geometric determination rules include a first determination rule applicable to the double-arm movement opening and closing component: jgd i Represents the detection box of the i-th joint point; e i = (jgd i , jcd i ) represents a vector pointing from the center point of jgd i to the center point of jcd i ; dist(jcd1, jcd2) represents the Euclidean distance between jcd1 and jcd2, and dist(jgd1, jgd2) represents the Euclidean distance between jgd1 and jgd2, satisfying 0 < T7 < T8 < 1; T1, T2, T7, and T8 are set thresholds. Among them, -1 < T1 < T2 < 1.

4. The method for identifying the opening and closing state of the opening and closing component according to claim 1, wherein The geometric determination rules include a third determination rule applicable to the double-arm movement opening and closing component: Among them, the preset box is a polygon box including the left and right joint points and the connection area of the opening and closing component in the pre-shot image.

5. The method for identifying the opening and closing state of the opening and closing component according to claim 1, wherein The geometric determination rules include a fourth determination rule applicable to the central rotation opening and closing component: Among them, xhcd represents the closed contact detection frame, xfcdg represents the fixed contact detection frame, and xfcdy represents the moving contact detection frame. Among them, it is judged whether the contact detection frame is within the action arm detection frame by whether the center point of the contact detection frame is within the action arm detection frame.

6. The method for identifying the opening and closing state of the opening and closing component according to claim 5, wherein If the number of contacts of the same contact type in the action arm detection frame exceeds two, then the necessary condition that the center points of the contact detection frames of this contact type should be collinear is enumerated to exclude the redundant contacts.

7. The method for identifying the opening and closing state of the opening and closing component according to claim 6, wherein The geometric determination rules include a fifth determination rule applicable to the central rotation opening and closing member, Fifth determination rule: Among them, xh represents the action arm detection frame in the closed state, and xf represents the action arm detection frame in the open state.

8. The method for identifying the opening and closing state of the opening and closing member according to claim 1, characterized in that The geometric determination rules include a sixth determination rule applicable to the telescopic opening and closing member: Among them, the contact type yh means that the fixed contact and the moving contact are in contact, which is called the closed contact detection frame; ygd m represents the intermediate joint point detection box, ygd e represents the end-side joint point detection box; The contact type yf represents the fixed contact detection frame; e yf,e =(yf,ygd e ) indicates that the direction from the center of yf to ygd e The vector of the center point; e m,e =(ygd m ,ygd e ) indicates that from ygd m The center point points to ygd e The vector of the center point; yfcd represents the moving contact detection frame; dist(yfcd,ygd e ) represents the distance from the center point of the moving contact detection frame to the center point of the end-side joint point detection frame; dist(yf,ygd e ) represents the distance from the center point of the fixed contact detection frame to the center point of the end-side joint point detection frame; T 11 is for setting a threshold value.

9. The method for identifying the opening and closing state of the opening and closing member according to claim 8, characterized in that The geometric determination rules include a seventh determination rule applicable to the telescopic opening and closing member: Among them, ybd represents the action arm detection frame.

10. The method for identifying the opening and closing state of the opening and closing member according to claim 9, wherein The geometric determination rules include an eighth determination rule applicable to the telescopic opening and closing member: Among them, the preset frame is a polygon frame including the end-side joint points, the fixed contacts, and the connection area in the pre-shot image.

11. The method for identifying the opening and closing state of the opening and closing member according to claim 1, characterized in that Image data under various shooting conditions is generated by means of data augmentation, including converting the image into the HSV color format and simulating different lighting changes by adjusting the saturation and / or brightness values.

12. The method for identifying the opening and closing state of the opening and closing member according to claim 1, characterized in that Image data under various shooting conditions is generated by means of data augmentation, including using OpenCV to generate random noises with different densities to simulate rain and snow of different sizes and superimposing them on the original image to simulate rainy and snowy weather.

13. The method for identifying the opening and closing state of the opening and closing member according to claim 1, characterized in that Image data under various shooting conditions is generated by means of data augmentation, including using deep learning-based style transfer to simulate rainy and snowy weather.

14. The method for identifying the opening and closing state of the opening and closing member according to claim 3, characterized in that The object detection model is YOLO or Transformer, and the center point of the object detection frame is determined by the vertex mean value, or the object detection model is mask-rcnn, and the center point of the object detection frame is determined by setting sampling points equidistantly inside the mask and calculating the average value of all sampling points.

15. The method for identifying the opening and closing state of the opening and closing member according to claim 4, characterized in that In the first to third determination rules, if the action arm detection frame in the open state is consistent with the circumscribed frame of the corresponding side contact detection frame and the joint point detection frame, it is considered that the action arm detection frame in the open state is accurately recognized; If the action arm detection frame in the closed state is consistent with the circumscribed frame of the two joint point detection frames, it is considered that the action arm detection frame in the closed state is accurately recognized; If the circumscribed frame of the two contact detection frames is located in the middle of the action arm detection frame, it is considered that the action arm detection frame in the closed state is accurately recognized.

16. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for identifying the opening and closing state of the opening and closing member as described in any one of claims 1 to 15.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for identifying the opening and closing state of the opening and closing member as described in any one of claims 1 to 15.

Images