A method for detecting the opening and closing state of a scissor-type knife switch
Through deep learning algorithms and boundary line models, the opening and closing state of the scissors knife switch is detected, which solves the problem of low detection accuracy in the existing technology, and achieves fast and accurate judgment of the opening and closing state, ensuring the safe and stable operation of the equipment.
Patent Information
- Application Number
- CN201910895607.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-09-21
AI Technical Summary
The prior art is difficult to quickly and accurately detect whether the scissor blade switch is opened and closed in place, resulting in arc generation and equipment damage. The image processing method has low detection accuracy under complex backgrounds, and edge blur affects the detection effect.
Acquire template images to establish a blade gate arm boundary model, and obtain a model marked the blade gate arm and joint node position through deep learning algorithm training. Use training model to detect the target image, obtain the knife gate state, calculate the joint node and angle bisector. Determine the edge line of the upper and lower blade arm and calculate the included angle to determine the opening and closing state.
It improves the accuracy and applicability of the knife switch opening and closing state detection, realizes automatic positioning, ensures that the detection area is clear, and enhances the accuracy and practical value of the detection.
Smart Images

Figure CN110705571B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image recognition, and in particular to a method for detecting the opening and closing state of a scissor-type knife switch. Background Art
[0002] The knife switch is a frequently used electrical appliance in high-voltage switchgear, and plays an isolating role in the circuit. In actual applications, due to the long-term operation of the knife switch, the knife switch may not open or close properly, which will cause an arc between the upper and lower knife switch arms of the knife switch. The arc is a gas discharge phenomenon. The instantaneous spark generated by the current passing through certain insulating media may cause a fire or threaten personal safety. In addition, the arc temperature is extremely high, which can easily burn the insulating material, causing leakage events or damage to the knife switch equipment. Therefore, the closed state of the knife switch needs to be accurately detected.
[0003] In practice, the detection of whether the scissor-type knife switch is in place mainly relies on manual observation; in the prior art, the degree of opening and closing of the knife switch is also determined by calculating the angle between the upper and lower knife switch arms. At present, there are many calculation schemes for calculating the angle between the upper and lower knife switch arms at home and abroad. One is to modify the internal circuit and add sensors or signal lights; the other is a method based on digital image processing to judge the switch state of the knife switch by monitoring the image. In the method based on digital image processing, due to the complex background, it is difficult to correctly match the knife switch, which reduces the detection accuracy. In addition, due to the influence of the shooting environment, the edge of the knife switch arm in the image is blurred, which affects the detection effect and cannot accurately detect the angle between the upper and lower knife switch arms. Summary of the invention
[0004] In order to solve the problem in the prior art that it is difficult to quickly and accurately detect whether a scissor-type knife switch is fully opened or closed, the present invention proposes a method for detecting the opening and closing state of a scissor-type knife switch.
[0005] The technical solution of the present invention is achieved in this way:
[0006] A method for detecting the opening and closing state of a scissor-type knife switch, collecting a template image of the scissor-type knife switch, establishing a boundary line model of a knife switch arm of the knife switch of this type according to the template image and saving it as a model file;
[0007] Using the same equipment to collect target images of the same scissor-type knife switch at the same location;
[0008] Sample images of scissor-type knife gates are obtained through various means, and the sample images are trained using a deep learning algorithm to obtain a training model that marks the positions of knife gate arms and knife gate arm joints;
[0009] The target image is detected using the training model to obtain the rectangular area where one or more knife switches are located in the target image. The knife switches in the target image are located in combination with the model file and the knife switch status is obtained at the same time. The edge of the knife switch is detected according to the positioning result to obtain the edge line set.
[0010] According to the model file and the training model, the gate arm joint points and angle bisectors of the scissor-type gate in the closed or virtually closed state in the target image are calculated; the distance from the gate arm joint points of the target image to each edge line in the edge line set is calculated, and the edge lines corresponding to the distances not within the threshold range are eliminated given the upper distance threshold and the lower distance threshold, so as to obtain the updated edge line set; the updated edge line set is divided into upper and lower parts by the angle bisector, and the upper gate arm edge line set and the lower gate arm edge line set are obtained respectively and symmetrically paired to determine the upper gate arm edge line and the lower gate arm edge line; the angle between the upper gate arm and the lower gate arm is calculated, and the gate state is obtained according to the angle.
[0011] Preferably, the method of "establishing a boundary line model of the knife gate arm of this model of knife gate according to the template image and saving it as a model file" is: setting a program on a computer, using the program to open the template image, stroking the left and right boundaries and the upper and lower boundaries of the upper and lower knife gate arms of the scissors-type knife gate in the template image to form a quadrilateral template, and the upper and lower knife gate arms are within the quadrilateral template, and the four endpoint coordinates of the quadrilateral template are saved as a model file; if the knife gate in the template image is in a closed state, stroke it according to the actual position of the knife gate arm; if the knife gate in the template image is in an open or virtually closed state, estimate the position of the knife gate arm in the closed state according to the upper and lower fixed boundaries of the knife gate, and stroke its position in the closed state.
[0012] Preferably, the method of "obtaining the knife gate state" is: according to the rectangular area of the knife gate positioning result, the upper left end point and the length-width ratio of the rectangular area are obtained, and the length-width ratio threshold and the distance ratio threshold are given; the target image is overlapped with the template image, and the distance from the upper left end point to the upper boundary of the knife gate arm in the template image is calculated. If the ratio of the distance to the total length of the knife gate arm is greater than the given distance ratio threshold, and the length-width ratio is greater than the given length-width ratio threshold, then it is determined that the knife gate is in the open state, otherwise the knife gate is in the closed or virtual closed state;
[0013] The aspect ratio refers to the ratio of the length to the width of the rectangular area in the target image during positioning, and the distance ratio refers to the ratio of the distance from the upper boundary of the upper gate arm of the scissor-type gate to the upper boundary of the upper gate in the closed state to the total length of the upper and lower gate arms in the closed state. The upper boundary of the upper gate arm can be the upper boundary of the rectangular area during positioning.
[0014] Preferably, the method of "calculating the joint points of the scissors-type knife gate arm in the closed or virtually closed state in the target image according to the model file and the training model" is: calculate the center point of the quadrilateral template according to the model file, set the estimation threshold R, correspond the center point to the target image, and establish a joint point rectangular frame containing the joint points of the knife gate arm in the closed state in the target image with the center point as the center, the width of the knife gate arm as the width, and R times the width of the knife gate arm as the length; obtain all the corrected joint point rectangular frames containing the marked joint point positions after positioning in the target image according to the training model, calculate the corrected joint point rectangular frame with the largest overlapping area with the joint point rectangular frame in the corrected joint point rectangular frame, and the center of the corrected joint point rectangular frame is the knife gate arm joint point.
[0015] Preferably, the method of "calculating the bisector of the knife gate arm angle" is:
[0016] Get the distortion coefficient of the camera that captures the target image, and calculate the camera matrix camera_matrix = [focal_length, 0, center.x; 0, focal_length, center.y; 0, 0, 1] according to the target image size, where focal_length is the width of the target image, center.x and center.y are the x-coordinate and y-coordinate of the midpoint of the target image respectively;
[0017] Estimate the corresponding world coordinates according to the four endpoint coordinates in the model file, obtain the rotation matrix and translation matrix of the coordinate system where the camera is located with respect to the world coordinate system according to the four endpoint coordinates, world coordinates, camera matrix and distortion coefficient in the model file, and obtain the transformation matrix according to the rotation matrix and translation matrix;
[0018] The coordinates of the joint points of the knife gate arm in the ideal image are obtained according to the transformation matrix, and the angle bisector of the ideal image passing through the joint points and parallel to the ground is obtained;
[0019] The corresponding coordinates of any point on the angle bisector of the ideal image in the target image are obtained by the transformation matrix calculator, and the line connecting the corresponding coordinates and the joint point of the knife gate arm is the angle bisector of the target image.
[0020] Preferably, the specific method of "using the angle bisector to divide the updated edge line set into upper and lower parts, and respectively obtaining the upper gate arm edge line set and the lower gate arm edge line set" is:
[0021] Calculate the unit direction vector in the target image that is perpendicular to the angle bisector and points upward or downward starting from the joint point.
[0022] Traverse and update the edge lines in the edge line set, record the two endpoints of any edge line as T1 and T2, and record the vector between T1 and the joint point of the knife gate arm as The vector between T2 and the knife gate arm joint point is denoted as calculate and For a unit direction vector pointing upward like Then T1 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector. Then T2 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector; for a unit direction vector pointing downward like Then T1 is on the lower side of the angle bisector, otherwise it is on the upper side of the angle bisector. Then T2 is on the lower side of the angle bisector, otherwise it is on the upper side of the angle bisector;
[0023] If T1 and T2 are both on the upper side of the angle bisector, then the edge line is the upper gate arm edge line; if T1 and T2 are both on the lower side of the angle bisector, then the edge line is the lower gate arm edge line; if T1 and T2 are on both sides of the angle bisector respectively, calculate the length of the upper and lower line segments of the edge line with the angle bisector as the dividing line. This edge line is the edge line on the longer side of the upper and lower side segments, and obtain the upper gate arm edge line set and the lower gate arm edge line set.
[0024] Preferably, the method of "symmetrically pairing the upper gate arm edge line set and the lower gate arm edge line set to determine the final upper gate arm edge line and lower gate arm edge line" is:
[0025] Given an angle threshold, the angle of any edge line in the upper knife gate arm edge line set relative to the horizontal position is recorded as M h , h represents different upper knife gate arm edge lines, any two upper knife gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the upper knife gate arm edge lines in the same group are sorted from long to short; the angle of any edge line in the lower knife gate arm edge line set relative to the horizontal position is recorded as N m , m represents different lower gate arm edge lines, any two lower gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the lower gate arm edge lines in the same group are sorted from long to short;
[0026] Preferably, the method of "traversing all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line set and performing one-to-one pairing, and calculating the corrected gate arm edge lines and pairing scores of any two groups" is as follows: calculate the upper midpoint between the two top endpoints and the lower midpoint between the two bottom endpoints according to the model file, traverse all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line set, calculate the average angle of each group, the average angle of each group in the upper gate arm edge line set is combined with the upper midpoint to obtain the corrected upper gate arm edge line of each group in the upper gate arm edge line set, and the intersection point of the corrected upper gate arm edge line and the angle bisector is recorded as point A; the average angle of each group in the lower gate arm edge line set is combined with the lower midpoint to obtain the corrected lower gate arm edge line of each group of the lower gate arm edge lines, and the intersection point of the corrected lower gate arm edge line and the angle bisector is recorded as point B, and the distance d between point A and point B is calculated AB ; The distance from the upper fixed boundary to the lower fixed boundary of the knife gate arm is d0, d AB / d0>0.5, then point A is in the same group as point B;
[0027] The sum of the squares of the lengths of any edge lines in the same group of the gate arm edge line set; β is an adjustable sensitivity parameter; the corrected upper gate arm edge line at point A and the corrected lower gate arm edge line at point B corresponding to the highest pairing score S are the final upper gate arm edge line and lower gate arm edge line.
[0028] Preferably, the method of "calculating the angle between the upper gate arm and the lower gate arm" is: taking the final edge line of the upper gate arm and the edge line of the lower gate arm as the reference, if the angle of the final edge line of the upper gate arm relative to the straight line where the angle bisector of the gate arm in the closed state is α1, and the angle of the final edge line of the lower gate arm relative to the straight line where the angle bisector of the gate arm in the closed state is α2, the final angle between the upper and lower gate arms is θ=|α2-α1|.
[0029] The beneficial effects of the present invention are as follows: the method for detecting the opening and closing state of a scissor-type knife gate of the present invention obtains a template image, a model file, a target image and a training model, and uses the training model to detect the target image to realize the positioning of the knife gate in the target image and obtain the knife gate state, so that when performing subsequent processing, the targeted area is clearer, and the accuracy of determining the edge line of the upper knife gate arm and the edge line of the lower knife gate arm is improved. The sample images used by the training model bring together different states of the scissor-type knife gates under different environments, which helps to improve the applicability and detection accuracy of the training model and help realize automatic positioning of the knife gate; the knife gate arm joint points and angle bisectors of the scissor-type knife gate in the closed and virtual closed states in the target image are calculated according to the model file and the training model, and the knife gate arm joint points and angle bisectors are used to obtain the upper knife gate arm edge line set and the lower knife gate arm edge line set to improve accuracy; then the upper knife gate arm edge line and the lower knife gate arm edge line are determined by pairing; finally, the angle between the upper knife gate arm and the lower knife gate arm is calculated to improve the accuracy of the angle calculation of the upper knife gate arm and the lower knife gate arm. The method of detecting whether the knife gate arms are fully opened or closed by calculating the angles of the upper and lower knife gate arms has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0031] Figure 1 is the template image of the present invention;
[0032] Figure 2 It is a structural diagram for calculating the joint points of the knife gate arm;
[0033] Figure 3 A schematic diagram of the structure for dividing the updated edge line set into upper and lower parts by using the angle bisector.
[0034] In the figure:
[0035] 1. Upper knife gate arm; 2. Lower knife gate arm; 3. Knife gate arm joint point; 4. Center point; 5. Joint point rectangular frame; 6. Correction joint point rectangular frame; 7. Angle bisector. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] Example 1: Figure 1 As shown, a method for detecting the opening and closing state of a scissor-type knife switch is provided. The method comprises the following steps: collecting a template image of a scissor-type knife switch, establishing a boundary line model of a knife switch arm of the knife switch model according to the template image and saving the model file; collecting a target image of the same scissor-type knife switch at the same position using the same device; obtaining sample images of the scissor-type knife switch through multiple means, training the sample images using a deep learning algorithm to obtain a training model for marking the positions of the knife switch arms and the joint points of the knife switch arms; detecting the target image using the training model to obtain a rectangular area where one or more knife switches are located in the target image, locating the knife switch in the target image in combination with the model file, and obtaining the estimated state of the knife switch at the same time; if the estimated state of the knife switch is open, directly outputting the state of the knife switch as open. Open, otherwise perform edge detection on the knife gate according to the positioning result to obtain the edge line set; calculate the knife gate arm joint point 3 and the angle bisector of the scissor-type knife gate in the target image according to the model file and the training model; calculate the distance from the knife gate arm joint point 3 of the target image to each edge line in the edge line set, give the upper distance threshold and the lower distance threshold, remove the edge lines corresponding to the distance not within the threshold range, and obtain the updated edge line set; use the angle bisector to distinguish the updated edge line set from top to bottom, obtain the upper knife gate arm edge line set and the lower knife gate arm edge line set respectively, and perform symmetrical pairing to determine the final upper knife gate arm edge line and lower knife gate arm edge line; calculate the angle between the upper knife gate arm 1 and the lower knife gate arm 2, and obtain the knife gate state according to the angle. The upper distance threshold and the lower distance threshold are given according to the actual situation.
[0038] The method of "establishing the boundary line model of the knife gate arm of this model of knife gate according to the template image and saving it as a model file" in Example 1 is: set up a program on the computer, use the program to open the template image, and stroke the left and right boundaries and the upper and lower boundaries of the upper and lower knife gate arms of the scissors-type knife gate in the template image to form a quadrilateral template. The upper and lower knife gate arms are within the quadrilateral template, and the four endpoint coordinates of the quadrilateral template are saved as a model file. If the knife gate in the template image is in a closed state, stroke it according to the actual position of the knife gate arm. If the knife gate in the template image is in an open or virtually closed state, estimate the position of the knife gate arm in the closed state according to the upper and lower fixed boundaries of the knife gate, and stroke its position in the closed state. For a scissors-type knife gate, no matter what state the collected template image is in, it only needs to stroke it according to the position of its closed state, that is, for any image of the same knife gate, the model file is the same, which can simplify user operations. The state of the knife gate in the template image is manually identified.
[0039] The method of "using the training model to detect the target image, obtaining the rectangular area where one or more knife switches are located in the target image, and locating the knife switches in the target image in combination with the model file" is:
[0040] The target image is detected using the training model to obtain all the rectangular frames of the knife switch areas in the target image; the closed quadrilateral frame area, upper midpoint and lower midpoint of the knife switch in the closed state are obtained according to the model file, the upper midpoint is the midpoint of the upper boundary of the closed quadrilateral frame, and the lower midpoint is the midpoint of the lower boundary of the closed quadrilateral frame, and the overlapping area of each knife switch area rectangular frame with the closed quadrilateral frame area is calculated; given an overlapping area threshold, the area containing the upper midpoint and the lower midpoint with the largest overlapping area and greater than the overlapping area threshold is obtained. If the area exists, then the area is the final rectangular area where the knife switch is located, otherwise the area containing the lower midpoint with the largest overlapping area and greater than the overlapping area threshold is obtained, and the area is the final rectangular area where the knife switch is located, thereby realizing the positioning of the knife switch.
[0041] The method of "obtaining the estimated state of the knife gate" is as follows: according to the rectangular area of the knife gate positioning result, the upper left end point and the length-width ratio of the rectangular area are obtained, and the length-width ratio threshold and the distance ratio threshold are given; the target image is overlapped with the template image, and the distance from the upper left end point to the upper boundary of the knife gate arm in the template image is calculated. If the ratio of the distance to the total length of the knife gate arm is greater than the given distance ratio threshold, and the length-width ratio is greater than the given length-width ratio threshold, then the knife gate is determined to be in an open state, otherwise the knife gate is in a closed or virtual state; the length-width ratio refers to the ratio of the length and width of the rectangular area in the target image, and the distance ratio refers to the ratio of the distance from the upper boundary of the upper knife gate arm of the scissor-type knife gate to the upper boundary of the upper knife gate in the closed state to the total distance of the upper and lower knife gate arms in the closed state. The upper boundary of the upper knife gate arm can be the upper boundary of the rectangular area during positioning. The length-width ratio threshold and the distance ratio threshold are not unique, and they are defined according to actual conditions and user needs. The length of the scissor-type knife gate is shortening when it is opened.
[0042] like Figure 2 As shown, the above-mentioned method of "calculating the joint points of the knife gate arm of the scissors-type knife gate in the target image according to the model file and the training model" is: calculate the center point of the rectangular frame template according to the model file, set the estimation threshold R, correspond the center point 4 to the target image, and establish a joint point rectangular frame 5 containing the joint points of the knife gate arm in a closed state in the target image with the center point 4 as the center, the width of the knife gate arm as the width, and R times the width of the knife gate arm as the length; obtain all the corrected joint point rectangular frames 6 containing the marked joint point positions after positioning in the target image according to the training model, calculate the corrected joint point rectangular frame with the largest overlapping area with the joint point rectangular frame in the corrected joint point rectangular frame, and the center of the corrected joint point rectangular frame is the knife gate arm joint point.
[0043] The method of "calculating the bisector of the knife gate arm angle" is: obtain the distortion coefficient of the camera that collects the target image, and calculate the camera matrix camera_matrix = [focal_length, 0, center.x; 0, focal_length, center.y; 0, 0, 1] according to the target image size, where focal_length is the target image width, center.x and center.y are the x-coordinate and y-coordinate of the midpoint of the target image respectively; estimate the corresponding world coordinates according to the four endpoint coordinates in the model file, and use the reference "Pose Estimation based on Four Coplanar Point Correspondences》, according to the four endpoint coordinates, world coordinates, camera matrix and distortion coefficient in the model file, the rotation matrix and translation matrix of the coordinate system where the camera is located with respect to the world coordinate system can be obtained, and the transformation matrix is obtained according to the rotation matrix and translation matrix. Suppose the transformation matrix is R_matrix, and the joint point of the knife gate arm is artis_p. According to the transformation matrix, the joint point coordinates of the knife gate arm in the ideal image are obtained artis_q=(artis_q.x, artis_q.y), and the angle bisector y=(artis_q.y / artis_q.x*x) passing through the joint point of the ideal image and parallel to the ground is obtained; any point angle_p on the angle bisector of the ideal image is obtained through the transformation matrix calculator to obtain the corresponding coordinates in the target image, and the line connecting the corresponding coordinate angle_q and the joint point artis_p of the knife gate arm is the angle bisector of the target image.
[0044] The ideal image in theory refers to the image of the knife gate taken by the camera perpendicular to the knife gate when the center point of the camera and the knife gate arm joint point are in the same straight line. In actual shooting, the knife gate or the camera generally has a certain degree of positional offset. There will be some errors between the knife gate arm joint points of the ideal image obtained by using the knife gate arm joint points and the transformation matrix and the knife gate arm joint points of the theoretical ideal image. This error does not affect the calculation of the angle bisector.
[0045] The method of "performing edge detection on the target image and obtaining an edge line set" is: performing edge detection on the target image. The edge detection method used here is LSD edge detection. By inputting the target image of the scissor-type knife gate to be detected, all the edge lines of the target image can be obtained. At the same time, any method that can help detect the edge of the knife gate arm can be used, such as EDLines, Hough transform straight line detection, etc., to obtain the edge line set.
[0046] The specific method of "using the angle bisector to divide the updated edge line set into upper and lower parts to obtain the upper gate arm edge line set and the lower gate arm edge line set respectively" is as follows:
[0047] Calculate the unit direction vector in the target image that is perpendicular to the angle bisector and points upward or downward starting from the joint point. Traverse and update the edge lines in the edge line set, record the two endpoints of any edge line as T1 and T2, and record the vector between T1 and the joint point of the knife gate arm as The vector between T2 and the knife gate arm joint point is denoted as calculate and For a unit direction vector pointing upward like Then T1 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector. Then T2 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector; for a unit direction vector pointing downward like Then T1 is on the lower side of the angle bisector, otherwise it is on the upper side of the angle bisector. Then T2 is on the lower side of the angle bisector, otherwise it is on the upper side of the angle bisector;
[0048] If T1 and T2 are both on the upper side of the angle bisector, then the edge line is the upper gate arm edge line; if T1 and T2 are both on the lower side of the angle bisector, then the edge line is the lower gate arm edge line; if T1 and T2 are on both sides of the angle bisector respectively, calculate the length of the upper and lower line segments of the edge line with the angle bisector as the dividing line. This edge line is the edge line on the longer side of the upper and lower side segments, and obtain the upper gate arm edge line set and the lower gate arm edge line set.
[0049] like Figure 3 As shown, n is a unit vector, T1 and T2 are the two endpoints of the edge line, P1 and P2 are the vectors connecting the joint point of the knife gate arm and T1 and T2, and the angle between P1 and P2 and n is an acute angle, so Therefore, T1 and T2 are both on the upper side of the angle bisector, and the edge line formed by T1 and T2 is the edge line of the upper knife gate arm.
[0050] Unit direction vector The specific calculation method is: set the angle bisector of the knife gate arm on the target image to y = kx + b, and the coordinates of the knife gate arm joint point to (C x ,C y ), set the equation of the vertical line perpendicular to the angle bisector and passing through the joint point of the knife gate arm to y = cx + d, c = -1 / k, C y =-1 / k*C x +d,d=C y +1 / k*C x ; Calculate the unit direction vector starting from the joint point of the knife gate arm and pointing upward or downward according to the straight line equation of the vertical line Assume the end point coordinates of the unit direction vector are (n x ,n y ),
[0051] Then (n y -C y ) / (n x -C x )=-1 / k,(n y -C y ) 2 +(n x -C x ) 2 =1, find n x and n y , the unit direction vector
[0052] The method of “symmetrically pairing the upper knife gate arm edge line set and the lower knife gate arm edge line set to determine the upper knife gate arm edge line and the lower knife gate arm edge line” is as follows: given an angle threshold, the angle of any edge line in the upper knife gate arm edge line set relative to the horizontal position is recorded as M h , h represents different upper knife gate arm edge lines, any two upper knife gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the upper knife gate arm edge lines in the same group are sorted from long to short; the angle of any edge line in the lower knife gate arm edge line set relative to the horizontal position is recorded as N m , m represents different lower gate arm edge lines, any two lower gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the lower gate arm edge lines in the same group are sorted from long to short;
[0053] Traverse all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line set and pair them one by one, calculate any two groups of corrected gate arm edge lines and pairing scores, and the corrected upper gate arm edge line and the corrected lower gate arm edge line corresponding to the combination with the highest score are the final upper gate arm edge line and lower gate arm edge line.
[0054] The method of "traversing all the same groups of the upper knife gate arm edge line set and all the same groups of the lower knife gate arm edge line set and pairing them one by one, and calculating any two groups of corrected knife gate arm edge lines and pairing scores" is: Figure 2As shown, the upper midpoint between the two top endpoints and the lower midpoint between the two bottom endpoints are calculated according to the model file, and the average angle of each group is calculated. The average angle of each group in the upper gate arm edge line set is combined with the upper midpoint to obtain the corrected upper gate arm edge line of each group in the upper gate arm edge line set, and the intersection of the corrected upper gate arm edge line and the angle bisector is recorded as point A; the average angle of each group in the lower gate arm edge line set is combined with the lower midpoint to obtain the corrected lower gate arm edge line of each group in the lower gate arm edge line set, and the intersection of the corrected lower gate arm edge line and the angle bisector is recorded as point B, and the distance d between point A and point B is calculated. AB ; The distance from the upper fixed boundary to the lower fixed boundary of the knife gate arm is d0, d AB / d0>0.5, the same group as point A and the same group as point B are eliminated;
[0055] The calculation formula for any two pairs of matching scores is set as The coefficient S T S represents the sum of the squares of the lengths of any edge line in the same group of the upper knife gate arm edge line set; B Represents the sum of the squares of the lengths of any edge lines in the same group of the lower gate arm edge line set; β is an adjustable sensitivity parameter; the corrected upper gate arm edge line at point A and the corrected lower gate arm edge line at point B corresponding to the highest pairing score S are the final upper gate arm edge line and lower gate arm edge line.
[0056] Assume that the number of edge lines in any group of the upper knife gate arm edge line set is K, and each edge line can be expressed as L i , calculate the length of the edge line according to its starting and ending coordinates as LLength i , where i=0,1,...K, then S T =LLength0 2 +LLength1 2 ...+LLength K 2 ; Assume that the number of edge lines in any group of the lower knife gate arm edge line set is Q, and each edge line can be expressed as R j , calculate the length of the edge line according to its starting and ending coordinates as RLength j , where j = 0, 1, ..., Q, then S B =RLength0 2 +RLength1 2 ...+RLength Q 2β is an adjustable sensitivity parameter. In actual use, different scores can be obtained by taking different β values. The pairing results are determined according to different scores. The β value corresponding to the best pairing result is selected as the final β value. Different users can also determine it according to actual conditions. The corrected upper gate arm edge line at point A corresponding to the highest pairing score S and the corrected lower gate arm edge line at point B are the final upper gate arm edge line and lower gate arm edge line.
[0057] The method of "calculating the angle between the upper gate arm and the lower gate arm" is: taking the final edge line of the upper gate arm and the edge line of the lower gate arm as the reference, if the angle of the final edge line of the upper gate arm relative to the straight line where the angle bisector of the gate arm in the closed state is α1, and the angle of the final edge line of the lower gate arm relative to the straight line where the angle bisector of the gate arm in the closed state is α2, the final angle between the upper and lower gate arms is θ=|α2-α1|.
[0058] The method of "obtaining the knife gate state based on the angle" is: if the estimated state of the knife gate is open, the current knife gate state is directly output as open, and there is no need to accurately calculate the angle between the knife gate arms; if the estimated state of the knife gate is closed or virtually closed, the knife gate state is determined according to the angle between the upper and lower knife gate arms; two angle thresholds To and Tc for the "open" and "closed" states of the scissor-type knife gate are given according to user needs; if the angle between the upper knife gate arm and the lower knife gate arm is greater than To, it is determined that the knife gate in the current frame image is in the open state; if the actual angle between the upper knife gate arm and the lower knife gate arm is less than Tc, it is determined that the knife gate in the current frame image is in the closed state; if the actual angle between the upper knife gate arm and the lower knife gate arm is between To and Tc, it is determined that the knife gate in the current frame image is in the virtually closed state.
[0059] When establishing the model file and obtaining the estimated state of the knife gate according to the training model, the closed state of the knife gate refers to the contact between the upper knife gate arm and the upper fixed boundary and the upper and lower knife gate arms are in the same straight line, the virtual closed state refers to the contact between the upper knife gate arm and the upper fixed boundary but the upper and lower knife gate arms are not in the same straight line, and the open state refers to the non-contact between the upper knife gate arm and the upper fixed boundary. The knife gate state here is general, which is different from the knife gate state obtained according to the angle between the knife gate arms. The knife gate state obtained according to the angle between the knife gate arms is the final state of the knife gate in the target image.
[0060] The scissor-type knife switch refers to a vertical telescopic isolating switch, which uses the middle joint point as a reference to divide the knife switch into upper and lower knife switch arms. During the opening and closing process of the knife switch, the knife switch arms use the joint point to perform telescopic opening and closing. When fully closed, the two knife switch arms are in the same straight line, and when fully opened, the two knife switch arms are folded together. The knife switch in the present invention refers to a scissor-type knife switch.
[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for detecting the opening and closing state of a scissor-type knife switch, characterized in that: A template image of a scissor-type knife switch is collected, and a boundary line model of a knife switch arm of the knife switch is established according to the template image and saved as a model file; a target image of the same scissor-type knife switch is collected at the same position using the same device; Sample images of scissor-type knife switches are obtained through various means, and the sample images are trained using a deep learning algorithm to obtain a training model that marks the positions of knife switch arms and knife switch arm joints; the training model is used to detect the target image, and the rectangular area where one or more knife switches are located in the target image is obtained, and the knife switch in the target image is located in combination with the model file, and the estimated state of the knife switch is obtained at the same time; If the estimated state of the knife gate is open, the knife gate state is directly output as open, otherwise the edge detection of the knife gate is performed according to the positioning result to obtain the edge line set; the knife gate arm joint points and angle bisectors of the scissor-type knife gate in the target image are calculated according to the model file and the training model; Calculate the distance from the knife gate arm joint point of the target image to each edge line in the edge line set, give an upper distance threshold and a lower distance threshold, remove the edge lines corresponding to the distances that are not within the threshold range, and obtain an updated edge line set; The updated edge line set is divided into upper and lower parts by using the angle bisector, and the upper knife gate arm edge line set and the lower knife gate arm edge line set are obtained respectively and symmetrically matched, so as to determine the final upper knife gate arm edge line and the lower knife gate arm edge line; Calculate the angle between the upper knife gate arm and the lower knife gate arm, and obtain the knife gate state according to the angle; The specific method of "using the angle bisector to divide the updated edge line set into upper and lower parts, and respectively obtaining the upper knife gate arm edge line set and the lower knife gate arm edge line set" is: in the target image, calculate the unit direction vector perpendicular to the angle bisector and pointing upward or downward with the joint point as the starting point. ; Traverse and update the edge lines in the edge line set, record the two endpoints of any edge line as T1 and T2, and record the vector between T1 and the joint point of the knife gate arm as , the vector between T2 and the knife gate arm joint point is recorded as ,calculate and ; For a unit direction vector pointing upward ,like , then T1 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector. , then T2 is on the upper side of the angle bisector, otherwise it is on the lower side of the angle bisector; for the unit direction vector pointing downward ,like , then T1 is on the lower side of the angle bisector, otherwise it is on the upper side of the angle bisector. , then T2 is on the lower side of the angle bisector, otherwise on the upper side of the angle bisector; if T1 and T2 are both on the upper side of the angle bisector, then the edge line is the upper knife gate arm edge line, if T1 and T2 are both on the lower side of the angle bisector, then the edge line is the lower knife gate arm edge line, if T1 and T2 are on both sides of the angle bisector respectively, calculate the length of the upper and lower line segments of the edge line with the angle bisector as the dividing line, the edge line is the edge line on the longer side of the upper and lower side line segments, obtain the upper knife gate arm edge line set and the lower knife gate arm edge line set; The method of "symmetrically pairing the upper knife gate arm edge line set and the lower knife gate arm edge line set to determine the final upper knife gate arm edge line and lower knife gate arm edge line" is: given an angle threshold, the angle of any edge line in the upper knife gate arm edge line set relative to the horizontal position is recorded as , Representing different upper knife gate arm edge lines, any two upper knife gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the upper knife gate arm edge lines in the same group are sorted from long to short; the angle of any edge line in the lower knife gate arm edge line set relative to the horizontal position is recorded as , Representing different lower gate arm edge lines, any two lower gate arm edge lines whose angle difference is within the angle threshold are divided into the same group, and the lower gate arm edge lines in the same group are sorted from long to short; traverse all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line set and pair them one by one, calculate the corrected gate arm edge lines and pairing scores of any two groups, and the corrected upper gate arm edge line and the corrected lower gate arm edge line corresponding to the combination with the highest score are the final upper gate arm edge line and the lower gate arm edge line; The method of "traversing all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line and pairing them one by one, and calculating the corrected gate arm edge lines and pairing scores of any two groups" is as follows: calculate the upper midpoint between the two top endpoints and the lower midpoint between the two bottom endpoints according to the model file, traverse all the same groups of the upper gate arm edge line set and all the same groups of the lower gate arm edge line set, calculate the average angle of each group, and combine the average angle of each group in the upper gate arm edge line set with the upper midpoint to obtain the corrected upper gate arm edge line of each group of upper gate arm edge lines, and record the intersection of the corrected upper gate arm edge line and the angle bisector as point A; combine the average angle of each group in the lower gate arm edge line set with the lower midpoint to obtain the corrected lower gate arm edge line of each group of lower gate arm edge lines, and record the intersection of the corrected lower gate arm edge line and the angle bisector as point B, and calculate the distance between point A and point B ; The distance from the upper fixed boundary to the lower fixed boundary of the knife gate arm is , Then the group where point A is located and the group where point B is located are eliminated; The calculation formula for any two pairs of matching scores is set as , where the coefficient It represents the sum of the squares of the lengths of any edge line in the same group of the upper knife gate arm edge line set; It represents the sum of the squares of the lengths of any edge line in the same group of the edge line set of the lower knife gate arm; is an adjustable sensitivity parameter; the corrected upper knife gate arm edge line where point A is located and the corrected lower knife gate arm edge line where point B is located corresponding to the highest pairing score S are the final upper knife gate arm edge line and lower knife gate arm edge line; The method of "calculating the angle between the upper and lower knife gate arms" is: taking the final edge line of the upper knife gate arm and the edge line of the lower knife gate arm as the reference, if the angle of the final edge line of the upper knife gate arm relative to the straight line where the angle bisector of the knife gate arm in the closed state is The angle between the edge line of the lower knife gate arm and the straight line of the angle bisector of the knife gate arm in the closed state is , the final angle between the upper and lower knife gate arms is .
2. The method for detecting the opening and closing state of a scissor-type knife switch according to claim 1, characterized in that: The method of "establishing a boundary line model of the knife gate arm of this model of knife gate according to the template image and saving it as a model file" is as follows: set up a program on the computer, use the program to open the template image, and stroke the left and right boundaries and the upper and lower boundaries of the upper and lower knife gate arms of the scissors-type knife gate in the template image to form a quadrilateral template. The upper and lower knife gate arms are within the quadrilateral template, and the coordinates of the four endpoints of the quadrilateral template are saved as a model file; if the knife gate in the template image is in a closed state, stroke it according to the actual position of the knife gate arm. If the knife gate in the template image is in an open or virtually closed state, estimate the position of the knife gate arm in the closed state according to the upper and lower fixed boundaries of the knife gate, and stroke its position in the closed state.
3. The method for detecting the opening and closing state of a scissor-type knife switch according to claim 2, characterized in that: The method of "obtaining the estimated state of the knife gate" is: according to the rectangular area of the knife gate positioning result, obtain the upper left end point and the length-width ratio of the rectangular area, give the length-width ratio threshold and the distance ratio threshold; overlap the target image with the template image, calculate the distance from the upper left end point to the upper boundary of the knife gate arm in the template image, if the ratio of this distance to the total length of the knife gate arm is greater than the given distance ratio threshold, and the length-width ratio is greater than the given length-width ratio threshold, then the knife gate is judged to be in the open state, otherwise the knife gate is in the closed or virtual closed state.
4. The method for detecting the opening and closing state of a scissor-type knife switch according to claim 2, characterized in that: The method of "calculating the joint points of the scissor-type knife gate arm in the closed or virtually closed state in the target image according to the model file and the training model" is: calculate the center point of the quadrilateral template according to the model file, set the estimation threshold R, correspond the center point to the target image, and establish a joint point rectangular frame containing the joint points of the knife gate arm in the closed state in the target image with the center point as the center, the width of the knife gate arm as the width, and R times the width of the knife gate arm as the length; obtain all the corrected joint point rectangular frames containing the marked joint point positions after positioning in the target image according to the training model, calculate the corrected joint point rectangular frame with the largest overlapping area with the joint point rectangular frame in the corrected joint point rectangular frame, and the center of the corrected joint point rectangular frame is the knife gate arm joint point.
5. The method for detecting the opening and closing state of a scissor-type knife switch according to claim 1, characterized in that: The method of "calculating the bisector of the knife gate arm angle" is: Get the distortion coefficient of the camera that captures the target image, and calculate the camera matrix camera_matrix=[focal_length, 0, center.x; 0,focal_length,center.y; 0, 0, 1] according to the target image size, where focal_length is the width of the target image, center.x and center.y are the x-coordinate and y-coordinate of the midpoint of the target image respectively; Estimate the corresponding world coordinates according to the four endpoint coordinates in the model file, obtain the rotation matrix and translation matrix of the coordinate system where the camera is located with respect to the world coordinate system according to the four endpoint coordinates, world coordinates, camera matrix and distortion coefficient in the model file, and obtain the transformation matrix according to the rotation matrix and translation matrix; The coordinates of the joint points of the knife gate arm in the ideal image are obtained according to the transformation matrix, and the angle bisector of the ideal image passing through the joint points and parallel to the ground is obtained; The corresponding coordinates of any point on the angle bisector of the ideal image in the target image are obtained by the transformation matrix calculator, and the line connecting the corresponding coordinates and the joint point of the knife gate arm is the angle bisector of the target image.
