A method for detecting the position of a rotary switch based on multi-point fitting

Abstract

The present invention discloses a method for detecting the posture of a knob switch based on multi-point fitting, which belongs to the field of robot manipulator control technology. The method comprises: constructing a detection model of the knob switch; collecting the target switch image, obtaining the pixel coordinate range of the key area and verifying the reliable points; randomly extracting multiple groups of reliable points and using multiple least squares fitting to obtain the direction line of the knob operating handle, and obtaining the current posture information of the knob switch. The present invention uses a depth camera to verify the depth information of the sampling point to ensure that the sampling point is actually located on the knob surface to avoid misdetection of background or interference; through multi-point sampling and multiple fitting, the influence of single-point error can be eliminated, the knob posture detection result can be optimized, the accuracy and stability can be ensured, and the knob switch operating handle at different rotation angles can be effectively identified to meet the accuracy requirements of the knob switch of the electrical cabinet operated by the automated operating equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot manipulator control, and in particular relates to a knob switch posture detection method based on multi-point fitting. Background Art

[0002] In recent years, with the rapid development of the national power grid and high-speed railway systems, the automation level of my country's power and railway systems has significantly increased. As a key component of industrial automation control, electrical cabinet knob switches are widely used to adjust equipment operating modes, start and stop states, and other functions. Traditional electrical cabinet knob switches rely on manual operation, which not only leads to low operational efficiency but also increases the risk of human error. This makes it difficult to ensure the stability and accuracy of the knob switches, especially in high-precision applications.

[0003] To achieve unmanned substation management, many substations have begun introducing automated equipment such as mobile robotic arms to replace manual operations, such as operating rotary switches on electrical cabinets. However, in actual applications, existing automated equipment generally faces the following problems when operating rotary switches: First, due to the inability of the automated equipment to accurately identify the position and posture of the rotary switches, the operation task is often incomplete and may even lead to safety hazards; second, the use of adhesive markers to assist in identification increases maintenance difficulty and affects the neatness and standardization of the electrical cabinet panels; finally, the reliability of some existing methods is often not well guaranteed, limiting the application of existing technologies in complex environments and situations requiring high precision. Therefore, how to enable the robotic arm to accurately detect the position and posture of the rotary switches to complete high-precision operations is a major challenge in the current application of automated equipment.

[0004] In order to solve these problems, the present invention proposes a knob switch posture detection method based on multi-point fitting. It combines deep learning technology and depth camera equipment, and uses precise image acquisition and algorithm processing to ensure that the robotic arm can accurately identify the current posture of the knob, thereby achieving precise operation of the knob switch.

[0005] Currently, there are some related technologies for identifying the gear position of knob switches. In "Research on the Design and Motion Planning of Intelligent Emergency Operation Robots for Substations," Chen Yang proposed a method that uses affixed indicator strips and uses color feature screening, combined with calculating the minimum circumscribed rectangle of the indicator strips, to obtain the position and posture of the knob switch handle. This method can detect the position and posture of the knob switch under certain conditions, but it relies on the attached indicator strips as auxiliary markers. Once the indicator strips fall off or are damaged, the recognition accuracy will be seriously affected. Secondly, this method still cannot completely get rid of the influence of the indicator strip position deviation on the recognition results, making it difficult to achieve accurate positioning of the knob operation point, which may affect the accuracy and stability of actual operation.

[0006] In their paper "Power Station Knob Switch State Recognition Based on the YOLO-tiny-RFB Model," Shi Mengan, Lu Zhenyu, and others proposed a switch position recognition method based on a deep learning target detection algorithm. This method classifies switches in different positions into different categories for target detection. While this method can accurately identify some positions of rotary switches, it cannot determine the current angle of the knob's operating handle and still cannot accurately locate the knob's operating point.

[0007] In their Chinese invention patent document, "A Method for Detecting and Operating a Knob Switch in an Electrical Cabinet" (CN118254180B), Zhu Min, Chen Jie, and others proposed a method for detecting the position of a rotary switch based on key point detection. While this method can detect and operate rotary switches under certain conditions, it still suffers from several drawbacks: first, it cannot determine the reliability of key points; second, it relies solely on single key point detection and fitting, potentially leading to significant noise or outliers influencing the overall results; and finally, it performs only a single least squares fit on the key point fitted line, rather than averaging multiple fits to further reduce errors. This can lead to insufficient detection and operation accuracy in complex scenarios.

[0008] The limitations of these existing methods indicate that further optimization is still needed in practical applications to improve their robustness and adaptability, so as to meet the high-precision requirements of automation equipment in the operation of electrical cabinet knob switches.

[0009] In robotic automation, machine vision technology, as one of the core technologies, determines whether the robot can accurately identify and operate the knob switches in the electrical cabinet. Accurately identifying the position information of the knob switch is a prerequisite for ensuring that the robot can complete the task accurately. However, there are still some technical problems that need to be solved in the current robot vision technology for identifying the knob switches in the electrical cabinet. These problems affect the accuracy and stability of knob recognition and operation. Specifically, the research on the vision technology for robot recognition of the knob switches in the electrical cabinet still has the following technical problems:

[0010] 1. Existing recognition methods based on traditional vision technology often require attaching auxiliary markers, such as calibration plates, to the knobs to help the robot identify the knob's position information. This method not only increases maintenance costs but can also reduce recognition accuracy due to the falling or damage of the markers, affecting the stability of the knob's operation.

[0011] 2. Current deep learning-based detection methods often ignore the depth information of the sampling points when identifying knob postures. This may result in the sampling points not being located on the knob surface, thus affecting the final posture recognition results and the accuracy of the knob operation. In addition, most methods only detect a single key point and do not perform extended detection on the key areas of the knob handle. This makes them susceptible to interference from noise or abnormal points, affecting detection stability.

[0012] 3. The existing knob detection methods usually only perform a fitting operation on a single key point, lack multiple fitting optimizations or average value processing of multi-point detections in the area, and fail to consider the impact of environmental changes or error accumulation, resulting in insufficient fitting accuracy and stability of detection results, which may lead to error accumulation and affect the accuracy of subsequent operations. Summary of the Invention

[0013] The technical problem to be solved by the present invention is the above-mentioned defect. Specifically, in response to the problem in the prior art that automated operating equipment recognizes the posture information of the knob switch of the electrical cabinet, a knob switch posture detection method based on depth information, multi-point detection and fitting optimization is proposed. Specifically, the present invention solves the problem that the traditional method is difficult to confirm whether the detection point is located on the knob plane, and only detects a single key point, is easily interfered by noise or abnormal points, and lacks multiple optimization fittings, thereby effectively improving the reliability of the monitoring point and significantly improving the stability and accuracy of the fitting results through multiple fitting and average optimization.

[0014] The technical solutions of the present invention are as follows.

[0015] A method for detecting the posture of a knob switch based on multi-point fitting is disclosed. The method involves a system comprising an electrical cabinet and a mobile robotic arm, wherein an operating panel of the electrical cabinet is equipped with a knob switch, the knob switch being a multi-position self-locking knob switch, and one end of an operating handle of the knob switch being marked with an indicator arrow; a depth camera is installed at the end of the mobile robotic arm, and when photographing the knob switch using the depth camera, the mobile robotic arm is first moved to the front of the area where the knob switch of the electrical cabinet is located. After unfolding, the mobile robotic arm is kept such that the optical axis of the depth camera is perpendicular to the operating surface of the electrical cabinet and the plane where the depth camera lens is located is parallel to the operating surface of the electrical cabinet. The depth camera is then used to photograph the area where the knob switch is located, and during the photographing process, the operating handle of the knob switch is ensured to be clearly visible and the knob switch is located at the center of the image.

[0016] The posture detection method comprises the following steps:

[0017] Step 1: Build a target detection model for the knob switch;

[0018] Use the depth camera to capture N images of the knob switch, and use the labelme annotation software to annotate the captured images to obtain three key areas of the knob switch operating handle, which are recorded as the arrow key area h, the center key area m, and the tail key area t. Among them, the arrow key area h covers the part of the operating handle where the arrow is located, the center key area m covers the geometric center of the operating handle, and the tail key area t covers the edge of the end of the operating handle;

[0019] Select K images from the labeled images to form a training set, and the remaining NK images to form a test set, where K < N. Use the YOLOv8 object detection algorithm to train and test the labeled files and the labeled images, and obtain an object detection model for the knob switch, which is recorded as model A.

[0020] Step 2: capture the target image and obtain pixel information of three key areas of the target image;

[0021] Use a depth camera to capture the area where the knob switch is located, and record the resulting image as the target image;

[0022] Model A is used to detect the target image. Specifically, model A selects the entire area of ​​the knob switch operating handle in the target image through a detection frame 1. The width of the detection frame 1 in the pixel coordinate system of the target image is ω1 and the height is σ1. The pixel coordinates of the upper left corner of the detection frame 1 are (u r1 , v r1 ); The pixel coordinates of the arrow key area h, the center key area point m and the tail key area t of the target image in the pixel coordinate system of the target image are obtained through the detection frame 1, wherein the pixel coordinates of the upper left corner and the lower right corner of the arrow key area h are The pixel coordinates of the upper left corner and lower right corner of the central key area m are The pixel coordinates of the upper left corner and lower right corner of the tail key area t are

[0023] Step 3: traverse the points in the key area and classify their reliability, and determine the current gear position of the knob switch;

[0024] Step 3.1: Use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and record this distance as the operating plane depth value d calib ; Then all the pixels in the three key areas are traversed point by point according to the following coordinate ranges:

[0025] Arrow key area h:

[0026] Central key area m:

[0027] Tail key area t:

[0028] Among them, u i , v i is the pixel coordinate of any pixel i in the three key areas, i is the serial number of the pixel; during the traversal process, the depth value d of each pixel i traversed is recorded i and pixel coordinates u i , v i ;

[0029] Step 3.2: After completing the traversal of all pixels in the three key areas in step 3.1, calculate the depth value d of each pixel i and the depth value d of the operating plane calib Depth error Δd i , and make the following judgment based on the given depth threshold δ:

[0030] If Δd i >δ, then the pixel is judged to be located on the knob surface and is a reliable point;

[0031] If Δd i ≤δ, then the pixel is determined not to be on the knob surface and is an unreliable point;

[0032] Step 3.3: discard unreliable points and record the set of reliable points obtained in the arrow key area h as set P h , the set of reliable points obtained in the central key area m is recorded as set P m , the set of reliable points obtained in the tail key area t is recorded as set P t ;

[0033] Step 3.4: Based on the information of the detection frame 1 and the three key areas of the target image, the current position of the knob switch is determined according to the given confirmation criteria: if the current position of the knob switch is different from the expected position, proceed to step 4; if the current position of the knob switch is the same as the expected position, end this operation;

[0034] Step 4, get the set P in step 3.3 h , set P m and set P t In the N sample The random sampling of the sub-reliable points, specifically, each time from the set P h , set Px and set P t We extract one reliable point from each of the three points to form a set of data to be fitted, N sample Get N times of drawing sample Set of data to be fitted;

[0035] N sample The least square method is used to perform multi-point fitting on the set of data to be fitted, and the direction line of the knob switch operating handle is obtained. At the same time, the angle θ1 between the direction line and the positive direction of the image U axis is calculated and recorded as the current gear angle θ1;

[0036] Step 5: record the angle between the operating handle of the desired gear of the knob switch and the positive direction of the image U axis as the desired gear angle θ i , 0≤θ i <π and |θ1-θ i |≤θ max , where θ max The maximum allowable rotation range of the current gear angle θ1 is selected according to the following rules:

[0037] When θ1-θ i ≥0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle clockwise |θ1-θ i |To complete the gear adjustment;

[0038] When θ1-θ i <0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle counterclockwise |θ1-θ i |To complete the gear adjustment;

[0039] Step 6: After the gear adjustment is completed, the depth camera is used again to photograph the area where the adjusted knob switch is located, and the image obtained by this shooting is recorded as the test image;

[0040] The inspection image is inspected using the model A described in step 1 to obtain a detection frame 2 at the location of the knob switch in the inspection image. The width of the detection frame 2 in the pixel coordinate system of the inspection image is ω2, the height is σ2, and the pixel coordinates of the upper left corner of the detection frame 2 are (u r2 , v r2 ); obtain pixel coordinates of the arrow key area h2, the center key area m2 and the tail key area t2 of the inspection image in the pixel coordinate system of the inspection image through the detection frame 2;

[0041] Step 7: Based on the information of the detection frame 2 and the three key areas of the test image obtained in step 6, the gear state after rotation is judged according to the confirmation criteria described in step 3.4, and the current gear angle θ of the adjusted knob switch is calculated based on the detection results of the test image. new , 0≤θ new <π; then make the following judgment based on the preset angle error threshold ε:

[0042] If |θ1-θ new|≤ε, the gear state of the knob switch after adjustment is considered to be consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear state, and the process returns to step 2.

[0043] Preferably, the knob switch is a three-position self-locking knob switch, and the position confirmation criteria of the knob switch in steps 3.4 and 7 are:

[0044] First, take the set P described in step 3.3 h The average value of the pixel coordinates of all reliable points in is recorded as The calculation formula is:

[0045]

[0046] Where M is the set P h The number of reliable points in u m , v m For the set P h The pixel coordinates of any reliable point in the set P h The serial number of the reliable point;

[0047] Then make the following judgment:

[0048] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the left position;

[0049] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the middle position;

[0050] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the right position;

[0051] In the position confirmation criteria of the knob switch in step 3.4, ω is ω1, is σ1, u is u r1 , v is v r1 In the position confirmation criteria of the knob switch in step 7, ω is ω2, is σ2, u is ur2, and v is v r2 .

[0052] Preferably, the step 4 is to sample The process of using the least squares method to perform multi-point fitting on the data to be fitted is as follows:

[0053] For each set of data to be fitted, three reliable points are fitted into a straight line using the least squares method. The slope of the fitted straight line is k.j , the intercept is b j ;

[0054] N sample The results of the fitting are recorded, and the set of all fitting results is saved and recorded as the slope set K and the intercept set B, K = {k1, k2, ..., k j ,...,k Nsample}, B={b1, b2, ..., b j ,...,b Nsample};

[0055] The fitting results in the slope set K and the intercept set B are averaged and recorded as slope k avg and intercept b avg , the slope k avg and intercept b avg That is, the slope and intercept of the direction line of the knob switch operating handle; the angle between the direction line and the positive direction of the image U axis is recorded as θ1, θ1 = arctan (k avg ), 0≤θ1<π.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The present invention directly identifies the position information of the knob operating handle through the YOLOv8 target detection model, without relying on the attachment of auxiliary markers. This greatly reduces maintenance costs and avoids the problem of decreased detection accuracy caused by the falling off or damage of markers, thereby improving the long-term stability of the system.

[0058] 2. This invention uses a depth camera to obtain depth information from sampling points, and uses this depth information to determine the validity of the detection point's area, ensuring that the selected point is truly located on the knob plane. Compared to traditional methods that lack the ability to confirm the reliability of detection points, this judgment process in this invention significantly reduces the error caused by incorrect sampling c, thereby improving the overall pose detection accuracy.

[0059] 3. Compared to previous target detection algorithms that only use a single point as a valid point, this method determines the location of key points by averaging multiple points within a region. It then uses the least squares method to fit a line multiple times, averaging the results. This effectively reduces the impact of noise or outliers in a single fit. Compared to traditional methods that only detect a single point and perform a single fit, this method significantly improves fitting accuracy and results stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A structural diagram of an electrical cabinet knob in the implementation of the method of the present invention;

[0061] Figure 2Schematic diagram of annotation of captured images in the detection model according to an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of determining the current gear position of the knob switch in an embodiment of the present invention;

[0063] Figure 4 This is a diagram showing the locations of the electrical cabinet and depth camera in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of a straight line fitting of key points in an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of the method of the present invention;

[0066] Figure 7 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below with reference to the accompanying drawings.

[0068] The present invention provides a method for detecting the position of a knob switch based on multi-point fitting. The system involved in the method includes an electrical cabinet and a mobile robotic arm. The operating panel of the electrical cabinet is equipped with a knob switch, which is a multi-position self-locking knob switch, and one end of the operating handle of the knob switch is marked with an indicator arrow. The end of the mobile robotic arm is equipped with a depth camera. When using the depth camera to photograph the knob switch, the mobile robotic arm is first moved to the area where the knob switch of the electrical cabinet is located. After unfolding, the depth camera optical axis is kept perpendicular to the operating surface of the electrical cabinet, and the plane where the depth camera lens is located is parallel to the operating surface of the electrical cabinet. Then, the depth camera is used to photograph the area where the knob switch is located. During the photographing process, the knob switch operating handle is ensured to be clearly visible and the knob switch is located at the center of the image.

[0069] Figure 1 The structural diagram of the knob switch of the electrical cabinet in the implementation of the method of the present invention is given. Figure 4 The position between the electrical cabinet and the depth camera in an embodiment of the present invention is given.

[0070] In this embodiment, the depth camera is an Intel RealSense D415i real-sense depth camera, which has a ranging function.

[0071] In this embodiment, the knob switch is a three-position self-locking knob switch, that is, it has three positions. When the operating handle is in the left position, the angle between the operating handle of the three-position self-locking knob switch and the positive direction of the image U axis is 135°, and the indicator arrow points to the upper left; when the operating handle is in the middle position, the angle between the operating handle of the three-position self-locking knob switch and the positive direction of the image U axis is 90°, and the indicator arrow points directly upward; when the operating handle is in the right position, the angle between the operating handle of the three-position self-locking knob switch and the positive direction of the image U axis is 45°, and the indicator arrow points to the upper right.

[0072] Figure 6 A schematic diagram of the method of the present invention is shown in FIG. Figure 7 is a flow chart of the method of the present invention, Figure 6 and Figure 7 It can be seen that the posture detection method of the present invention includes the following steps:

[0073] Step 1: Build a target detection model for the knob switch.

[0074] The depth camera is used to capture N images of the knob switch, and the captured images are annotated using the labelme annotation software to obtain three key areas of the knob switch operating handle, which are respectively recorded as the arrow key area h, the center key area m, and the tail key area t. Among them, the arrow key area h covers the part of the operating handle where the arrow is located, the center key area m covers the geometric center of the operating handle, and the tail key area t covers the edge part of the end of the operating handle.

[0075] K images are selected from the labeled images to form a training set, and the remaining NK images are selected to form a test set, where K < N. The labeled files obtained after the labeling are trained and tested together with the labeled images using the YOLOv8 object detection algorithm to obtain an object detection model for the knob switch, which is recorded as model A.

[0076] Figure 2 Schematic diagram of annotation of collected images in the detection model in an embodiment of the present invention.

[0077] In this example, the acquisition distance, lighting conditions, and knob switch model were selected. Specifically, the distance from the electrical cabinet operating surface was divided into five equal parts within the range of 20 cm to 70 cm, resulting in six image acquisition distances; three different lighting conditions were used: indoor lighting on during the day, indoor lighting off during the day, and indoor lighting on at night; and five different models of three-position self-locking knob switches were used. Therefore, this example consists of N = 3 × 6 × 3 × 5 = 270 images, of which 54 images were randomly selected to form the test set, accounting for 20%, and the remaining 216 images constitute the training set, accounting for 80%.

[0078] In this embodiment, the marking process of the three key areas is as follows:

[0079] First, this method uses 1abelme annotation software to annotate a training frame of the knob switch operating handle. The training frame must completely cover the knob switch operating handle, and its center coincides with the geometric center of the knob switch operating handle; the area m of the geometric center of the knob switch operating handle is annotated. This area must surround the geometric center point of the knob, and the rectangular frame covers the area near the center of the knob handle and ensures the symmetry of the area; the area h where the tip of the indicator arrow of the knob switch operating handle is located is annotated, and a small area around the tip of the indicator arrow is covered to accurately indicate the directionality of the knob; the annotated rectangular frame of the arrow key area must include the tip and nearby area of ​​the indicating part of the knob handle; the tail edge area t of the end of the knob switch operating handle without an arrow is annotated, and the rectangular frame covers the central symmetrical area of ​​the end edge of the knob; in addition, this method can also be adapted to other annotation tools that support similar functions.

[0080] Step 2: Take a target image and obtain pixel information of three key areas of the target image.

[0081] Use a depth camera to capture the area where the knob switch is located, and record the obtained image as the target image.

[0082] Model A is used to detect the target image. Specifically, model A selects the entire area of ​​the knob switch operating handle in the target image through a detection frame 1. The width of the detection frame 1 in the pixel coordinate system of the target image is ω1 and the height is σ1. The pixel coordinates of the upper left corner of the detection frame 1 are (u r1 , v r1 ); The pixel coordinates of the arrow key area h, the center key area point m and the tail key area t of the target image in the pixel coordinate system of the target image are obtained through the detection frame 1, wherein the pixel coordinates of the upper left corner and the lower right corner of the arrow key area h are The pixel coordinates of the upper left corner and lower right corner of the central key area m are The pixel coordinates of the upper left corner and lower right corner of the tail key area t are

[0083] Step 3: traverse the points in the key area and classify their reliability, and determine the current gear position of the knob switch.

[0084] Step 3.1: Use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and record this distance as the operating plane depth value d calib ; Then all the pixels in the three key areas are traversed point by point according to the following coordinate ranges:

[0085] Arrow key area h:

[0086] Central key area m:

[0087] Tail key area t:

[0088] Among them, u i , v i is the pixel coordinate of any pixel i in the three key areas, i is the serial number of the pixel; during the traversal process, the depth value d of each pixel i traversed is recorded i and pixel coordinates u i , v i ;

[0089] Step 3.2: After completing the traversal of all pixels in the three key areas in step 3.1, calculate the depth value d of each pixel i and the depth value d of the operating plane calib Depth error Δd i , and make the following judgment based on the given depth threshold δ:

[0090] If Δd i >δ, then the pixel is judged to be located on the knob surface and is a reliable point;

[0091] If Δd i ≤δ, then the pixel is determined not to be on the knob surface and is an unreliable point;

[0092] Step 3.3: discard unreliable points and record the set of reliable points obtained in the arrow key area h as set P h , the set of reliable points obtained in the central key area m is recorded as set P m , the set of reliable points obtained in the tail key area t is recorded as set P t ;

[0093] In step 3.4, based on the information of the detection frame 1 and the three key areas of the target image, the current position of the knob switch is judged according to the given confirmation criteria: if the current position of the knob switch is different from the expected position, proceed to step 4; if the current position of the knob switch is the same as the expected position, end this operation.

[0094] Step 4, get the set P in step 3.3 h , set P m and set P t In the N sample The random sampling of the sub-reliable points, specifically, each time from the set P h , set P m and set P t We extract one reliable point from each of the three points to form a set of data to be fitted, N sample Get N times of drawingsample Set of data to be fitted.

[0095] N sample The least squares method is used to perform multi-point fitting on the set of data to be fitted, and the direction line of the knob switch operating handle is obtained. At the same time, the angle θ1 between the direction line and the positive direction of the image U axis is calculated and recorded as the current gear angle θ1.

[0096] In this embodiment, the pair N sample The implementation process of multi-point fitting using the least squares method for the data to be fitted is as follows:

[0097] For each set of data to be fitted, three reliable points are fitted into a straight line using the least squares method. The slope of the fitted straight line is k. j , the intercept is b j ;

[0098] The results of each fitting are recorded, and the set of all fitting results is saved and recorded as the slope set K and the intercept set B, K = {k1, k2, ..., k j ,...,k Nsample}, B={b1, b2, ..., b j ,...,b Nsample};

[0099] The fitting results in the slope set K and the intercept set B are averaged and recorded as slope k avg and intercept b avg , the slope k avg and intercept b avg That is, the slope and intercept of the direction line of the knob switch operating handle; the angle between the direction line and the positive direction of the image U axis is recorded as θ1, θ1 = arctan (k avg ), 0≤θ1<π.

[0100] Figure 5 Schematic diagram of straight line fitting of key points in an embodiment of the present invention.

[0101] Step 5: record the angle between the operating handle of the desired gear of the knob switch and the positive direction of the image U axis as the desired gear angle θ i , 0≤θ i <π and |θ1-θ i |≤θ max , where θ max The maximum allowable rotation range of the current gear angle θ1 is selected according to the following rules:

[0102] When θ1-θ i ≥0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle clockwise |θ1-θi |To complete the gear adjustment;

[0103] When θ1-θ i <0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle counterclockwise |θ1-θ i |To complete the gear adjustment.

[0104] During the gear adjustment process, adjust the knob switch to the target position using the clamping device and then release it. Before releasing it, stabilize the knob handle position with the clamping device to ensure that the knob remains in the target position during the release process, and verify the current position of the knob to ensure that the operation is complete.

[0105] Step 6: After the gear adjustment is completed, the depth camera is used again to photograph the area where the adjusted knob switch is located, and the image obtained by this shooting is recorded as the inspection image.

[0106] The inspection image is inspected using the model A described in step 1 to obtain a detection frame 2 at the location of the knob switch in the inspection image. The width of the detection frame 2 in the pixel coordinate system of the inspection image is ω2, the height is σ2, and the pixel coordinates of the upper left corner of the detection frame 2 are (u r2 , v r2 ); The pixel coordinates of the arrow key area h2, the center key area m2 and the tail key area t2 of the inspection image are obtained through the detection frame 2 in the pixel coordinate system of the inspection image.

[0107] Step 7: Based on the information of the detection frame 2 and the three key areas of the test image obtained in step 6, the gear state after rotation is judged according to the confirmation criteria described in step 3.4, and the current gear angle θ of the adjusted knob switch is calculated based on the detection results of the test image. new , 0≤θ new <π; then make the following judgment based on the preset angle error threshold ε:

[0108] If |θ1-θ new |≤ε, the gear state of the knob switch after adjustment is considered to be consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear state, and the process returns to step 2.

[0109] In this embodiment, the knob switch is a three-position self-locking knob switch. The position confirmation criteria of the knob switch in steps 3.4 and 7 are:

[0110] First, take the set P described in step 3.3 h The average value of the pixel coordinates of all reliable points in is recorded as The calculation formula is:

[0111]

[0112] Where M is the set P h The number of reliable points in u m , v m For the set P h The pixel coordinates of any reliable point in the set P h The serial number of the reliable point;

[0113] Then make the following judgment:

[0114] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the left position;

[0115] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the middle position;

[0116] When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the right position;

[0117] In the position confirmation criteria of the knob switch in step 3.4, ω is ω1, is σ1, u is u r1 , v is v r1 In the position confirmation criteria of the knob switch in step 7, ω is ω2, is σ2, u is u r2 , v is v r2 .

[0118] Figure 3 Schematic diagram of determining the current gear position of the knob switch in an embodiment of the present invention.

[0119] In the present invention, the pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin and in pixels, including a pixel U axis and a pixel V axis that are perpendicular to each other. The horizontal coordinate on the pixel U axis is the number of columns of the pixel in its image, and the direction is parallel to the image plane and to the right. The vertical coordinate on the pixel V axis is the number of rows of the pixel in its image, and the direction is perpendicular to the pixel U axis and downward. The specific situation of the pixel coordinate system can be seen Figure 2 .

Claims

1. A method for detecting the pose of a knob switch based on multi-point fitting. The system involved in this method includes an electrical cabinet and a mobile robotic arm. The operation panel of the electrical cabinet is equipped with a knob switch, which is a multi-position self-locking knob switch, and one end of the operating handle of the knob switch is marked with an indicating arrow. The end of the mobile robotic arm is equipped with a depth camera. When using the depth camera to photograph the knob switch, first move the mobile robotic arm to the area in front of the knob switch on the electrical cabinet. After unfolding, keep the optical axis of the depth camera perpendicular to the operating surface of the electrical cabinet and the plane where the depth camera lens is located parallel to the operating surface of the electrical cabinet. Then use the depth camera to photograph the area where the knob switch is located, and ensure that the operating handle of the knob switch is clearly visible during the photographing process, and the knob switch is located at the center of the image. It is characterized by: The pose detection method includes the following steps: Step 1, construct a target detection model for the knob switch; Use the depth camera to take N images of the knob switch, and use the labelme annotation software to annotate the collected images to obtain three key areas of the operating handle of the knob switch, which are respectively denoted as the arrow key area h, the center key area m, and the tail key area t. Among them, the arrow key area h covers the part where the indicating arrow of the operating handle is located, the center key area m covers the geometric center of the operating handle, and the tail key area t covers the edge part of the end of the operating handle. Select K images from the annotated images to form a training set, and the remaining N - K images form a test set, where K < N. Use the YOLOv8 object detection algorithm to train and test the annotated files and the annotated images together to obtain a target detection model for the knob switch, which is denoted as model A. Step 2, take a target image and obtain the pixel point information of the three key areas of the target image; Use the depth camera to photograph the area where the knob switch is located, and record the obtained image as the target image. Model A is used to detect the target image. Specifically, model A selects the entire area of ​​the knob switch operating handle in the target image through a detection frame 1. The width of the detection frame 1 in the pixel coordinate system of the target image is , height is , the pixel coordinates of the upper left corner of the detection frame 1 are (u r1 ,v r1 ); The pixel coordinates of the arrow key area h, the center key area point m and the tail key area t of the target image in the pixel coordinate system of the target image are obtained through the detection frame 1, wherein the pixel coordinates of the upper left corner and the lower right corner of the arrow key area h are 、 , the pixel coordinates of the upper left corner and lower right corner of the central key area m are 、 , the pixel coordinates of the upper left corner and lower right corner of the tail key area t are 、 ; Step 3, traverse the points in the key areas, classify their reliability, and judge the current gear of the knob switch; Step 3.1: Use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and record this distance as the operating plane depth value. ; Then all the pixels in the three key areas are traversed point by point according to the following coordinate ranges: Arrow key area h: ; Central key area m: ; Tail key area t: ; in, is the pixel coordinate of any pixel i in the three key areas, where i is the pixel number; during the traversal process, the depth value of each pixel i is recorded. and pixel coordinates ; Step 3.2, after completing the traversal of all pixels in the three key areas in step 3.1, calculate the depth value of each pixel Depth value of the operating plane Depth error , and according to the given depth threshold Make the following judgment: like , then it is judged that the pixel point is located on the knob surface and is a reliable point; like , then it is determined that the pixel point is not located on the knob surface and is an unreliable point; Step 3.3, discard unreliable points and record the set of reliable points obtained in the arrow key area h as the set , the set of reliable points obtained in the central key area m is recorded as the set , the set of reliable points obtained in the tail key area t is recorded as the set ; Step 3.4, according to the information of the detection box 1 and the three key areas of the target image, judge the current gear of the knob switch according to the given confirmation criterion: If the current gear of the knob switch is different from the expected gear, go to step 4; If the current gear of the knob switch is the same as the expected gear, end this operation. Step 4, get the set in step 3.3 ,gather and collection In The random sampling of the second most reliable points, specifically, each time from the set ,gather and collection Extract one reliable point from each to get three reliable points, and form a set of data to be fitted. Times drawn Set of data to be fitted; right The least square method is used to perform multi-point fitting on the set of data to be fitted, and the direction line of the knob switch operating handle is obtained. At the same time, the angle between the direction line and the positive direction of the image U axis is calculated. , and record it as the current gear angle ; Step 5: record the angle between the operating handle of the desired gear of the knob switch and the positive direction of the image U axis as the desired gear angle , and ,in, is the current gear angle The maximum permissible rotation amplitude; the direction of rotation is selected according to the following rules: when When the knob is turned, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle clockwise. To complete the gear adjustment; when , control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle counterclockwise To complete the gear adjustment; Step 6, after completing the gear adjustment, use the depth camera to photograph the area where the adjusted knob switch is located again, and record the image obtained from this photographing as the inspection image. The inspection image is detected using the model A described in step 1 to obtain a detection frame 2 at the location of the knob switch in the inspection image. The width of the detection frame 2 in the pixel coordinate system of the inspection image is , height is , the pixel coordinates of the upper left corner of detection frame 2 are ; Obtain pixel coordinates of the arrow key area h2, the center key area m2 and the tail key area t2 of the inspection image in the pixel coordinate system of the inspection image through the detection frame 2; Step 7: Based on the information of the detection frame 2 and the three key areas of the test image obtained in step 6, the gear state after rotation is judged according to the confirmation criteria described in step 3.4, and the current gear angle of the knob switch after adjustment is calculated based on the detection results of the test image. , ; Then according to the preset angle error threshold Make the following judgment: like , it is considered that the gear state of the knob switch after adjustment is consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear state, and return to step 2.

2. The method for detecting the position of a knob switch based on multi-point fitting according to claim 1, characterized in that: The knob switch is a three-position self-locking knob switch. The gear confirmation criteria for the knob switch in steps 3.4 and step 7 are as follows: First, take the set described in step 3.3 The average value of the pixel coordinates of all reliable points in is recorded as , and its calculation formula is: ； ； Where M is the set The number of reliable points in u m ,v m For collection The pixel coordinates of any reliable point in the set m The serial number of the reliable point; Then make the following judgments: When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the left position; When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the middle position; When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the right position; ω, б are the width and height of the detection frame in the pixel coordinate system of the test image, respectively. r， v r ) is the pixel coordinate of the upper left corner of the detection frame; in the position confirmation criteria of the knob switch in step 3.4, ω is , б is u r for u r1 , v r v r1 In the gear confirmation criteria of the knob switch in step 7, ω is , б is ,u r for u r2 , v r v r2 .

3. The method for detecting the position of a knob switch based on multi-point fitting according to claim 1, wherein: Step 4 The process of using the least squares method to perform multi-point fitting on the data to be fitted is as follows: For each set of data to be fitted, three reliable points are fitted into a straight line using the least squares method. The slope of the fitted straight line is , the intercept is ; right The results of the fitting are recorded, and the set of all fitting results is saved and recorded as the slope set K and the intercept set B, K={k1,k2,...,k j ,...,k Nsample },B={b1,b2,...,b j ,...,b Nsample }; The fitting results in the slope set K and the intercept set B are averaged and recorded as the slope and intercept , the slope and intercept The slope and intercept of the direction line of the knob switch operating handle are recorded as , , .

Images