Knob switch pose detection method based on multi-point fitting

Abstract

The invention discloses a knob switch pose detection method based on multi-point fitting, and belongs to the technical field of robot mechanical arm control. The method comprises the following steps: constructing a detection model of the knob switch; collecting a target switch image, obtaining a pixel coordinate range of a key area, and verifying a reliable point; and multiple groups of reliable points are randomly extracted, and a direction line of the knob operation handle is obtained by adopting multiple least square fitting, and the current pose information of the knob switch is obtained. Depth information of a sampling point is verified through the depth camera, it is ensured that the sampling point is actually located on the surface of the rotary knob, and mistaken detection of a background or an interferent is avoided; through multi-point sampling and multi-time fitting, the influence of a single-point error can be eliminated, a knob pose detection result can be optimized, precision and stability can be ensured, knob switch operation handles under different rotation angles can be effectively identified, and the precision requirement of automatic operation equipment for operating an electrical cabinet knob switch is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot manipulator control. Specifically, it relates to a method for detecting the position and pose of a knob switch based on multi-point fitting. Background Art

[0002] In recent years, with the rapid development of the national power grid system and high-speed railway system in China, the automation level of the power and railway systems has been significantly improved. As an important part of industrial automation control, the knob switches of electrical cabinets are widely used to adjust functions such as the working mode and start / stop state of equipment. Traditional knob switches of electrical cabinets mostly rely on manual operation, which not only leads to low operation efficiency but also increases the risk of human error. Especially in application scenarios with high-precision requirements, it is difficult to ensure the stability and accuracy of the knob switch.

[0003] To achieve unmanned management of substations, many substations have begun to introduce automated equipment such as mobile manipulators to replace manual operations, such as operating the knob switches on electrical cabinets. However, in the actual application process, the existing automated equipment generally faces the following problems when operating the knob switch: First, since the automated equipment cannot accurately identify the position and pose information of the knob switch, it often cannot complete the operation task and may even pose a safety hazard; Second, the method of using adhesive indicators to assist in identification increases the maintenance difficulty and affects the cleanliness and standardization of the electrical cabinet panel; Finally, the reliability of some existing methods often cannot be well guaranteed, restricting the application of the existing technology in complex environments and occasions with high-precision requirements. Therefore, how to enable the manipulator to accurately detect the position and pose of the knob switch so as to complete high-precision operations is a major challenge in the application of current automated equipment.

[0004] To solve these problems, the present invention proposes a method for detecting the position and pose of a knob switch based on multi-point fitting. By combining deep learning technology and a depth camera device, and using precise image acquisition and algorithm processing, it ensures that the manipulator can accurately identify the current position and pose of the knob, thereby achieving precise operation of the knob switch.

[0005] Currently, there are some related technologies for identifying the positions of knob switches. Chen Yang proposed a method in "Research on the Design and Motion Planning of Intelligent Emergency Operation Robots for Substations" by pasting an indicator strip and using color features for screening, combined with calculating the minimum circumscribed rectangle of the indicator strip to obtain the position and pose of the knob switch handle. This method can detect the position and pose of the knob switch under specific conditions, but this method relies on the pasted indicator strip as an auxiliary identifier. Once the indicator strip falls off or is damaged, the recognition accuracy will be severely affected; Second, this method still cannot completely get rid of the influence of the position deviation of the indicator strip on the recognition result, and it is difficult to achieve precise positioning of the knob operation point, which may affect the accuracy and stability of actual operations.

[0006] Shi Mengan, Lu Zhenyu and others proposed a switch position recognition method based on deep learning target detection algorithm in "Power station knob switch state recognition based on YOLO-tiny-RFB model", which divides switches in different positions into different categories for target detection. This method can accurately recognize knob switches in some positions, but it cannot obtain the current angle of the knob switch operating handle, and it still cannot accurately locate the knob switch operating point.

[0007] Zhu Min, Chen Jie and others proposed a method for detecting the position and posture of a knob switch based on key point detection in the Chinese invention patent document "A method for detecting and operating the position and posture of a knob switch in an electrical cabinet" (CN118254180B). This method can realize the position and posture detection and operation of the knob switch under certain conditions, but there are still some shortcomings: first, the method cannot determine the reliability of the key points; second, the method only performs a single key point detection and fitting operation, which may cause noise or abnormal points to have a greater impact on the overall result; finally, the method only performs a least squares operation on the key point fitting line once, and does not further reduce the error by taking the average value after multiple fittings, which may lead to insufficient detection and operation accuracy in complex scenarios.

[0008] The limitations of these existing methods indicate that further optimization is still needed in practical applications to improve their robustness and adaptability, so as to meet the high-precision requirements of automation equipment in the operation of electrical cabinet knob switches.

[0009] In robot automation operations, machine vision technology, as one of the core technologies, determines whether the robot can accurately identify and operate the knob switch in the electrical cabinet. Accurately identifying the position information of the knob switch is a prerequisite for ensuring that the robot completes the task accurately. However, there are still some technical problems that need to be solved in the visual technology of the robot identifying the knob switch of the electrical cabinet. These problems affect the accuracy and stability of the knob identification and operation. Specifically, the research on the visual technology of the robot identifying the knob switch of the electrical cabinet still has the following technical problems:

[0010] 1. Existing recognition methods based on traditional visual technology usually require the attachment of auxiliary markers, such as calibration plates, on the knob switch to help the robot identify the position information of the knob. This method not only increases maintenance costs, but may also reduce recognition accuracy due to the fall-off or damage of the marker, affecting the stability of the knob operation.

[0011] 2. In current deep learning-based detection methods, the recognition of the knob pose often ignores the judgment of the depth information of the sampling points, which may lead to the sampling points not being located on the surface of the knob, thus affecting the final pose recognition result and the accuracy of knob operation. Moreover, most methods only detect a single key point and do not perform extended detection on the key areas of the knob handle, making them vulnerable to interference from noise or outliers and affecting the detection stability.

[0012] 3. Currently existing knob detection methods usually only perform a single fitting operation on a single key point, lacking multiple fitting optimizations or the averaging process of multi-point detection within the region, and failing to consider the influence of environmental changes or error accumulation, resulting in insufficient fitting accuracy and detection result stability, which may lead to the accumulation of errors and affect the accuracy of subsequent operations. Summary of the Invention

[0013] The technical problem to be solved by the present invention is the above-mentioned defects. Specifically, aiming at the problems existing in the recognition of the pose information of the knob switch of the automated operation equipment in the prior art, a knob switch pose detection method based on depth information, multi-point detection and fitting optimization is proposed. Specifically, the present invention solves the problems that it is difficult for traditional methods to confirm whether the detection point is on the knob plane, only detects a single key point, is vulnerable to interference from noise or outliers, and lacks multiple optimization fittings, thereby effectively improving the reliability of the monitoring points and significantly improving the stability and accuracy of the fitting results through multiple fittings and average optimization.

[0014] The technical solution of the present invention is as follows.

[0015] A knob switch pose detection method based on multi-point fitting. The system involved in this method includes an electrical cabinet and a mobile robotic arm. The operation panel of the electrical cabinet is equipped with a knob switch, which is a multi-gear self-locking knob switch, and one end of the operation handle of the knob switch is marked with an indicating arrow. The end of the mobile robotic arm is equipped with a depth camera. When using the depth camera to photograph the knob switch, first move the mobile robotic arm to the area in front of the knob switch of the electrical cabinet. After unfolding, keep the optical axis of the depth camera perpendicular to the operation surface of the electrical cabinet and the plane where the depth camera lens is located parallel to the operation surface of the electrical cabinet. Then use the depth camera to photograph the area where the knob switch is located, and ensure that the operation handle of the knob switch is clearly visible and the knob switch is located at the center of the image during the photographing process.

[0016] The pose detection method includes the following steps:

[0017] Step 1, construct a target detection model for the knob switch.

[0018] Use the depth camera to capture N images of the rotary switch, and use the labelme annotation software to annotate the captured images to obtain three key regions of the operating handle of the rotary switch, which are respectively denoted as the arrow key region h, the center key region m, and the tail key region t. Among them, the arrow key region h covers the part where the indicating arrow of the operating handle is located, the center key region m covers the geometric center of the operating handle, and the tail key region t covers the edge part of the end of the operating handle;

[0019] Select K images from the annotated images to form a training set, and the remaining N - K images form a test set, where K < N; use the YOLOv8 object detection algorithm to train and test the annotated files and the annotated images together to obtain an object detection model for the rotary switch, and denote it as model A;

[0020] Step 2, capture the target image and obtain the pixel point information of three key regions of the target image;

[0021] Use the depth camera to capture the area where the rotary switch is located, and denote the obtained image as the target image;

[0022] Use model A to detect the target image. Specifically, model A selects the overall area of the operating handle of the rotary switch in the target image through a detection box 1. The width of this detection box 1 in the pixel coordinate system of the target image is ω 1 and the height is σ 1 , and the pixel coordinates of the upper left corner of the detection box 1 are (u r1 , v r1 ); obtain the pixel coordinates of the arrow key region h, the center key region point m, and the tail key region t of the target image in the pixel coordinate system of the target image through this detection box 1. Among them, the pixel coordinates of the upper left corner and the lower right corner of the arrow key region h are respectively The pixel coordinates of the upper left corner and the lower right corner of the center key region m are respectively The pixel coordinates of the upper left corner and the lower right corner of the tail key region t are respectively

[0023] Step 3, traverse the points in the key regions, classify their reliability, and judge the current gear of the rotary switch;

[0024] Step 3.1, use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and denote this distance as the operating plane depth value d calib ; then traverse all the pixel points in the three key regions one by one according to the following coordinate ranges:

[0025] Arrow key region h:

[0026] Center key region m:

[0027] Tail key area t:

[0028] where u i , v i are the pixel coordinates of any pixel point i in the three key areas, and i is the serial number of the pixel point; during the traversal process, record the depth value d i and pixel coordinates u i , v i ;

[0029] Step 3.2, after completing the traversal of all pixel points in the three key areas in Step 3.1, calculate the depth error Δd i between the depth value d calib of each pixel point and the depth value d i of the operation plane, and make the following judgment according to the given depth threshold δ:

[0030] If Δd i > δ, it is determined that the pixel point is located on the knob surface and is a reliable point;

[0031] If Δd i ≤ δ, it is determined that the pixel point is not located on the knob surface and is an unreliable point;

[0032] Step 3.3, discard the unreliable points, and denote the set of reliable points obtained in the arrow key area h as set P h , the set of reliable points obtained in the center key area m as set P m , and the set of reliable points obtained in the tail key area t as set P t ;

[0033] Step 3.4, according to the information of the detection frame 1 and the three key areas of the target image, judge the current gear of the knob switch according to the given confirmation criterion: if the current gear of the knob switch is different from the expected gear, go to Step 4; if the current gear of the knob switch is the same as the expected gear, end this operation;

[0034] Step 4, in the sets P h , set P m and set P t obtained in Step 3.3, randomly extract reliable points N sample times respectively. Specifically, each time, extract one reliable point from each of the sets P h , set Px and set P t to obtain three reliable points and form a set of data to be fitted. After N sample extractions, N sampleGroup of data to be fitted;

[0035] For N sample groups of data to be fitted, perform multi-point fitting using the least squares method to obtain the direction line of the operating handle of the rotary switch, and simultaneously calculate the angle θ between this direction line and the positive direction of the U-axis of the image 1 , and denote it as the current gear angle θ 1 ;

[0036] Step 5, denote the angle between the operating handle of the expected gear of the rotary switch and the positive direction of the U-axis of the image as the expected gear angle θ i , 0 ≤ θ i <π and |θ 1 -θ i | ≤ θ max , where θ max is the maximum allowable rotation amplitude of the current gear angle θ 1 ; Select the rotation direction according to the following rules:

[0037] When θ 1 -θ i ≥ 0, control the clamping device to clamp the operating handle of the rotary switch, and rotate the operating handle of the rotary switch clockwise by |θ 1 -θ i | to complete the gear adjustment;

[0038] When θ 1 -θ i <0, control the clamping device to clamp the operating handle of the rotary switch, and rotate the operating handle of the rotary switch counterclockwise by |θ 1 -θ i | to complete the gear adjustment;

[0039] Step 6, after completing the gear adjustment, use the depth camera to take a picture of the area where the adjusted rotary switch is located again, and denote the image obtained from this shooting as the inspection image;

[0040] Detect the inspection image using the model A described in Step 1 to obtain the detection frame 2 of the position of the rotary switch in the inspection image. The width of this detection frame 2 in the pixel coordinate system of the inspection image is ω 2 、height is σ 2 , and the pixel coordinates of the upper left corner of the detection frame 2 are (u r2 , v r2 ); Obtain the pixel coordinates of the arrow key area h2, the center key area m2, and the tail key area t2 of the inspection image in the pixel coordinate system of the inspection image through this detection frame 2;

[0041] Step 7: According to the detection box 2 obtained in Step 6 and the information of the three key regions of the inspection image, judge the rotated gear state according to the confirmation criteria described in Step 3.4, and calculate the adjusted current gear angle θ of the knob switch based on the detection result of the inspection image. new , 0 ≤ θ new < π; then make the following judgment according to the preset angle error threshold ε:

[0042] If |θ 1 - θ new | ≤ ε, it is considered that the gear state of the adjusted knob switch is consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear, and return to Step 2.

[0043] Preferably, the knob switch is a three-gear self-locking knob switch, and the gear confirmation criteria for the knob switch in Step 3.4 and Step 7 are as follows:

[0044] First, take the average value of the pixel coordinates of all reliable points in the set P described in Step 3.3 h and denote it as Its calculation formula is:

[0045]

[0046] In the formula, M is the number of reliable points in the set P h , u m , v m is the pixel coordinate of any reliable point in the set P h , and m is the serial number of the reliable point in the set P h ;

[0047] Then make the following judgment:

[0048] When the arrow key area h satisfies and , it is considered that the current knob switch gear is in the left gear;

[0049] When the arrow key area h satisfies and , it is considered that the current knob switch gear is in the middle gear;

[0050] When the arrow key area h satisfies and , it is considered that the current knob switch gear is in the right gear;

[0051] In the gear confirmation criteria for the knob switch in Step 3.4, ω is ω 1 , is σ 1 , u is u r1 , v is vr1 ; In the gear confirmation criterion of the rotary switch in step 7, ω is ω 2 , is σ 2 , u is ur 2 , v is v r2 .

[0052] Preferably, the process of multi-point fitting of the N sample groups of data to be fitted using the least squares method is as follows:

[0053] For three reliable points in each group of data to be fitted, a straight line is fitted using the least squares method, and the slope of the fitted straight line is denoted as k j , and the intercept is b j ;

[0054] Record the results of the N sample times of fitting, save the set of all fitting results and denote them as the slope set K and the intercept set B respectively, K = {k 1 , k 2 ,..., k j ,..., k Nsample}, B = {b 1 , b 2 ,..., b j ,..., b Nsample};

[0055] Calculate the average values of the fitting results in the slope set K and the intercept set B respectively and denote them as the slope k avg and the intercept b avg . This slope k avg and the intercept b avg are the slope and intercept of the direction line of the operating handle of the rotary switch; Denote the angle between this direction line and the positive direction of the U-axis of the image as θ 1 , θ 1 = arctan(k avg ), 0 ≤ θ 1 < π.

[0056] Compared with the prior art, the beneficial effects of the present invention include:

[0057] 1. The present invention directly identifies the pose information of the rotary operating handle through the YOLOv8 object detection model, without relying on pasting auxiliary markers, greatly reducing the maintenance cost and at the same time avoiding the problem of reduced detection accuracy caused by the falling off or damage of the markers, and improving the long-term stability of the system.

[0058] 2. The present invention utilizes a depth camera to obtain the depth information of sampling points, and determines the validity of the area where the detection points are located through the depth information, ensuring that the selected points are truly located on the knob plane. Compared with the deficiency of the traditional method that cannot confirm the reliability of the detection points, this judgment process of the present invention greatly reduces the error caused by incorrect sampling c, thereby improving the overall pose detection accuracy.

[0059] 3. Compared with the previous object detection algorithm that only takes a single point as the valid point, the present invention determines the position of the key point by calculating the average value of multiple points in the area, and uses the least squares method to fit the straight line multiple times and takes the average value of the fitting results, thereby effectively reducing the influence of noise or abnormal points on the results in a single fitting. Compared with the traditional method that only detects a single point and performs a single fitting operation, the present invention significantly improves the fitting accuracy and the stability of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural diagram of the electrical cabinet knob in the implementation of the method of the present invention;

[0061] Figure 2 It is a schematic diagram of the annotation of the acquired image in the detection model in the embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram for judging the current gear of the knob switch in the embodiment of the present invention;

[0063] Figure 4 It is a position diagram of the electrical cabinet and the depth camera in the embodiment of the present invention;

[0064] Figure 5 It is a schematic diagram of the straight line fitting of the key points in the embodiment of the present invention;

[0065] Figure 6 It is a simplified diagram of the method of the present invention;

[0066] Figure 7 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] The present invention will be described in detail below with reference to the accompanying drawings.

[0068] The present invention provides a method for detecting the pose of a rotary switch based on multi-point fitting. The system involved in this method includes an electrical cabinet and a mobile robotic arm. The operation panel of the electrical cabinet is equipped with a rotary switch, which is a multi-position self-locking rotary switch, and an indicating arrow is marked at one end of the operating handle of the rotary switch. A depth camera is installed at the end of the mobile robotic arm. When using the depth camera to photograph the rotary switch, first move the mobile robotic arm to the area in front of the rotary switch of the electrical cabinet. After unfolding, keep the optical axis of the depth camera perpendicular to the operating surface of the electrical cabinet, and the plane where the depth camera lens is located parallel to the operating surface of the electrical cabinet. Then use the depth camera to photograph the area where the rotary switch is located, and ensure that the operating handle of the rotary switch is clearly visible during the photographing process, and the rotary switch is located at the center of the image.

[0069] Figure 1 The structural diagram of the rotary switch of the electrical cabinet in the implementation of the method of the present invention is given. Figure 4 The position between the electrical cabinet and the depth camera in the embodiment of the present invention is given.

[0070] In this embodiment, the depth camera is an Intel RealSense D415i depth camera, and this camera has a ranging function.

[0071] In this embodiment, the rotary switch is a three-position self-locking rotary switch, that is, it has three positions. When the operating handle is in the left position, the included angle between the operating handle of the three-position self-locking rotary switch and the positive direction of the U-axis of the image is 135°, and the indicating arrow points to the upper left; when the operating handle is in the middle position, the included angle between the operating handle of the three-position self-locking rotary switch and the positive direction of the U-axis of the image is 90°, and the indicating arrow points directly upward; when the operating handle is in the right position, the included angle between the operating handle of the three-position self-locking rotary switch and the positive direction of the U-axis of the image is 45°, and the indicating arrow points to the upper right.

[0072] Figure 6 It is a simple diagram of the method of the present invention. Figure 7 It is a flow chart of the method of the present invention, which consists of Figure 6 and Figure 7 It can be seen that the pose detection method of the present invention includes the following steps:

[0073] Step 1, construct a target detection model of the rotary switch.

[0074] Use the depth camera to take N images of the rotary switch, and use the labelme annotation software to annotate the collected images to obtain three key areas of the operating handle of the rotary switch, which are respectively recorded as the arrow key area h, the center key area m, and the tail key area t. Among them, the arrow key area h covers the part where the indicating arrow of the operating handle is located, the center key area m covers the geometric center of the operating handle, and the tail key area t covers the edge part of the end of the operating handle.

[0075] Select K images from the labeled images to form a training set, and the remaining N - K images form a test set, where K < N; Use the YOLOv8 object detection algorithm to train and test the labeled file and the labeled images after annotation to obtain an object detection model of the knob switch, denoted as model A.

[0076] Figure 2 It is a schematic diagram of the annotation of the captured image in the detection model of the embodiment of the present invention.

[0077] In this embodiment, the selection of the acquisition distance, lighting conditions, and knob switch models is carried out. Specifically, the distance is divided into five equal parts within the range of 20 cm to 70 cm from the operation surface of the electrical cabinet, and there are 6 image acquisition distances in total; There are 3 different lighting conditions: indoor lighting on during the day, indoor lighting off during the day, and indoor lighting on at night; There are 5 different models of three - gear self - locking knob switches. Therefore, this example consists of N = 3×6×3×5 = 270 images. Randomly select 54 images to form a test set, accounting for 20%, and the remaining 216 images form a training set, accounting for 80%.

[0078] In this embodiment, the annotation process of the three key regions is as follows:

[0079] First, this method uses the labelme annotation software to annotate the training box of the knob switch operating handle. The training box needs to completely cover the knob switch operating handle, and its center coincides with the geometric center of the knob switch operating handle; Annotate the region m of the geometric center of the knob switch operating handle. This region needs to surround the geometric center point of the knob, and the rectangular box covers the area near the center of the knob handle and ensures the symmetry of this region; Annotate the region h where the tip of the indicating arrow of the knob switch operating handle is located, and cover a small area around the tip of the indicating arrow to accurately represent the directivity of the knob; The annotation rectangular box of the arrow key region needs to include the tip and the nearby region of the indicating part of the knob handle; Annotate the tail edge region t of the end of the knob switch operating handle without an arrow, and the rectangular box covers the centrosymmetric region of the end edge of the knob; In addition, this method can also be adapted to support other annotation tools with similar functions.

[0080] Step 2, capture the target image and obtain the pixel point information of the three key regions of the target image.

[0081] Use a depth camera to capture the area where the knob switch is located, and record the obtained image as the target image.

[0082] Use model A to detect the target image. Specifically, model A selects the overall area of the knob switch operating handle in the target image through a detection box 1, and the width of this detection box 1 in the pixel coordinate system of the target image is ω 1, with a height of σ 1 , the pixel coordinates of the upper left corner of the detection box 1 are (u r1 , v r1 ); the pixel coordinates of the arrow key area h, the center key area point m, and the tail key area t of the target image in the pixel coordinate system of the target image are obtained through the detection box 1. Among them, the pixel coordinates of the upper left corner and the lower right corner of the arrow key area h are respectively The pixel coordinates of the upper left corner and the lower right corner of the center key area m are respectively The pixel coordinates of the upper left corner and the lower right corner of the tail key area t are respectively

[0083] Step 3: Traverse the points within the key areas, classify their reliability, and determine the current gear of the knob switch.

[0084] Step 3.1: Use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and record this distance as the operation plane depth value d calib ; Subsequently, all pixel points in the three key areas are traversed point by point within the following coordinate ranges:

[0085] Arrow key area h:

[0086] Center key area m:

[0087] Tail key area t:

[0088] Among them, u i , v i are the pixel coordinates of any pixel point i in the three key areas, and i is the serial number of the pixel point; during the traversal process, record the depth value d i and the pixel coordinates u i , v i ;

[0089] Step 3.2: After completing the traversal of all pixel points in the three key areas in Step 3.1, calculate the depth error Δd i between the depth value d calib of each pixel point and the operation plane depth value d i , and make the following judgment according to the given depth threshold δ:

[0090] If Δd i > δ, then judge that this pixel point is located on the knob surface and is a reliable point;

[0091] If Δd i ≤ δ, then determine that this pixel point is not located on the knob surface and is an unreliable point;

[0092] Step 3.3, discard the unreliable points, and denote the set of reliable points obtained within the arrow key area h as set P h , and denote the set of reliable points obtained within the central key area m as set P m , and denote the set of reliable points obtained within the tail key area t as set P t ;

[0093] Step 3.4, according to the information of the detection frame 1 and the three key areas of the target image, judge the current gear of the knob switch according to the given confirmation criterion: if the current gear of the knob switch is different from the expected gear, go to Step 4; if the current gear of the knob switch is the same as the expected gear, end the current operation.

[0094] Step 4, in the set P h , set P m and set P t obtained in Step 3.3, randomly extract reliable points N sample times respectively. Specifically, each time, extract one reliable point from set P h , set P m and set P t respectively to obtain three reliable points, and form a set of data to be fitted. After N sample times of extraction, N sample sets of data to be fitted are obtained.

[0095] Use the least squares method to perform multi-point fitting on the N sample sets of data to be fitted to obtain the direction line of the operating handle of the knob switch, and at the same time calculate the included angle θ 1 between this direction line and the positive direction of the U-axis of the image, and denote it as the included angle θ 1 of the current gear.

[0096] In this embodiment, the implementation process of using the least squares method to perform multi-point fitting on the N sample sets of data to be fitted is as follows:

[0097] For the three reliable points in each set of data to be fitted, fit them into a straight line using the least squares method, and denote the slope of the fitted straight line as k j , and the intercept as b j ;

[0098] Record the results of each fitting, save the set of all fitting results and denote them as the slope set K and the intercept set B respectively. K = {k 1 , k 2 ,..., k j ,..., k Nsample}, B = {b 1 , b 2 ,..., bj , ..., b Nsample};

[0099] Calculate the average values of the fitting results in the slope set K and the intercept set B respectively, and denote them as the slope k avg and the intercept b avg , and this slope k avg and the intercept b avg are the slope and intercept of the direction line of the operating handle of the knob switch; Denote the angle between this direction line and the positive direction of the U-axis of the image as θ 1 , θ 1 = arctan(k avg ), 0 ≤ θ 1 < π.

[0100] Figure 5 This is the schematic diagram of the straight line fitting of key points in the embodiment of the present invention.

[0101] Step 5, Denote the angle between the operating handle of the expected gear of the knob switch and the positive direction of the U-axis of the image as the expected gear angle θ i , 0 ≤ θ i < π and |θ 1 - θ i | ≤ θ max , where θ max is the maximum allowable rotation amplitude of the current gear angle θ 1 ; Select the rotation direction according to the following rules:

[0102] When θ 1 - θ i ≥ 0, control the clamping device to clamp the operating handle of the knob switch, and rotate the operating handle of the knob switch clockwise by |θ 1 - θ i | to complete the gear adjustment;

[0103] When θ 1 - θ i < 0, control the clamping device to clamp the operating handle of the knob switch, and rotate the operating handle of the knob switch counterclockwise by |θ 1 - θ i | to complete the gear adjustment.

[0104] During the gear adjustment process, adjust the knob switch to the target position through the clamping device and then release it. Before releasing, stabilize the position of the knob handle through the clamping device to ensure that the knob remains at the target position during the release process, and verify the current position of the knob to ensure the completion of the operation.

[0105] Step 6, After completing the gear adjustment, use the depth camera to take a picture of the area where the adjusted knob switch is located again, and denote the image obtained from this shooting as the inspection image.

[0106] The inspection image is detected using the model A described in step 1 to obtain a detection box 2 for the position of the knob switch in the inspection image. The width of the detection box 2 in the pixel coordinate system of the inspection image is ω 2 and the height is σ 2 . The pixel coordinates of the upper left corner of the detection box 2 are (u r2 , v r2 ). Pixel coordinates of the arrow key area h2, the center key area m2, and the tail key area t2 of the inspection image in the pixel coordinate system of the inspection image are obtained through the detection box 2

[0107] Step 7: According to the information of the detection box 2 obtained in step 6 and the three key areas of the inspection image, judge the rotated gear state according to the confirmation criterion described in step 3.4, and calculate the current gear angle θ of the adjusted knob switch according to the detection result of the inspection image new , 0 ≤ θ new < π; then make the following judgment according to the preset angle error threshold ε

[0108] If |θ 1 - θ new | ≤ ε, it is considered that the gear state of the adjusted knob switch is consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear, and return to step 2

[0109] In this embodiment, the knob switch is a three-gear self-locking knob switch, and the gear confirmation criteria for the knob switch in step 3.4 and step 7 are as follows

[0110] First, take the average value of the pixel coordinates of all reliable points in the set P h described in step 3.3, and denote it as . Its calculation formula is

[0111]

[0112] In the formula, M is the number of reliable points in the set P h , u m , v m is the pixel coordinate of any reliable point in the set P h , and m is the serial number of the reliable point in the set P h ;

[0113] Then make the following judgment

[0114] When the arrow key area h satisfies and , it is considered that the current knob switch gear is in the left gear

[0115] When the arrow key area h satisfies and it is considered that the current knob switch gear is in the middle gear;

[0116] When the arrow key area h satisfies and it is considered that the current knob switch gear is in the right gear;

[0117] In the gear confirmation criterion of the knob switch in step 3.4, ω is ω 1 , is σ 1 , u is u r1 , v is v r1 ; in the gear confirmation criterion of the knob switch in step 7, ω is ω 2 , is σ 2 , u is u r2 , v is v r2 .

[0118] Figure 3 This is the schematic diagram for judging the current gear of the knob switch in the embodiment of the present invention.

[0119] In the present invention, the pixel coordinate system is a rectangular coordinate system established with the upper left corner of the image as the origin and in pixels, including mutually perpendicular pixel U-axis and pixel V-axis. The abscissa on the pixel U-axis is the column number of the pixel in its image, and the direction is parallel to the image and to the right. The ordinate on the pixel V-axis is the row number of the pixel in its image, and the direction is perpendicular to the pixel U-axis and downward. The specific situation of the pixel coordinate system can be seen in Figure 2 .

Claims

1. A method for detecting the position and posture of a knob switch based on multi-point fitting, the method involving a system comprising an electrical cabinet and a mobile robotic arm, the operating panel of the electrical cabinet being provided with a knob switch, the knob switch being a multi-position self-locking knob switch, and one end of the operating handle of the knob switch being marked with an indication arrow; the end of the mobile robotic arm being provided with a depth camera, when the depth camera is used to photograph the knob switch, the mobile robotic arm is first moved to the area where the knob switch of the electrical cabinet is located, and after unfolding, the optical axis of the depth camera is kept perpendicular to the operating surface of the electrical cabinet, and the plane where the depth camera lens is located is parallel to the operating surface of the electrical cabinet, and then the depth camera is used to photograph the area where the knob switch is located, and during the photographing process, the knob switch operating handle is ensured to be clearly visible, and the knob switch is located at the center of the image; It is characterized in that The posture detection method comprises the following steps: Step 1: Build a target detection model for the knob switch; Use the depth camera to take N images of the knob switch, and use labelme annotation software to annotate the captured images to obtain three key areas of the knob switch operating handle, which are respectively recorded as arrow key area h, center key area m and tail key area t, where the arrow key area h covers the part where the operating handle indicates the arrow, the center key area m covers the geometric center of the operating handle, and the tail key area t covers the edge of the end of the operating handle; Select K images from the labeled images to form a training set, and the remaining NK images to form a test set, where K < N; train and test the labeled files obtained after the labeling is completed together with the labeled images using the YOLOv8 target detection algorithm to obtain a target detection model for the knob switch, which is recorded as model A; Step 2, photographing the target image and obtaining pixel information of three key areas of the target image; Use a depth camera to capture the area where the knob switch is located, and record the obtained image as the target image; Model A is used to detect the target image. Specifically, model A selects the entire area of ​​the knob switch operating handle in the target image through a detection frame 1. The width of the detection frame 1 in the pixel coordinate system of the target image is ω1 and the height is σ1. The pixel coordinates of the upper left corner of the detection frame 1 are (u r1 , v r1 ); The pixel coordinates of the arrow key area h, the center key area point m and the tail key area t of the target image in the pixel coordinate system of the target image are obtained through the detection frame 1, wherein the pixel coordinates of the upper left corner and the lower right corner of the arrow key area h are respectively The pixel coordinates of the upper left corner and lower right corner of the central key area m are The pixel coordinates of the upper left corner and lower right corner of the tail key area t are Step 3, traverse the points in the key area and classify their reliability, and determine the current gear position of the knob switch; Step 3.1: Use the depth camera to obtain the straight-line distance between the depth camera plane and the electrical cabinet plane, and record the distance as the operating plane depth value d calib ; Then all the pixels in the three key areas are traversed point by point according to the following coordinate ranges: Arrow key area h: Central key area m: Tail key area t: Among them, u i , v i is the pixel coordinate of any pixel i in the three key areas, i is the serial number of the pixel; during the traversal process, the depth value d of each pixel i traversed is recorded i and pixel coordinates u i , v i ; Step 3.2, after completing the traversal of all pixels in the three key areas in step 3.1, calculate the depth value d of each pixel i and the depth value d of the operating plane calib The depth error Δd i , and make the following judgment based on the given depth threshold δ: If Δd i >δ, then the pixel is judged to be located on the knob surface and is a reliable point; If Δd i ≤δ, then the pixel is judged not to be located on the knob surface and is an unreliable point; Step 3.3, discard unreliable points and record the set of reliable points obtained in the arrow key area h as set P h , the set of reliable points obtained in the central key area m is recorded as set P m , the set of reliable points obtained in the tail key area t is recorded as set P t ; Step 3.4, based on the information of the detection frame 1 and the three key areas of the target image, the current gear position of the knob switch is judged according to the given confirmation criteria: if the current gear position of the knob switch is different from the expected gear position, go to step 4; if the current gear position of the knob switch is the same as the expected gear position, end this operation; Step 4: Get the set P in step 3.3 h , set P m and set P t In the sample The random sampling of the reliable points is, specifically, each time from the set P h , set P m and set P t One reliable point is extracted from each of them to obtain three reliable points, and they form a set of data to be fitted, N sample Get N times of drawing sample Set the data to be fitted; To N sample The least square method is used to perform multi-point fitting on the data to be fitted, and the direction line of the knob switch operating handle is obtained. At the same time, the angle θ1 between the direction line and the positive direction of the image U axis is calculated and recorded as the current gear angle θ1; Step 5: record the angle between the operating handle of the desired gear position of the knob switch and the positive direction of the image U axis as the desired gear position angle θ i , 0≤θ i <π and |θ1-θ i |≤θ max , where θ max is the maximum allowable rotation range of the current gear angle θ1; the rotation direction is selected according to the following rules: When θ1-θ i ≥0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle clockwise |θ1-θ i |To complete the gear adjustment; When θ1-θ i <0, control the clamping device to clamp the knob switch operating handle and rotate the knob switch operating handle counterclockwise |θ1-θ i |To complete the gear adjustment; Step 6: After the gear adjustment is completed, the depth camera is used again to photograph the area where the adjusted knob switch is located, and the image obtained by this shooting is recorded as the inspection image; The inspection image is inspected using the model A described in step 1 to obtain a detection frame 2 at the location of the knob switch in the inspection image. The width of the detection frame 2 in the pixel coordinate system of the inspection image is ω2 and the height is σ2. The pixel coordinates of the upper left corner of the detection frame 2 are (u r2 , v r2 ); The pixel coordinates of the arrow key area h2, the center key area m2 and the tail key area t2 of the inspection image in the pixel coordinate system of the inspection image are obtained through the detection frame 2; Step 7: Based on the information of the detection frame 2 and the three key areas of the inspection image obtained in step 6, the gear state after rotation is judged according to the confirmation criteria described in step 3.4, and the current gear angle θ of the knob switch after adjustment is calculated based on the detection result of the inspection image. new , 0≤θ new <π; then make the following judgment based on the preset angle error threshold ε: If |θ1-θ new |≤ε, it is considered that the gear state of the adjusted knob switch is consistent with the expected gear state, and the expected operation of the knob switch is completed; otherwise, it is considered that the knob switch has not reached the expected gear state, and return to step 2.

2. According to the method for detecting the position and posture of a knob switch based on multi-point fitting according to claim 1, it is characterized in that: The knob switch is a three-position self-locking knob switch. The position confirmation criteria of the knob switch in step 3.4 and step 7 are: First, take the set P described in step 3.3 h The average value of the pixel coordinates of all reliable points in is recorded as The calculation formula is: Where M is the set P h The number of reliable points in , u m , v m For the set P h The pixel coordinates of any reliable point in the set P h The serial number of the reliable point; Then make the following judgment: When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the left position; When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the middle position; When the arrow key area h satisfies and When , it is considered that the current knob switch position is in the right position; In the position confirmation criteria of the knob switch in step 3.4, ω is ω1, is σ1, u is u r1 , v is v r1 In the position confirmation criteria of the knob switch in step 7, ω is ω2, is σ2, u is u r2 , v is v r2 .

3. The method for detecting the position and posture of a knob switch based on multi-point fitting according to claim 1, characterized in that: Step 4 describes N sample The process of using the least squares method to perform multi-point fitting on the data to be fitted is as follows: For each set of data to be fitted, three reliable points are fitted into a straight line using the least squares method. The slope of the fitted straight line is k. j , the intercept is b j ; To N sample The results of the fitting are recorded, and the set of all fitting results is saved and recorded as the slope set K and the intercept set B, K = {k1, k2, ..., k j , ..., k Nsample }, B = {b1, b2, ..., b j , ..., b Nsample }; The fitting results in the slope set K and the intercept set B are averaged and recorded as slope k avg and intercept b avg , the slope k avg and intercept b avg The slope and intercept of the direction line of the knob switch operating handle are respectively the slope and intercept of the direction line of the knob switch operating handle. The angle between the direction line and the positive direction of the image U axis is θ1, θ1 = arctan(k avg ), 0≤θ1<π.

Images