Calibration-free visual guidance alignment method based on multiple ArUco codes

Abstract

The invention provides a calibration-free visual guidance alignment method based on multiple ArUco codes, and belongs to the technical field of image processing, and the method comprises the steps: installing an ArUco code positioning module on an operation target; the positioning module deploys the number, position and angle of ArUco codes according to a positioning requirement; a mechanical arm is controlled to approach a target through a manual teaching operation method to complete alignment, mechanical arm alignment images are collected in real time through a camera at the tail end of the mechanical arm, and pose information of all joints of the mechanical arm when the mechanical arm alignment images are synchronized; a guiding track is generated through a mechanical arm alignment image collected by a camera and mechanical arm joint pose information; and when the robot performs automatic positioning, the mechanical arm is driven to complete alignment according to the guide track by using the image feature relationship. According to the method, the problem that a complicated calibration process and specific texture or feature identification are needed when workpieces with complicated surface characteristics such as transparent parts, reflective parts and black parts are processed by a traditional positioning method is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a multi-ArUco-code-based calibration-free visual guidance alignment method. Background Art

[0002] With the rapid development of industrial automation, visual alignment technology has become increasingly important in the manufacturing industry. Traditional visual alignment methods, such as hand-eye calibration, rely on complex calibration processes and the recognition of specific textures or features, which have limitations when dealing with workpieces with complex surface characteristics such as transparent parts, reflective parts, and black parts. Due to their surface characteristics, these workpieces often lead to difficulties in pose detection and poor alignment accuracy, affecting the efficiency and stability of the automated production line. To address this challenge, researchers have tried various methods, such as depth simulation methods enhanced by domain randomization to improve the perception ability of objects with high-gloss, transparent, or diffuse reflection materials. However, these methods often require complex simulation datasets and deep learning models, with high computational costs and difficulty in generalizing to the diversity in practical applications.

[0003] ArUco code (multi-binary visual marker), as a widely used binary marker, is widely applied in the field of pose estimation due to its strong robustness and easy detection characteristics. The application of traditional ArUco code (multi-binary visual marker) requires precise camera calibration and complex environmental settings. Summary of the Invention

[0004] Aiming at the above deficiencies in the prior art, the present invention provides a multi-ArUco-code-based calibration-free visual guidance alignment method, aiming to overcome the limitations of the prior art. By innovatively using multiple ArUco codes and combining advanced image processing algorithms, the present invention realizes calibration-free visual alignment, which is applicable to workpieces such as transparent parts and reflective parts with sparse textures and difficult pose detection.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a multi-ArUco-code-based calibration-free visual guidance alignment method, including the following steps:

[0006] Install a positioning module with ArUco codes on the operation target;

[0007] Deploy the positions and angles of ArUco codes on the positioning module according to the positioning requirements;

[0008] Based on the positions and angles of ArUco codes, control the robotic arm to align with the target, and use the camera installed at the end of the robotic arm to collect the robotic arm alignment images in real time;

[0009] Generate an alignment guidance trajectory by using the collected robotic arm alignment images and robotic arm joint information, where the alignment guidance trajectory is the path from the current point to the target point.

[0010] When the robot aligns again, the image feature relationship is used to drive the robotic arm to complete the alignment along the generated alignment guidance trajectory, realizing calibration-free visual guidance alignment based on multiple ArUco codes.

[0011] The beneficial effects of the present invention are as follows: By innovatively using multiple ArUco codes and combining advanced image processing algorithms, the present invention realizes calibration-free visual alignment, which is applicable to workpieces such as transparent parts and reflective parts with sparse textures and difficult pose detection. This method not only improves the alignment accuracy but also reduces the dependence on complex calibration processes, thereby enhancing the efficiency and stability of the automated production line.

[0012] Further, installing a positioning module with ArUco codes on the operation target specifically includes:

[0013] On the operation target, according to the shape, size, and alignment accuracy requirements of the target object, the positioning module determines the positions and arrangement patterns of multiple ArUco codes to complete the installation of the positioning module.

[0014] The beneficial effects of the above further solution are as follows: By accurately determining the positions and arrangement patterns of multiple ArUco codes according to the shape, size, and alignment accuracy requirements of the target object, this solution can enhance the adaptability and versatility of the system, enabling the positioning module to adapt to the characteristics of diverse workpieces, especially showing superior stability and accuracy on workpieces with irregular sizes or complex surfaces, thereby improving the alignment efficiency and robustness.

[0015] Still further, the conditions for deploying the positions and angles of ArUco codes are:

[0016] The positions and angles of ArUco codes need to satisfy that the camera can continuously detect multiple ArUco codes at different positions.

[0017] The beneficial effects of the above further solution are as follows: This solution can ensure that the camera can detect multiple ArUco codes at different angles and positions, enhancing the continuity and reliability of visual guidance, enabling the system to effectively resist detection difficulties caused by workpiece occlusion, reflection, or angle deviation during the alignment task, thereby significantly improving the alignment accuracy and the robustness of task execution.

[0018] Still further, based on the positions and angles of ArUco codes, controlling the robotic arm to align with the target and using the camera installed at the end of the robotic arm to collect real-time alignment images of the robotic arm specifically includes:

[0019] Based on the position and angle of the ArUco code, control the robotic arm to approach the target area. During the control process, keep the camera facing the target surface containing multiple ArUco codes to ensure that the camera captures the alignment image of the robotic arm.

[0020] When the robotic arm approaches the target area, continuously collect video frames using the end camera, and perform grayscale conversion and filtering denoising on the alignment image of the robotic arm in each video frame to enhance the edges of the ArUco code. Among them, in each video frame, detect and identify the ArUco code.

[0021] Use the ArUco library to extract the four vertex features of each ArUco code. Among them, detect n ArUco codes in the alignment image detection of the robotic arm, and the four vertex coordinates of each ArUco code are i represents the number of each code.

[0022] Match the feature points in multiple alignment image frames of the robotic arm with the IDs of each ArUco code to ensure that the same ArUco code is consistently recognized in different frames, and extract the center point of each ArUco code To be used as a reference point for positioning, and complete the real-time acquisition of the image.

[0023] The beneficial effect of the above further solution is that this solution can efficiently extract the feature points of the ArUco code and achieve stable and fast target positioning by optimizing the orientation of the camera, continuously collecting video frames, and processing images. At the same time, combined with the grayscale conversion and filtering denoising processing methods, it effectively improves the accuracy and stability of ArUco code detection on the surface of complex workpieces, thereby reducing the alignment error caused by environmental interference or visual noise.

[0024] Furthermore, the center point has the following expression:

[0025]

[0026] The beneficial effect of the above further solution is that this solution helps to simplify the feature extraction process of visual alignment and improve the accuracy of the positioning reference point by calculating and extracting the center point of each ArUco code as the positioning reference point.

[0027] Furthermore, generate an alignment guidance trajectory using the collected alignment image of the robotic arm and the robotic arm joint information, specifically:

[0028] According to the collected alignment image of the robotic arm, obtain the rotation matrix R i and displacement vector t i ;

[0029] Convert the rotation matrix R of each ArUco code i and the displacement vector t i to the coordinate system of the robotic arm

[0030] Let the initial position of the robotic arm be P start and set the target point position to P goal , and based on the conversion result, generate a trajectory from the initial position to the target position through multi-point path planning;

[0031] According to the trajectory from the initial position to the target position, use B-spline to generate a smooth trajectory to complete the generation of the alignment guidance trajectory.

[0032] The beneficial effect of the above further solution is: By combining the joint information of the robotic arm with the alignment image collected by the camera, this solution uses the PnP algorithm ((Perspective-n-Point), a method for solving the correspondence between three-dimensional points and two-dimensional points) to obtain the poses of multiple ArUco codes and generate a smooth and optimal alignment guidance trajectory, significantly reducing the computational complexity in path planning, while improving the smoothness of the robotic arm movement and the accuracy of alignment, and is applicable to high-precision operation tasks in dynamic or complex environments.

[0033] Furthermore, the rotation matrix R i and the displacement vector t i are expressed as follows:

[0034] R i ,t i =PnP(P world ,P image ,K,D)

[0035] where P world represents the known marker world coordinates, P image represents the detected coordinates in the alignment image of the robotic arm, K represents the camera internal parameter matrix, and D represents the distortion parameter.

[0036] The beneficial effect of the above further solution is: By clarifying the mathematical relationship between the marker world coordinates, the detected image coordinates, and the camera internal parameters and distortion parameters, this solution realizes the precise coordinate conversion between the robotic arm and the target position, providing a highly accurate mathematical basis for visual guidance, thereby effectively improving the overall accuracy and stability of the alignment system.

[0037] Furthermore, the coordinate system is expressed as follows:

[0038]

[0039] Among them, R cam_arm represents a rotation matrix, and t cam_arm represents a displacement.

[0040] The beneficial effect of the above further solution is that through the transformation of the rotation matrix and the displacement vector, the conversion from the ArUco code coordinate system to the robotic arm coordinate system is accurately completed, providing the necessary data support for multi-sensor information fusion and path planning of complex tasks, thereby further improving the alignment accuracy and the flexibility of task execution.

[0041] Furthermore, when the robot aligns again, the robotic arm is driven along the generated alignment guidance trajectory by using the image feature relationship to accurately complete the alignment, realizing calibration-free vision-guided alignment based on multiple ArUco codes. Specifically:

[0042] When the robot aligns again, the camera is used to capture the robotic arm alignment image in real time, and by detecting the feature point positions of the ArUco codes, the center point coordinates of each ArUco code in the current robotic arm alignment image are calculated

[0043]

[0044] The current center point coordinates are compared with the center point coordinates ;

[0045] According to the comparison result, the displacement deviation between the current feature point coordinates and the reference coordinates is calculated. Among them, this displacement deviation represents the offset of the robotic arm end from the target point and is used to control the correction of the robotic arm movement direction and speed; during this process, the robotic arm moves according to the alignment guidance trajectory generated during the initial alignment; the alignment guidance trajectory is a smooth path obtained through optimization experiments based on the path planning result between the initial position and the target position of the robotic arm;

[0046] By collecting the camera images in real time during the movement of the robot and combining the currently calculated displacement deviation to adjust the movement state, it is ensured that the robotic arm approaches the target point along the alignment guidance trajectory;

[0047] Through multiple experiments, the path with the smallest displacement deviation is gradually obtained and used as the optimal path. Based on the optimal path as a benchmark and combined with the real-time feedback of the robotic arm alignment image, dynamic adjustment is completed, enabling the robotic arm to approach the target with the optimal path at different initial positions, realizing high-precision calibration-free vision-guided alignment based on multiple ArUco codes. Brief Description of the Drawings

[0048] Figure 1 is the method flow chart of the present invention.

[0049] Figure 2It is a specific structure diagram of the dedicated auxiliary positioning module for ArUco codes.

[0050] Figure 3 It is a structure diagram for ArUco code recognition and robotic arm trajectory.

[0051] Figure 4 It is a structure diagram for the optimal trajectory of the robotic arm. Specific implementation manners

[0052] The following describes the specific implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0053] Embodiment

[0054] As Figure 1 shown, the present invention provides a multi-ArUco-code calibration-free vision-guided alignment method, and its implementation method is as follows:

[0055] S1. Install a positioning module with ArUco codes on the operation target, and its implementation method is as follows:

[0056] As Figure 2 shown, on the operation target, the positioning module determines the positions and arrangement modes of multiple ArUco codes according to the shape, size and alignment accuracy requirements of the target object, and completes the installation of the positioning module;

[0057] S2. Deploy the positions and angles of the ArUco codes on the positioning module according to the positioning requirements. The conditions for deploying the positions and angles of the ArUco codes are:

[0058] The positions and angles of the ArUco codes need to satisfy that the camera can continuously detect multiple ArUco codes at different positions.

[0059] S3. Based on the positions and angles of the ArUco codes, control the robotic arm to align with the target, and use the camera installed at the end of the robotic arm to collect the robotic arm alignment images in real time, and its implementation method is as follows:

[0060] Based on the positions and angles of the ArUco codes, control the robotic arm to approach the target area. During the control process, keep the camera facing the target surface containing multiple ArUco codes to ensure that the camera captures the robotic arm alignment images;

[0061] When the robotic arm approaches the target area, continuously collect video frames using the end camera, and perform grayscale conversion and filtering denoising on the robotic arm alignment images of each video frame to enhance the edges of the ArUco codes. Among them, in each video frame, detect and identify the ArUco codes;

[0062] Use the ArUco library to extract the four vertex features of each ArUco code. Among them, detect n ArUco codes in the robotic arm alignment image detection, and the four vertex coordinates of each ArUco code are i represents the number of each code;

[0063] Match the feature points in multiple robotic arm alignment image frames with the IDs of each ArUco code to ensure that the same ArUco code is consistently recognized in different frames, and extract the center point of each ArUco code To be used as a reference point for positioning and complete the real-time acquisition of images.

[0064] In this embodiment, control the robotic arm to align with the target through the manual teaching method, and use the camera installed at the end of the robotic arm to collect images in real time.

[0065] In this embodiment, manually control the robotic arm to gradually approach the target area. During the control process, keep the camera always facing the target surface containing multiple ArUco codes to ensure that the camera can capture clear ArUco code images. When the robotic arm approaches the target area, the end camera continuously collects video frames. In each video frame, use image processing algorithms to detect and identify the ArUco codes. To improve the recognition accuracy, perform grayscale conversion and filtering denoising on each frame of the image to enhance the edges of the ArUco codes.

[0066] In this embodiment, apply Gaussian filtering to reduce noise and enhance edges. The filtering formula is as follows:

[0067]

[0068] where I filtered(x,y) represents the pixel value of the finally filtered image, j represents the offset in the vertical direction (y direction), i represents the offset in the horizontal direction (x direction), k represents the boundary length of the filter kernel, I(x, y) represents the pixel value of the original image, and G(i, j) represents the Gaussian kernel.

[0069] In this embodiment, use the ArUco library to extract the four vertex features of each ArUco code. n ArUco codes are detected in the robotic arm alignment image, and the four vertex coordinates of each ArUco code are Match the feature points in multiple image frames with each ArUco code ID to ensure consistent recognition of the same ArUco code in different frames, and extract the center point of each ArUco code Use it as the base point for positioning. The calculation formula for the center point is:

[0070]

[0071] S4. Utilize the collected arm alignment images and arm joint information to generate an alignment guidance trajectory. Among them, the alignment guidance trajectory is the path from the current point to the target point, and its implementation method is as follows:

[0072] According to the collected arm alignment images, combined with the camera internal parameters and the PnP (Perspective-n-Point) method, estimate the pose of the ArUco code relative to the camera, and obtain the rotation matrix R of each ArUco code i and the displacement vector t i ;

[0073] Convert the rotation matrix R of each ArUco code i and the displacement vector t i to the coordinate system of the robotic arm

[0074] Let the initial position of the robotic arm be P start and set the target point position to P goal , and generate a trajectory from the initial position to the target position through a multi-point path;

[0075] According to the trajectory from the initial position to the target position, use B-spline to generate a smooth trajectory to complete the generation of the alignment guidance trajectory.

[0076] In this embodiment, as Figure 3 shown, through the camera internal parameters and the PnP (Perspective-n-Point) method, estimate the pose of the ArUco code relative to the camera, that is, obtain the rotation matrix R of each ArUco code i and the displacement vector t i . The formula is as follows:

[0077] R i ,t i =PnP(P world ,P image ,K,D)

[0078] where, P world represents the known marker world coordinates, P image represents the detected coordinates in the arm alignment image, K represents the camera internal parameter matrix, and D represents the distortion parameter.

[0079] Convert the rotation matrix R of each ArUco code i and the displacement vector t i to the coordinate system of the robotic arm. Assume the fixed relationship between the camera and the robotic arm is the rotation matrix R cam_arm and the displacement t cam_arm . Then the position of the ArUco code in the robotic arm coordinate system is:

[0080]

[0081] where R cam_arm represents the rotation matrix and t cam_arm represents the displacement.

[0082] In this embodiment, assume the initial position of the robotic arm is P start , and the target point position is P goal . Generate a trajectory from the initial position to the target position through multi-point paths. Use B-spline to generate a smooth trajectory to reduce mutations on the path and improve the motion stability of the robotic arm. The B-spline formula is as follows:

[0083]

[0084] where P(t) represents the point on the B-spline curve at parameter t, i represents the index variable used to traverse the number of the control point P i , n represents the total number of control points, N i,k (t) represents the B-spline basis function, P i represents the control point, and t represents the parameter.

[0085] S5. When the robot aligns again, use the image feature relationship to drive the robotic arm to complete the alignment along the generated alignment guidance trajectory, realizing markerless vision-guided alignment based on multiple ArUco codes. The implementation method is as follows:

[0086] When the robot aligns again, use the camera to capture the robotic arm alignment image in real time, and calculate the center point coordinates of each ArUco code in the current robotic arm alignment image by detecting the feature point positions of the ArUco codes

[0087]

[0088] Compare the current center point coordinates with the center point coordinates ;

[0089] According to the comparison result, calculate the displacement deviation between the current feature point coordinates and the reference coordinates. Among them, this displacement deviation represents the offset between the end of the robotic arm and the target point, and is used to control the correction of the movement direction and speed of the robotic arm; during this process, the robotic arm moves according to the alignment guidance trajectory generated by the initial alignment. The alignment guidance trajectory is a smooth path obtained through optimization experiments based on the path planning result between the initial position and the target position of the robotic arm, and has high repeatability and accuracy.

[0090] During the movement of the robot, the camera images are collected in real time, and the movement state is adjusted by combining the currently calculated displacement deviation to ensure that the robotic arm approaches the target point along the alignment guidance trajectory. If the deviation is large, the path is readjusted according to the position of the feature points detected by the camera to reduce the offset between the robotic arm and the target point, thereby optimizing the alignment accuracy.

[0091] Through multiple experiments, gradually obtain the path with the smallest displacement deviation as the optimal path. Taking the optimal path as the benchmark, the robotic arm uses this path as the benchmark in subsequent alignment operations and combines real-time image feedback to complete dynamic adjustment, enabling the robotic arm to approach the target with the optimal path at different initial positions, realizing high-precision calibration-free visual guidance alignment based on multiple ArUco codes.

[0092] In this embodiment, the trajectory accurately completes the alignment, which means that no matter where the robotic arm starts, it will finally find the optimal path.

[0093] In this embodiment, as Figure 4 shown, when realigning, the camera captures the robotic arm alignment image in real time and detects the position of the feature points of the ArUco code. Calculate the center point coordinates of each ArUco code in the current image and compare them with the above-recorded center points. Calculate the displacement deviation Δx i , Δy i : Among them, this deviation value represents the offset between the end of the robotic arm and the target point, and is used to control the correction of the movement direction and speed of the robotic arm.

[0094] Find the path with the smallest deviation through multiple experiments, that is, the optimal path. During each operation, use the optimal path as the benchmark to guide the robotic arm to approach the target, ensuring that the robotic arm can approach the target with the optimal path at different initial positions.

Claims

1. A multi-ArUco-code-based vision-guided alignment method without calibration, characterized in that It includes the following steps: Install a positioning module with ArUco codes on the operation target; Deploy the positions and angles of the ArUco codes on the positioning module according to the positioning requirements; Based on the positions and angles of the ArUco codes, control the robotic arm to align with the target, and use the camera installed at the end of the robotic arm to collect the robotic arm alignment images in real time; Generate an alignment guidance trajectory by using the collected robotic arm alignment images and the robotic arm joint information, where the alignment guidance trajectory is the path from the current point to the target point; When the robot aligns again, use the image feature relationship to drive the robotic arm to complete the alignment along the generated alignment guidance trajectory, realizing calibration-free visual guidance alignment based on multiple ArUco codes.

2. The visual guidance alignment method based on multiple ArUco codes without calibration according to claim 1, wherein The installation of the positioning module with ArUco codes on the operation target is specifically as follows: On the operation target, the positioning module determines the positions and arrangement modes of multiple ArUco codes according to the shape, size and alignment accuracy requirements of the target object, and completes the installation of the positioning module.

3. The multi-ArUco-code-based vision-guided alignment method without calibration according to claim 1, wherein The conditions for deploying the positions and angles of the ArUco codes are as follows: The positions and angles of the ArUco codes need to satisfy that the camera can continuously detect multiple ArUco codes at different positions.

4. The visual guidance alignment method based on multiple ArUco codes without calibration according to claim 1, wherein Based on the positions and angles of the ArUco codes, control the robotic arm to align with the target, and use the camera installed at the end of the robotic arm to collect the robotic arm alignment images in real time, which is specifically as follows: Based on the positions and angles of the ArUco codes, control the robotic arm to approach the target area. During the control process, keep the camera facing the target surface containing multiple ArUco codes to ensure that the camera captures the robotic arm alignment images; When the robotic arm approaches the target area, continuously collect video frames by using the end camera, and perform grayscale conversion and filtering denoising processing on the robotic arm alignment images of each video frame to enhance the edges of the ArUco codes. In each video frame, detect and identify the ArUco codes; Extract the four vertex features of each ArUco code using the ArUco library. Among them, n ArUco codes are detected in the image detection of the robotic arm alignment, and the four vertex coordinates of each ArUco code are i represents the number of each code; Match the feature points in the multi-robot alignment image frames with each ArUco code ID to ensure that the same ArUco code is consistently recognized in different frames, and extract the center point of each ArUco code to be used as a reference point for positioning and complete the real-time acquisition of images.

5. The visual guidance alignment method based on multiple ArUco codes without calibration according to claim 4, wherein The center point has the following expression:

6. The method for vision-guided alignment based on multiple ArUco codes without calibration according to claim 1, characterized in that Generate an alignment guidance trajectory by using the collected robotic arm alignment images and the robotic arm joint information, which is specifically as follows: According to the collected robot arm alignment image, the rotation matrix R of each ArUco code is obtained by estimating the posture of the ArUco code relative to the camera. i and displacement vector t i ; Convert the rotation matrix R of each ArUco code i and the displacement vector t i to the coordinate system of the robotic arm Set the initial position of the robotic arm as P start and set the target point position as P goal , and based on the conversion result, generate a trajectory from the initial position to the target position through a multi-point path; According to the trajectory from the initial position to the target position, use B-spline to generate a smooth trajectory to complete the generation of the alignment guidance trajectory.

7. The multi-ArUco-code calibration-free vision-guided alignment method according to claim 6, characterized in that The rotation matrix R i and the displacement vector t i are expressed as follows: R i ,t i = PnP(P world ,P image ,K,D) where P world represents the known marked world coordinates, P image represents the detected coordinates in the robot arm alignment image, K represents the camera internal parameter matrix, and D represents the distortion parameter.

8. The method for vision-guided alignment based on multiple ArUco codes without calibration according to claim 6, wherein The coordinate system has the following expression: where R cam_arm represents a rotation matrix, and t cam_arm represents a displacement.

9. The method for visual guidance alignment based on multiple ArUco codes without calibration according to claim 4, characterized in that, When the robot aligns again, use the image feature relationship to drive the robotic arm to complete the alignment along the generated alignment guidance trajectory, realizing calibration-free visual guidance alignment based on multiple ArUco codes, which is specifically as follows: When the robot aligns again, the camera is used to capture the alignment image of the robotic arm in real time, and by detecting the feature point positions of the ArUco codes, the center point coordinates of each ArUco code in the current alignment image of the robotic arm are calculated Compare the current center point coordinates with the center point coordinates for comparison; According to the comparison result, calculate the displacement deviation between the current feature point coordinates and the reference coordinates. The displacement deviation represents the offset of the end of the robotic arm from the target point and is used to control the correction of the movement direction and speed of the robotic arm. During this process, the robotic arm moves according to the alignment guidance trajectory generated during the initial alignment. The alignment guidance trajectory is a smooth path obtained through optimizing experiments based on the path planning result between the initial position and the target position of the robotic arm; During the movement of the robot, collect camera images in real time, and adjust the movement state in combination with the currently calculated displacement deviation to ensure that the robotic arm approaches the target point along the alignment guidance trajectory. Through multiple experiments, gradually obtain the path with the smallest displacement deviation as the optimal path, and based on the optimal path as the benchmark, complete dynamic adjustment by combining the real-time image feedback of the robotic arm alignment, so that the robotic arm can approach the target with the optimal path at different initial positions, realizing high-precision vision-guided alignment based on multiple ArUco codes without calibration.

Images