Ceramic tile laying control method and system based on pose estimation
By using the PnP algorithm to calculate the tiling position and RRT algorithm planning path on the tiling laying robot, the problems of low efficiency and quality of traditional tiling laying are solved, and efficient and accurate tiling laying is achieved.
Patent Information
- Application Number
- CN202411496331.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-10-25
AI Technical Summary
Traditional ceramic tile laying methods are inefficient and prone to quality problems. How to improve the efficiency of ceramic tile laying of tiles and reduce position errors.
The tile image is collected by the camera set on the robotic arm, the tile position is calculated using the PnP algorithm, and the robotic arm movement path is planned in combination with the RRT algorithm to ensure that the tile reaches the target position safely and effectively.
The tiles laying efficiency of the tiles laying robot is improved, the tiles laying position error is reduced, and an automated tiles processing process is realized.
Smart Images

Figure CN119017393A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of tile laying automation, and more particularly to a tile laying control method and system based on posture estimation. Background Art
[0002] In recent years, with the rapid development of the real estate industry, tile laying is indispensable. However, the traditional tile laying method mainly relies on manual labor, which is not only inefficient but also prone to quality problems. Therefore, in order to improve the efficiency of tile laying and meet market requirements, it is inevitable that tile laying robots will replace manual labor for tile laying.
[0003] In the working process of the tile laying robot, the control process of the robot arm is particularly important. In order to better optimize the control process of the robot arm, rigorous path planning and posture estimation are necessary. Therefore, how to solve the problem of motion control error of the robot arm during tile laying, improve the tile laying efficiency of the tile laying robot, and reduce the position error are problems that technicians in this field need to solve urgently. Summary of the invention
[0004] In view of this, the present invention provides a tile laying control method and system based on posture estimation.
[0005] In order to achieve the above object, the present invention provides the following technical solutions: A tile laying control method based on posture estimation comprises the following steps: Step 1: Capture the current image of the tiles to be laid by using a camera set on the mechanical arm of the tile laying robot; Step 2: performing target detection on the current image of the tiles to be laid, and after identifying the tiles to be laid, calculating the current posture of the tiles to be laid; Step 3: Get the current state of the robot arm, including the angles of each joint and the position of the end effector; Step 4: Determine the target position of the tiles to be laid according to the position of the tiles and the size of the tile joints; Step 5: Plan the movement path of the robotic arm according to the current posture of the tile to be laid, the current state of the robotic arm, and the target position of the tile to be laid; Step 6: Control the robot arm to grab, move and place the tiles to be laid according to the moving path.
[0006] Optionally, in step 2, calculating the current posture of the tile to be laid includes the following steps: Step 2.1, extracting feature points of the tiles to be laid; Step 2.2: Calibrate the camera to obtain camera parameters, including intrinsic and extrinsic parameters; Step 2.3: According to the camera parameters and the feature points of the tiles to be laid, the current posture of the tiles to be laid is calculated according to the PnP algorithm.
[0007] Optionally, in step 2.1, feature points of the tiles to be laid are extracted by Harris corner point detection, and the feature points include corner points, edges or texture features of the tiles to be laid.
[0008] Optionally, in step 2.2, the intrinsic parameters of the camera include focal length, principal point coordinates, and distortion coefficients, and the extrinsic parameters include rotation matrix and translation vector.
[0009] Optionally, in step 2.2, when calibrating the camera, the Zhang Zhengyou calibration method is used to calculate the intrinsic and extrinsic parameters of the camera.
[0010] Optionally, in step 4, the method for determining the target position of the tiles to be laid is: Step 4.1, define the reference coordinate system: define a global reference coordinate system, which corresponds to the working plane for laying tiles; Step 4.2, determine the position of the tiles: measure or calculate the position and posture of each tile in the reference coordinate system, that is, the coordinates of the center point and the rotation angle of each tile; Step 4.3, calculate the target position: According to the size of the tiles, the preset paving pattern and the size of the brick joints, calculate the center point distance between adjacent tiles to obtain the target position that the end effector needs to reach.
[0011] Optionally, in step 5, the RRT algorithm is used to plan the movement path of the robot arm, and the specific method is: Step 5.1, Initialize the configuration space: Select the configuration space according to the working environment and map information SLAM of the robot arm; Step 5.2, select the starting point and target point: according to the current posture of the tile to be laid and the target position of the tile to be laid, clarify the starting position and target position of the robot arm in the configuration space; Step 5.3, construct RRT: take the starting position as the root node, and generate a random expansion tree by adding leaf nodes through random sampling; in each iteration, select the tree node closest to the random sampling point, expand to the selected random sampling point, form a new node, and ensure that the new node does not collide with obstacles; Step 5.4, path optimization: When the leaf node of the random tree enters the target area, the initially found path is optimized; Step 5.5, apply inverse kinematics: After the RRT algorithm finds the path, use inverse kinematics to convert each node on the path into the joint angle of the robot arm; Step 5.6, generate joint trajectory: perform kinematic inverse solution on each node on the path to obtain the joint state of the robot, and then use the interpolation method to generate a set of joint state sequences of the robot on the path. These state sequences constitute the joint trajectory of the robot, that is, the moving path.
[0012] A tile laying control system based on posture estimation, comprising: An image acquisition module is used to acquire a current image of the tiles to be laid through a camera arranged on the mechanical arm of the tile laying robot; A tile posture calculation module is used to perform target detection on the current image of the tile to be laid, and after identifying the tile to be laid, calculate the current posture of the tile to be laid; The robot arm state acquisition module is used to obtain the current state of the robot arm, including the angles of each joint and the position and posture of the end effector; A target position determination module is used to determine the target position of the tiles to be laid according to the position of the tiles and the size of the tile joints; A path planning module is used to plan the movement path of the robot arm according to the current posture of the tile to be laid, the current state of the robot arm, and the target position of the tile to be laid; The execution module is used to control the robot arm to grasp, move and place the tiles to be laid according to the moving path.
[0013] It can be seen from the above technical solutions that the present invention provides a method and system for controlling tile laying based on posture estimation, which has the following beneficial effects compared with the prior art: The present invention combines the PnP algorithm, feature point extraction and other methods to obtain the current posture of the tiles to be laid, and at the same time obtains the current state of the robotic arm and calculates the target position of the tiles to be laid. The RRT algorithm is used to plan the movement path of the robotic arm to ensure that the tiles to be laid reach the target position safely and effectively. This scheme effectively improves the tile laying efficiency of the tile-laying robot and reduces the tile laying position error.
[0014] The present invention integrates all components into one system to ensure seamless collaboration of image recognition, pose estimation, path planning and motion control to achieve an automated tile processing flow. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0016] Figure 1 It is a schematic diagram of the method steps of the present invention. DETAILED DESCRIPTION
[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0018] The embodiment of the present invention discloses a tile laying control method based on posture estimation, see Figure 1 , including the following steps: Step 1: Capture the current image of the tiles to be laid by using a camera set on the mechanical arm of the tile-laying robot.
[0019] After acquiring the image, it is usually necessary to preprocess the image, including filtering, edge detection, morphological operations and other steps.
[0020] Step 2: Use a deep learning detection algorithm to perform target detection on the current image of the tiles to be laid. After identifying the tiles to be laid, estimate the current position of the tiles to be laid: Step 2.1: Extract feature points of the tiles to be laid by Harris corner point detection, wherein the feature points include corner points, edge or texture features of the tiles to be laid.
[0021] Step 2.2: In order to map the 2D feature points in the image into the three-dimensional space, the camera used needs to be calibrated to obtain the camera parameters, including intrinsic and extrinsic parameters; the camera's intrinsic parameters include focal length, principal point coordinates, distortion coefficient, etc., and the extrinsic parameters include rotation matrix and translation vector, which describe the position and posture of the camera in the world coordinate system.
[0022] Use the camera calibration algorithm Zhang Zhengyou calibration method to calibrate the camera, which includes the following operations: (1) Prepare the calibration plate: Select a calibration plate with obvious features, such as a chessboard, and ensure that the spacing between the feature points of the calibration plate is suitable for the field of view of the camera.
[0023] (2) Capture calibration images: Place the calibration plate in front of the camera and capture the calibration plate from multiple angles and distances to obtain calibration images to cover the entire field of view of the camera.
[0024] (3) Extracting feature points of the calibration image: Use computer vision algorithms to process the calibration image, identify and extract feature points on the calibration plate. For example, for a checkerboard calibration plate, it is necessary to detect the corner points of the checkerboard.
[0025] (4) Calculate camera parameters: Based on the extracted feature points and the known geometric information of the calibration plate, the camera calibration algorithm Zhang Zhengyou calibration method is used to calculate the camera's intrinsic and extrinsic parameters.
[0026] 1) Calculation method of internal reference: Taking the checkerboard calibration plate as an example, according to the feature points of the calibration plate extracted above, one corner point of the checkerboard calibration plate is located at the origin of the world coordinate system, and the other corner points are distributed at specific three-dimensional coordinates according to the size and shape of the checkerboard calibration plate. The coordinates of the corner points on the checkerboard calibration plate in the 3D world coordinate system are known. Let the size of each grid of the checkerboard calibration plate be Sx × Sy , and the first corner point is located at the origin of the world coordinate system, then i The 3D coordinates of the corner points are expressed as: ; in, i x and i y They are i The corner point is x and y The grid index in the direction.
[0027] On the 2D image plane, the corresponding corner pixel coordinates are: ; in, u i is the horizontal axis, v i is the vertical coordinate; The camera's intrinsic parameter matrix K includes parameters such as focal length and principal point coordinates, which can be expressed as: ; in, f x and f y Along the image x Axis and y The focal length of the axis, c x and c y are the pixel coordinates of the principal point.
[0028] Through the perspective projection model, the points in the 3D world coordinate system are projected onto the 2D image plane: ; Here, λ is the scaling factor.
[0029] Substituting the internal parameter matrix K into the above equation, we get: ; It can be expanded into two equations: ; ; Since the calibration plate is on the plane of Z=0, the Z coordinate can be ignored and the above equation can be simplified to: ; ; Establish a set of equations: According to the extracted corner point coordinates and the known calibration plate geometry information, establish a set of linear equations. Then for each corner point, there is a set of equations like this: ; This matrix equation can be expressed as: U=Ak; Where U is a column vector of known 2D image coordinates, A is a matrix constructed from known 3D world coordinates, and k is a column vector containing intrinsic parameters.
[0030] Solve the above equations by mathematical methods such as the least squares method to obtain the initial value of the internal parameter matrix: .
[0031] 2) Calculation method of external parameters: Use the findChessboardCorners and cornerSubPix functions in the computer vision library OpenCV to obtain the coordinates of the corner points on the image. Use the solvePnP function to solve the camera external parameters, and solve the camera's rotation matrix and translation vector by minimizing the reprojection error between the 3D points (the 3D coordinates of the chessboard corner points) and their projection points on the 2D image plane. Specifically: According to the camera parameters, a projection model is established to describe the relationship between a 3D space point and its projection point on the 2D image; Collect at least 3 (and at least one additional verification point) known 3D space points and their locations on the 2D image; The DLT direct linear transformation method in the PnP algorithm is used to calculate the camera posture, that is, the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, by combining the known 3D space points and their positions on the 2D image.
[0032] After calculating the camera pose, use additional verification points to verify the correctness of the calculation results. Based on the verification results, adjust the model parameters or algorithm to improve the accuracy of pose estimation.
[0033] Step 2.3: According to the camera parameters and the feature points of the tiles to be laid, the current posture of the tiles to be laid is calculated according to the PnP algorithm.
[0034] To further improve the accuracy of pose estimation, geometric optimization methods such as the iterative closest point (ICP) algorithm can also be used to optimize the pose by minimizing the difference between the 3D model points and the image feature points.
[0035] Step 3: Get the current state of the robot arm, including the angles of each joint and the position and posture of the end effector.
[0036] Step 3.1, start the robot: Make sure the robot is up and running. This involves loading the corresponding robot model and startup file. For example, enter roslaunch xxx_moveit_config demo.launch in the command line to start the robot.
[0037] Step 3.2, check the joint status: Use the rostopic echo / joint_states command to check the joint status of the robot arm and output information including the radian value of the joint angle.
[0038] Step 3.3, Get the end effector position and posture: You can use the interface provided by MoveIt! to get the end effector position and posture. Use the current command to return the current state of the motion group, including the state of the joints and the state of the end effector (position and posture).
[0039] Step 3.4: Use inverse kinematics control: Use inverse kinematics control method to control the joint to reach a specific position.
[0040] Step 3.5. Consider the dynamics of the control system: When controlling the movement of the robotic arm, it is necessary to consider the dynamic characteristics of the system, such as the equation of motion and potential energy function under the action of external forces, to ensure that the robotic arm can accurately reach the target position and place the brick.
[0041] Step 3.6, monitor IO status: For external devices that need to interact with the robotic arm, you can monitor the input and output status of the device by defining structure types and variables, initializing the structure, and obtaining the IO port status value.
[0042] Step 3.7, Configure MoveIt!: Before using MoveIt! to control the robot arm, you need to use the Setup Assistant tool to complete the configuration of the self-collision matrix, planning group, terminal fixture, etc. according to the URDF model of the robot. After the configuration is completed, a ROS function package is generated for subsequent control operations.
[0043] Step 3.8, drive joints: Use the controller plug-in in the ArbotiX or ros_control function package to drive the robot arm joints. This usually involves creating a YAML configuration file for the plug-in, and starting the plug-in through the launch file and loading the configuration parameters.
[0044] Step 4: Determine the target location for laying tiles based on the location of the tiles and the size of the tile joints: Step 4.1, define the reference coordinate system: define a global reference coordinate system, which corresponds to the working plane for laying tiles; Step 4.2, determine the position of the tiles: measure or calculate the position and posture of each tile in the reference coordinate system, that is, the coordinates of the center point and the rotation angle of each tile; Step 4.3, calculate the target position: According to the size of the tiles, the preset paving pattern and the size of the brick joints, calculate the center point distance between adjacent tiles to obtain the target position that the end effector needs to reach.
[0045] Step 5: On the basis of satisfying the inverse kinematics constraints, the RRT algorithm is used to plan the optimal movement path of the robot arm according to the current posture of the tile to be laid, the current state of the robot arm, and the target position of the tile to be laid. The specific method is as follows: Step 5.1, initializing the configuration space: according to the working environment of the robot and the map information SLAM, select the configuration space to ensure that the configuration space contains all possible postures of the robot; Step 5.2, select the starting point and target point: According to the current pose of the tile to be laid and the target position of the tile to be laid, specify the starting position (x_init) and target position (x_goal) of the robot in the configuration space; Step 5.3, construct RRT: take the starting position as the root node, and generate a random expansion tree by adding leaf nodes through random sampling; in each iteration, select the tree node closest to the random sampling point, expand to the selected random sampling point, form a new node, and ensure that the new node does not collide with obstacles; Step 5.4, path optimization: When the leaf node of the random tree enters the target area, the RRT* algorithm or other path optimization methods are used to optimize the initially found path to reduce the path cost and approach or reach the optimal solution; Step 5.5, apply inverse kinematics: After the RRT algorithm finds the path, use inverse kinematics to convert each node on the path into the joint angle of the robot arm, ensuring that the robot arm posture corresponding to each node is achievable and meets the joint motion constraints; Step 5.6, generate joint trajectory: perform kinematic inverse solution on each node on the path to obtain the joint state of the robot, and then use the interpolation method to generate a set of joint state sequences of the robot on the path. These state sequences constitute the joint trajectory of the robot, that is, the moving path.
[0046] After obtaining the above movement path, it is also necessary to generate a detailed trajectory of the robot arm moving along the path, which includes setting parameters such as speed, acceleration, and jerk to ensure smooth and efficient movement.
[0047] Step 6: Control the robot arm to grab, move, and place the tiles to be laid according to the movement path and the set relevant parameters. The robot arm controller will send a control signal to the servo driver to achieve the required movement. The control signal is usually based on PID control or other control algorithms, specifically: Step 6.1. Select the mechanical drive module: Select a suitable mechanical drive module according to the application requirements and the specifications of the robotic arm. The module usually includes a motor, servo drive, stepper drive, etc., which are used to control the various joints or actuators of the robotic arm.
[0048] Step 6.2: Programming control logic: Write a control program to convert the planned path into control instructions that the robot arm can understand. This usually involves using a specialized programming language or software platform such as the Robot Operating System (ROS).
[0049] Step 6.3, implement closed-loop control: In order to ensure that the robot arm can move accurately along the predetermined path, it is usually necessary to implement a closed-loop control system. This can be done by using sensors (such as encoders, laser rangefinders, etc.) to monitor the actual position of the robot arm, compare it with the predetermined path, and then adjust the control instructions according to the deviation.
[0050] As the robot moves to the target position according to the calculated trajectory, the robot needs to monitor its status in real time so that adjustments can be made. After recognizing that the tile has reached the correct position, the end effector is controlled to release the tile.
[0051] Another embodiment of the present invention further provides a tile laying control system based on posture estimation, comprising: An image acquisition module is used to acquire a current image of the tiles to be laid through a camera arranged on the mechanical arm of the tile laying robot; A tile posture calculation module is used to perform target detection on the current image of the tile to be laid, and after identifying the tile to be laid, calculate the current posture of the tile to be laid; The robot arm state acquisition module is used to obtain the current state of the robot arm, including the angles of each joint and the position and posture of the end effector; A target position determination module is used to determine the target position of the tiles to be laid according to the position of the tiles and the size of the tile joints; A path planning module is used to plan the movement path of the robot arm according to the current posture of the tile to be laid, the current state of the robot arm, and the target position of the tile to be laid; The execution module is used to control the robot arm to grasp, move and place the tiles to be laid according to the moving path.
[0052] As for the system module disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.
[0053] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0054] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A tile laying control method based on posture estimation, characterized in that: The following steps are involved: Step 1: Capture the current image of the tiles to be laid by using a camera set on the mechanical arm of the tile laying robot; Step 2: performing target detection on the current image of the tiles to be laid, and after identifying the tiles to be laid, calculating the current posture of the tiles to be laid; Step 3: Get the current state of the robot arm, including the angles of each joint and the position of the end effector; Step 4: Determine the target position of the tiles to be laid according to the position of the tiles and the size of the tile joints; Step 5: Plan the movement path of the robotic arm according to the current posture of the tile to be laid, the current state of the robotic arm, and the target position of the tile to be laid; Step 6: Control the robot arm to grab, move and place the tiles to be laid according to the moving path.
2. A tile laying control method based on posture estimation according to claim 1, characterized in that: In step 2, calculating the current posture of the tiles to be laid includes the following steps: Step 2.1, extracting feature points of the tiles to be laid; Step 2.2: Calibrate the camera to obtain camera parameters, including intrinsic and extrinsic parameters; Step 2.3: According to the camera parameters and the feature points of the tiles to be laid, the current posture of the tiles to be laid is calculated according to the PnP algorithm.
3. A tile laying control method based on posture estimation according to claim 2, characterized in that: In the step 2.1, feature points of the tiles to be laid are extracted by Harris corner point detection, and the feature points include corner points, edges or texture features of the tiles to be laid.
4. A tile laying control method based on posture estimation according to claim 2, characterized in that: In step 2.2, the intrinsic parameters of the camera include focal length, principal point coordinates, and distortion coefficients, and the extrinsic parameters include rotation matrix and translation vector.
5. A method for controlling tile laying based on posture estimation according to claim 4, characterized in that: In step 2.2, the camera is calibrated by using the Zhang Zhengyou calibration method to calculate the camera's intrinsic and extrinsic parameters.
6. The method for controlling tile laying based on posture estimation according to claim 1, characterized in that: In step 4, the method for determining the target position of the tiles to be laid is: Step 4.1, define the reference coordinate system: define a global reference coordinate system, which corresponds to the working plane for laying tiles; Step 4.2, determine the position of the tiles: measure or calculate the position and posture of each tile in the reference coordinate system, that is, the coordinates of the center point and the rotation angle of each tile; Step 4.3, calculate the target position: According to the size of the tiles, the preset paving pattern and the size of the brick joints, calculate the center point distance between adjacent tiles to obtain the target position that the end effector needs to reach.
7. The method for controlling tile laying based on posture estimation according to claim 1, characterized in that: In step 5, the RRT algorithm is used to plan the movement path of the robot arm. The specific method is as follows: Step 5.1, Initialize the configuration space: Select the configuration space according to the working environment and map information SLAM of the robot arm; Step 5.2, select the starting point and target point: according to the current posture of the tile to be laid and the target position of the tile to be laid, clarify the starting position and target position of the robot arm in the configuration space; Step 5.3, construct RRT: take the starting position as the root node, and generate a random expansion tree by adding leaf nodes through random sampling; in each iteration, select the tree node closest to the random sampling point, expand to the selected random sampling point, form a new node, and ensure that the new node does not collide with obstacles; Step 5.4, path optimization: When the leaf node of the random tree enters the target area, the initially found path is optimized; Step 5.5, apply inverse kinematics: After the RRT algorithm finds the path, use inverse kinematics to convert each node on the path into the joint angle of the robot arm; Step 5.6, generate joint trajectory: perform kinematic inverse solution on each node on the path to obtain the joint state of the robot, and then use the interpolation method to generate a set of joint state sequences of the robot on the path. These state sequences constitute the joint trajectory of the robot, that is, the moving path.
8. A tile laying control system based on posture estimation, characterized in that: include: An image acquisition module is used to acquire a current image of the tiles to be laid through a camera arranged on the mechanical arm of the tile laying robot; A tile posture calculation module is used to perform target detection on the current image of the tile to be laid, and after identifying the tile to be laid, calculate the current posture of the tile to be laid; The robot arm state acquisition module is used to obtain the current state of the robot arm, including the angles of each joint and the position and posture of the end effector; A target position determination module is used to determine the target position of the tiles to be laid according to the position of the tiles and the size of the tile joints; A path planning module is used to plan the movement path of the robot arm according to the current posture of the tile to be laid, the current state of the robot arm, and the target position of the tile to be laid; The execution module is used to control the robot arm to grasp, move and place the tiles to be laid according to the moving path.
Citation Information
Patent Citations
Ceramic tile paving method and automatic ceramic tile paving control system
CN113482301A
Robot brick paving method and device, electronic equipment and storage medium
CN114250957A
Mechanical arm intelligent control method and device based on dynamic vision and brick paving robot
CN116641532A
Tile laying control method, robot and storage medium
CN116690553A
Apparatus and method for placing a tile on a floor
EP2907938A1
