Goods shelf positioning method and system based on double two-dimensional codes

Abstract

The invention discloses a shelf positioning method based on double two-dimensional codes, and the method comprises the steps: (1) collecting a front image through a camera, recognizing two-dimensional code combinations in the image, and calculating the poses of all two-dimensional code combinations in a coordinate system of a mobile robot; (2) acquiring a two-dimensional code combination closest to the camera, and taking the pose of the acquired two-dimensional code combination in a coordinate system of the mobile robot as a positioning pose; wherein the camera is fixed on a gear shaper of the mobile robot. The double-two-dimensional code combination is identified and positioned by means of the RGBD camera with relatively low cost, high-precision positioning of the goods shelf is further realized, the method is also suitable for moving the goods shelf, and a guarantee is provided for agv high-precision automatic goods picking and placing.

Description

Technical Field

[0001] This invention belongs to the field of shelf positioning technology, and more specifically, this invention relates to a shelf positioning method and system based on dual QR codes. Background Technology

[0002] With the rapid development of industrial logistics automation, AGVs are increasingly widely used in warehousing and logistics scenarios. In logistics centers, factory production workshops, and warehouses, AGVs significantly improve warehousing operation efficiency and resource utilization through automated material handling. Reliable and accurate shelf identification and positioning are key to achieving efficient material handling and refined management, affecting not only the accuracy and safety of picking and placing goods but also playing a crucial role in the stability and accuracy of the entire logistics system.

[0003] Existing shelf identification and positioning technologies mainly rely on lidar for identification and positioning. For example, patent application number CN2022109005734, entitled "Shelf Identification Method and Related Equipment", uses lidar to scan and obtain the laser point cloud of at least one shelf, and determines the position of two symmetrical reflective strips of at least one shelf based on the laser point cloud, thereby determining the position of the shelf.

[0004] LiDAR can perform environmental modeling and distance measurement. However, high-precision LiDAR is not only more expensive, but its positioning accuracy may also be reduced due to factors such as the inability of its scanning points to distinguish between shelves and goods, and the obstruction of goods or obstacles. Summary of the Invention

[0005] This invention provides a shelf positioning method based on dual QR codes, aiming to improve at least one of the above-mentioned problems.

[0006] This invention is implemented as follows: a shelf positioning method based on dual QR codes, the method is as follows:

[0007] (1) Acquire images in front of the camera, identify the QR code combinations in the images, and calculate the pose of all QR code combinations in the mobile robot coordinate system;

[0008] (2) Obtain the QR code combination closest to the camera, and use the pose of the obtained QR code combination in the mobile robot coordinate system as the positioning pose;

[0009] The camera is fixed to the teeth of the mobile robot.

[0010] Furthermore, the specific process for calculating the pose of the QR code combination in the mobile robot's coordinate system is as follows:

[0011] (11) Calculate the pose of each QR code combination in the image in the depth camera coordinate system. ;

[0012] (12) Calculate the pose of each QR code combination in the mobile robot coordinate system. , pose The specific calculation formula is as follows:

[0013] ;

[0014] in, This represents the pose of the depth camera coordinate system within the mobile robot coordinate system. This indicates the orientation of the right-handed coordinate system within the local coordinate system of the QR code combination.

[0015] Furthermore, the pose of the QR code combination in the depth camera coordinate system. The specific process for obtaining it is as follows:

[0016] (111) Calculate the coordinates of the corner points of the two QR codes in the local coordinate system and the RGB camera coordinate system, and put them into the set local_pts and the set image_pts respectively according to the set order;

[0017] (112) Input the points in the set local_pts and set image_pts into the camera perspective projection model, and obtain the pose of the QR code combination center in the RGB camera coordinate system with the minimum reprojection error by using the Gauss-Newton method;

[0018] (113) Transformation matrix of RGB camera coordinate system in depth camera coordinate system The pose of the QR code combination in the RGB camera coordinate system is converted into the pose of the QR code combination in the depth camera coordinate system.

[0019] Furthermore, the specific method for determining the coordinates of the corner points of two QR codes in the local coordinate system is as follows:

[0020] Construct a local coordinate system for each pair of QR codes in the image, and calculate the pose of the two QR code centers in the local coordinate system.

[0021] Calculate the coordinates of the four corner points of each QR code in the QR code combination relative to the center of the corresponding QR code, and calculate the coordinates of the four corner points of the QR code in the local coordinate system based on the pose of the center of the corresponding QR code in the local coordinate system.

[0022] Furthermore, after step (113), the following is also included:

[0023] The depth image captured by the depth camera is converted into a depth point cloud in the depth camera coordinate system. Based on the coordinates and size of the two QR codes in the depth camera coordinate system in the two-dimensional combination, the point set belonging to the two QR codes is found in the depth point cloud.

[0024] Calculate the normal vector of the plane containing the point sets of the two QR codes, and check whether the angle between the normal vector of the plane containing the point sets of the two QR codes and the normal vector of the combined QR code is within the set angle threshold range. If so, the QR code combination positioning is considered successful.

[0025] Furthermore, the specific method for detecting point sets of two QR codes belonging to a QR code combination in a deep point cloud is as follows:

[0026] First, calculate the coordinates of the centers of the two QR codes in the depth camera coordinate system. , ;

[0027] respectively , Centered on the half-width of the QR code Using the search radius, search for points belonging to the two QR codes in the deep point cloud and put them into the point set pcd_a and the click pcd_b respectively.

[0028] Furthermore, the coordinates of the centers of the two QR codes in the depth camera coordinate system are... , The specific calculation formula is as follows:

[0029] ;

[0030] ;

[0031] in, , The coordinates of the centers of the two QR codes in the QR code combination under local coordinates. , The coordinates of the centers of the two QR codes in the QR code combination in the depth camera coordinate system.

[0032] Furthermore, the process of obtaining the normal vector of the plane containing the point sets of the two QR codes is as follows:

[0033] The coordinates of each point in point set pcd_a in the depth camera coordinate system are used as row vectors to construct matrix A_a, and the coordinates of each point in point set pcd_b in the depth camera coordinate system are used as row vectors to construct matrix A_b. The formulas for calculating the normal vectors of the planes containing the points in point sets pcd_a and pcd_b are as follows:

[0034] ;

[0035] in, It is represented as a column vector with an element value of -1.

[0036] Furthermore, the normal vector of the QR code combination in the depth camera coordinate system The specific acquisition process is as follows:

[0037] Read the pose of the center of the QR code combination in the depth camera coordinate system posture Calculate the normal vector of the QR code combination. The specific calculation formula is as follows:

[0038] ;

[0039] in, This represents the unit normal vector.

[0040] This invention is implemented as follows: a shelf positioning system based on dual QR codes, the system comprising:

[0041] The QR codes are located at both ends of the beams on each layer of the multi-layer shelving, forming a QR code combination. A camera is located on the tooth of the mobile robot, which integrates an RGB camera and a depth camera, and a processor that communicates with the camera.

[0042] The camera captures RGB and depth images of the area in front of the moving robot and sends them to the processor. The processor determines the position of the shelf layer to be picked up or placed based on the aforementioned shelf positioning method using dual QR codes.

[0043] This invention utilizes a relatively low-cost RGBD camera to identify and locate combinations of dual QR codes, thereby achieving high-precision positioning of shelves. It is also applicable to mobile shelves, ensuring high-precision automatic picking and placing of goods by AGVs. Attached Figure Description

[0044] Figure 1 A schematic diagram of a shelf positioning system based on dual QR codes provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the arrangement of QR codes on a multi-layer shelf provided in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of a local coordinate system constructed for each QR code combination, provided in an embodiment of the present invention.

[0047] Figure 4 The flowchart of the shelf positioning method based on dual QR codes provided in this embodiment of the invention is shown below. Detailed Implementation

[0048] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0049] Figure 1 This is a schematic diagram of a shelf positioning system based on dual QR codes provided in an embodiment of the present invention. For ease of explanation, only the parts related to the embodiment of the present invention are shown. The system includes:

[0050] The QR codes located at both ends of the beams on each level of the multi-layer shelving unite to form a QR code combination, such as... Figure 2 As shown, a camera is mounted on the tooth of the mobile robot. The camera integrates an RGB camera and a depth camera, and a processor is connected to the camera. The camera captures RGB and depth images of the front of the mobile robot and sends them to the processor. The processor determines the position of the shelf layer to be picked up or placed based on the following shelf positioning method based on dual QR codes.

[0051] In this embodiment of the invention, combined with Figure 4 The shelf positioning method based on dual QR codes is explained below:

[0052] (1) Acquire RGB images of the front of the mobile robot by camera, identify the QR code combinations in the RGB images, and calculate the pose of all QR code combinations in the mobile robot coordinate system;

[0053] (2) Obtain the QR code combination closest to the camera, and use the pose of the obtained QR code combination in the coordinate system of the mobile robot as the positioning pose to provide the mobile robot with accurate picking and placing of goods.

[0054] In this embodiment of the invention, the calculation process of the pose of the QR code combination in the mobile robot coordinate system is as follows:

[0055] (11) Calculate the pose of the QR code combination in the depth camera coordinate system. The specific calculation process is as follows:

[0056] (111) Calculate the coordinates of the corner points of the two QR codes in the local coordinate system and the RGB camera coordinate system, and put them into the set local_pts and the set image_pts respectively according to the set order;

[0057] Construct a local coordinate system for each pair of QR codes in the image, and calculate the pose of the two QR code centers in the local coordinate system.

[0058] In this embodiment of the invention, the QR code has a width of The square shape is defined with the midpoint of the line connecting the centers of the two QR codes in the QR code combination as the origin of the local coordinate system `bundle_tag`. The direction of the line connecting the centers of the two QR codes in the QR code combination is the x-axis, the height of the shelf is the y-axis, and the direction perpendicular to the xy plane is the z-axis. The local coordinate system of the QR code combination is as follows: Figure 3 As shown, first determine the pose of the centers of the two QR codes in the local coordinate system. , .

[0059] Calculate the local coordinates of all corner points of the two QR codes in the local coordinate system and put them into the set local_pts in a specified order;

[0060] In this embodiment of the invention, the coordinates of the four corner points of each QR code in the QR code combination relative to the center of the corresponding QR code are first determined. Based on the pose of the center of the corresponding QR code in the local coordinate system, the local coordinates of the four corner points of the QR code in the local coordinate system are calculated. Then, the four corner points of the two QR codes are put into the set local_pts in the order of the lower left corner point p1, the lower right corner point p2, the upper right corner point p3, and the upper left corner point p4.

[0061] The RGB camera captures RGB images, and extracts the ID numbers of the two QR codes and the pixel coordinates of the four corner points from the QR code combination in the RGB image. The pixel coordinates are put into the collection image_pts in the same specified order.

[0062] (112) Input the points in the set local_pts and set image_pts into the camera perspective projection model, and obtain the pose of the QR code combination center in the RGB camera coordinate system with the minimum reprojection error by using the Gauss-Newton method;

[0063] The points in the sets local_pts and image_pts are input into the camera perspective projection model. The pose of the QR code combination center under the RGB camera with the minimum reprojection error is obtained by the Gauss-Newton method. The specific camera perspective projection model is shown in formula (1):

[0064] (1)

[0065] in, As the normalization factor, This is the intrinsic parameter matrix of the camera (RGB). The extrinsic parameter matrix represents the pose of the QR code combination's local coordinate system (the center of the QR code combination) under the RGB camera.

[0066] (113) Transformation matrix of RGB camera coordinate system in depth camera coordinate system The pose of the QR code combination in the RGB camera coordinate system is converted into the pose of the QR code combination in the depth camera coordinate system.

[0067] Read the transformation matrix of the RGB camera coordinate system in the depth camera coordinate system Transformation matrix Provided by the camera manufacturer, the pose of the QR code combination in the depth camera coordinate system is obtained by transforming it using formula (2). Formula (2) is as follows:

[0068] (2)

[0069] In this embodiment of the invention, the method further includes the following after step (113):

[0070] (114) Convert the depth image captured by the depth camera into a depth point cloud in the depth camera coordinate system, and find the point set belonging to the two QR codes in the depth point cloud based on the coordinates and QR code size of the two QR codes in the depth camera coordinate system in the two-dimensional combination.

[0071] (115) Calculate the normal vector of the plane containing the point sets of the two QR codes, and check whether the angle between the normal vector of the plane containing the point sets of the two QR codes and the normal vector of the QR code combination is within the set angle threshold range. If so, the QR code combination positioning is considered successful.

[0072] In this invention, the intrinsic parameter matrix of the depth camera is K_depth, which represents the pixels in the depth image acquired by the depth camera. Projected onto a 3D depth camera coordinate system, forming a depth point cloud. The intrinsic parameter matrix of the depth camera is K_depth, which is represented as follows:

[0073] (3)

[0074] in, , These represent the length per pixel of the depth camera's focal length along the x-axis and y-axis of the image, respectively. , These represent the coordinates of the camera principal point (the intersection of the optical axis and the image plane) in the image coordinate system. The intrinsic parameter matrix K_depth of the depth camera is used to represent the pixel values ​​in the depth image. Projected onto a 3D depth camera coordinate system, where, The depth value d in the u-th row and v-th column of the depth image is represented by a pixel. The 3D coordinates obtained after projection onto the 3D depth camera coordinate system are: The projection process is as follows:

[0075] (4)

[0076] In this embodiment of the invention, the method for detecting point sets belonging to two QR codes in a QR code combination in a depth point cloud is as follows:

[0077] First, calculate the coordinates of the centers of the two QR codes in the depth camera coordinate system. , , respectively , Centered on the half-width of the QR code Using the search radius, search for points belonging to the two QR codes in the deep point cloud and put them into the point set pcd_a and the click pcd_b respectively.

[0078] In this embodiment, the coordinates of the centers of the two QR codes in the QR code combination in the depth camera coordinate system are... , The specific calculation formula is as follows:

[0079] (5)

[0080] (6)

[0081] in, , Find the coordinates of the center of QR code a and the center of QR code b in the local coordinate system. , The coordinates of the center of QR code a and the center of QR code b in the depth camera coordinate system.

[0082] In this embodiment of the invention, the process of obtaining the normal vector of the plane containing the point sets of the two QR codes is as follows:

[0083] The coordinates of each point in the point set pcd_a in the depth camera coordinate system As row vectors, construct matrix A_a, which represents the coordinates of each point in the point set pcd_b in the depth camera coordinate system. As row vectors, construct matrix A_b, and then construct the following system of linear equations:

[0084] (7)

[0085] in, , Let represent the normal vectors of the planes containing points in point set pcd_a and point set pcd_b, respectively. This is represented as a column vector with an element value of -1. Based on formula (7), the plane normal vector... Plane normal vector The specific calculation formula is as follows:

[0086] (8)

[0087] In this embodiment of the invention, the normal vector of the QR code combination in the depth camera coordinate system is... The specific acquisition process is as follows:

[0088] Read the pose of the center of the QR code combination in the depth camera coordinate system posture Calculate the normal vector of the QR code combination. The specific calculation formula is as follows:

[0089] (9)

[0090] in, This represents the unit normal vector.

[0091] The normal vector of the QR code combination is calculated based on formulas (10) and (11). The normal vectors of the depth point clouds of QR code a and QR code b in the combination. Normal vector The included angle Angle The specific calculation formula is as follows:

[0092] (10)

[0093] (11)

[0094] If the included angle Angle If all the QR codes are within the set angle threshold range, the QR code combination positioning is considered successful; otherwise, the QR code positioning is considered unsuccessful.

[0095] (12) Calculate the pose of the QR code combination in the mobile robot coordinate system.

[0096] After successfully recognizing and positioning the QR code combination, the pose of the combined QR code is converted to be consistent with the right-hand coordinate system and then transferred to the mobile robot's coordinate system to obtain the pose. :

[0097] (12)

[0098] in, The pose of the depth camera coordinate system in the mobile robot coordinate system is determined based on the installation dimensions of the mechanical design. This represents the orientation (rotation matrix) of the right-handed coordinate system within the local coordinate system of the QR code assembly. It is also a standard quantity.

[0099] This invention utilizes a relatively low-cost RGBD camera to identify and locate combinations of dual QR codes, thereby achieving high-precision positioning of shelves. It is also applicable to mobile shelves, ensuring high-precision automatic picking and placing of goods by AGVs.

[0100] The present invention has been described by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A shelf positioning method based on dual QR codes, characterized in that, The method is as follows: (1) Acquire images in front of the camera, identify the QR code combinations in the images, and calculate the pose of all QR code combinations in the mobile robot coordinate system; (2) Obtain the QR code combination closest to the camera, and use the pose of the obtained QR code combination in the mobile robot coordinate system as the positioning pose; The camera is fixed to the teeth of the mobile robot.

2. The shelf positioning method based on dual QR codes as described in claim 1, characterized in that, The specific process for calculating the pose of the QR code combination in the coordinate system of the mobile robot is as follows: (11) Calculate the pose of each QR code combination in the image in the depth camera coordinate system. ; (12) Calculate the pose of each QR code combination in the mobile robot coordinate system. , pose The specific calculation formula is as follows: ； in, This represents the pose of the depth camera coordinate system within the mobile robot coordinate system. This indicates the orientation of the right-handed coordinate system within the local coordinate system of the QR code combination.

3. The shelf positioning method based on dual QR codes as described in claim 2, characterized in that, pose of the QR code combination in the depth camera coordinate system The specific process for obtaining it is as follows: (111) Calculate the coordinates of the corner points of the two QR codes in the local coordinate system and the RGB camera coordinate system, and put them into the set local_pts and the set image_pts respectively according to the set order; (112) Input the points in the set local_pts and set image_pts into the camera perspective projection model, and obtain the pose of the QR code combination center in the RGB camera coordinate system with the minimum reprojection error by using the Gauss-Newton method; (113) Transformation matrix of RGB camera coordinate system in depth camera coordinate system The pose of the QR code combination in the RGB camera coordinate system is converted into the pose of the QR code combination in the depth camera coordinate system.

4. The shelf positioning method based on dual QR codes as described in claim 3, characterized in that, The specific method for determining the coordinates of the corner points of two QR codes in a local coordinate system is as follows: Construct a local coordinate system for each pair of QR codes in the image, and calculate the pose of the two QR code centers in the local coordinate system. Calculate the coordinates of the four corner points of each QR code in the QR code combination relative to the center of the corresponding QR code, and calculate the coordinates of the four corner points of the QR code in the local coordinate system based on the pose of the center of the corresponding QR code in the local coordinate system.

5. The shelf positioning method based on dual QR codes as described in claim 3, characterized in that, The process includes the following steps after step (113): The depth image captured by the depth camera is converted into a depth point cloud in the depth camera coordinate system. Based on the coordinates and size of the two QR codes in the depth camera coordinate system in the two-dimensional combination, the point set belonging to the two QR codes is found in the depth point cloud. Calculate the normal vector of the plane containing the point sets of the two QR codes, and check whether the angle between the normal vector of the plane containing the point sets of the two QR codes and the normal vector of the combined QR code is within the set angle threshold range. If so, the QR code combination positioning is considered successful.

6. The shelf positioning method based on dual QR codes as described in claim 5, characterized in that, The specific method for detecting point sets of two QR codes in a QR code combination in deep point cloud is as follows: First, calculate the coordinates of the centers of the two QR codes in the depth camera coordinate system. , ; respectively , Centered on the half-width of the QR code Using the search radius, search for points belonging to the two QR codes in the deep point cloud and put them into the point set pcd_a and the click pcd_b respectively.

7. The shelf positioning method based on dual QR codes as described in claim 6, characterized in that, The coordinates of the centers of the two QR codes in the depth camera coordinate system , The specific calculation formula is as follows: ； ； in, , The coordinates of the centers of the two QR codes in the QR code combination under local coordinates. , The coordinates of the centers of the two QR codes in the QR code combination in the depth camera coordinate system.

8. The shelf positioning method based on dual QR codes as described in claim 5, characterized in that, The process of obtaining the normal vector of the plane containing the point sets of the two QR codes is as follows: The coordinates of each point in point set pcd_a in the depth camera coordinate system are used as row vectors to construct matrix A_a, and the coordinates of each point in point set pcd_b in the depth camera coordinate system are used as row vectors to construct matrix A_b. The formulas for calculating the normal vectors of the planes containing the points in point sets pcd_a and pcd_b are as follows: ； in, It is represented as a column vector with an element value of -1.

9. The shelf positioning method based on dual QR codes as described in claim 5, characterized in that, The normal vector of the QR code combination in the depth camera coordinate system The specific acquisition process is as follows: Read the pose of the center of the QR code combination in the depth camera coordinate system posture Calculate the normal vector of the QR code combination. The specific calculation formula is as follows: ； in, This represents the unit normal vector.

10. A shelf positioning system based on dual QR codes, characterized in that, The system includes: The QR codes are located at both ends of the beams on each layer of the multi-layer shelving, forming a QR code combination. A camera is located on the tooth of the mobile robot, which integrates an RGB camera and a depth camera, and a processor that communicates with the camera. The camera captures RGB and depth images of the area in front of the moving robot and sends them to the processor. The processor determines the position of the shelf layer to be picked up or placed based on the shelf positioning method based on dual QR codes as described in any one of claims 1 to 9.

Images