Warehouse storage location detection method capable of identifying position angle

Abstract

The present application belongs to the field of warehouse storage location recognition, and specifically relates to a warehouse storage location detection method capable of recognizing position angles, comprising the following steps: 1) camera angle and scale calibration is performed on a camera on a carrying robot to convert image pixel coordinates into real scale coordinates; 2) an offset value between an image coordinate system and a carrying robot coordinate system is obtained, that is, a corresponding relationship between the image coordinate system and the carrying robot coordinate system is obtained, and then a cargo position in the carrying robot coordinate system is obtained; 3) according to the cargo position in the carrying robot coordinate system, a cargo bounding box is obtained through an image detection algorithm, a sub-image in the bounding box is processed, and a cargo angle is obtained; and the warehouse management flexibility can be greatly improved through the warehouse storage location detection method.

Description

Technical Field

[0001] This invention belongs to the field of warehouse location identification, specifically a warehouse location detection method that can identify the location angle. Background Technology

[0002] In unmanned logistics warehouses, a location detection system is needed to provide warehouse status information. This system uses overhead cameras positioned at a high altitude to acquire information within its field of view and then employs feature recognition methods to determine the status of certain locations. Current feature recognition methods include traditional image feature extraction and deep learning. The real-time detection results are then used for feature matching to calculate the type of goods.

[0003] The location detection system can calculate the position of goods in the image in real time. However, in order to establish a connection with the automated handling robot that picks up the goods, the location detection system needs to pre-configure the area of ​​each location in the image, and the angle cannot be arbitrarily placed. It cannot effectively calculate all position information within the camera's field of view. Summary of the Invention

[0004] The purpose of this invention is to provide a warehouse location detection method that can identify the position and angle, and a method for calculating the pose of goods in the coordinate system of a handling robot, so as to make the overall placement of goods more flexible.

[0005] The technical solution adopted by the present invention to achieve the above objectives is: a warehouse location detection method that can identify position angles, comprising the following steps:

[0006] 1) Perform camera angle and scale calibration on the camera on the handling robot to convert the image pixel coordinates into real scale coordinates;

[0007] 2) Obtain the offset value between the image coordinate system and the handling robot coordinate system, that is: obtain the correspondence between the image coordinate system and the handling robot coordinate system, and then obtain the position of the goods in the handling robot coordinate system;

[0008] 3) Based on the position of the goods in the coordinate system of the handling robot, the bounding box of the goods is obtained through image detection algorithm. The sub-images in the bounding box are processed to obtain the angle of the goods.

[0009] 4) Based on steps 1) to 3), obtain the position and angle of the goods in the coordinate system of the handling robot.

[0010] Step 1) specifically includes:

[0011] 1-1) Select four points on the ground that are not on the same straight line within the camera's field of view, and measure the relative positions of the true scale values ​​of the four points. That is, the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4).

[0012] 1-2) Find the positions of the four corresponding points in the image coordinate system from the camera image, and calculate the homography transformation matrix of the camera relative to the ground using these four sets of corresponding points;

[0013] 1-3) Decompose this homography transformation matrix to obtain the scaling factor;

[0014] 1-4) Perform affine transformation of the camera image using the homography transformation matrix, and combine it with the scale factor to convert the image pixel coordinates into real scale coordinates.

[0015] Steps 1-2) are specifically as follows:

[0016] Using these four corresponding points, calculate the homography matrix of the camera relative to the ground. The homography matrix is ​​a 3x3 matrix, denoted as H(h11-h33), where h33 is always equal to 1.

[0017]

[0018] Steps 1-3) are specifically as follows:

[0019] a. Multiply the homography matrix H on the left by the inverse of the camera matrix and then on the right by the camera matrix to obtain matrix A;

[0020] b. Perform SVD decomposition on A to obtain unitary matrices U and V;

[0021] c. Multiply the determinant of the transpose of U by the determinant of V to obtain the scaling factor s.

[0022] Steps 1-4) are specifically as follows:

[0023] Using this homography transformation matrix, an affine transformation is performed on the camera image. The pixel coordinates (Pix, Piy) in the image after the affine transformation are multiplied by the scale factor s, that is: the point (s*Pix, s*Piy) is converted from the image pixel coordinates to the true scale coordinates.

[0024] The offset values ​​between the image coordinate system and the transport robot coordinate system include: the offset Delta_X in the X-axis direction, the offset Delta_Y in the Y-axis direction, and the sine value Sin_Theta and the cosine value Cos_Theta of the angle offset.

[0025] Step 2) includes the following steps:

[0026] 2-1) Select two points in the image after affine transformation in step 1-4) and record the coordinates (C1_X, C2_Y) and (C2_X, C2_Y) in the image coordinate system;

[0027] 2-2) The transport robot travels to these two points and records their coordinates within the robot as (V1_X, V2_Y) and (V2_X, V2_Y), thus obtaining the correspondence between the two points:

[0028]

[0029] 2-3) Calculate the four offsets to be calculated: Delta_X, Delta_Y, Sin_Theta, and Cos_Theta;

[0030] 2-4) Let the position of the goods in the image coordinate system be point (Pcx, Pcy). The position of the goods in the robot coordinate system (Prx, Pry) is obtained by the following formula:

[0031]

[0032] Step 3) includes the following steps:

[0033] 3-1) Obtain the bounding box of the goods through an image detection algorithm, extract the sub-images within this bounding box, perform LSD line detection, and record the length and angle of the lines;

[0034] 3-2) Extract the lines from this set whose length is greater than 0.5 times the length of the longest side of the Bounding Box, and record the angles of these lines;

[0035] 3-3) Since the goods rely on the pallet, and the pallet and its load are placed in a rectangular space, the straight line is reduced from a 360° interval to a 90° interval, that is:

[0036] abgle = atan(tan(atan(x)×2))

[0037] Where x is the angle value in the interval (0, 360°);

[0038] 3-4) Sort the angle values ​​in the (0, 90°) interval from smallest to largest, then set a sliding window to traverse them, and record the number of lines in the window;

[0039] 3-5) After the traversal is completed, extract the window positions with the maximum number of positions, and take the average of all the angle values ​​in them, which is the angle of the goods in the coordinate system of the handling robot.

[0040] The present invention has the following beneficial effects and advantages:

[0041] 1. By using the method of the present invention for location detection, the flexibility of warehouse management can be greatly improved.

[0042] 2. This invention does not require pre-configuring the area of ​​each storage location in the image, and it can effectively calculate all location information even for goods placed at arbitrary angles.

[0043] 3. This invention solves for the position and angle of the goods in the coordinate system of the handling robot, making the detection method simpler and faster. Attached Figure Description

[0044] Figure 1 This is a flowchart of a preferred embodiment of the present invention;

[0045] Figure 2 This is the image before the affine transformation in a preferred embodiment of the present invention;

[0046] Figure 3 This is the image after affine transformation in a preferred embodiment of the present invention, at which point coordinate system calibration is performed;

[0047] Figure 4 This is the final calculation result in a preferred embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0049] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the present invention. The present invention provides a warehouse location detection method capable of identifying position angles, comprising the following steps:

[0050] 1) Perform camera angle and scale calibration on the camera on the handling robot to convert the image pixel coordinates into real scale coordinates;

[0051] 1. First, perform camera angle and scale calibration. Select four points on the ground within the camera's field of view that are not on the same straight line. Measure the relative positions of these four points at their true scale values. For example, the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4). Then, find the corresponding positions of the four points in the image coordinate system from the camera image. For example, the coordinates of the first point in the image are (Pcx1, Pcy1), the second point is (Pcx2, Pcy2), the third point is (Pcx3, Pcy3), and the fourth point is (Pcx4, Pcy4). Using these four sets of corresponding points, the homography transformation matrix of the camera relative to the ground can be calculated. The homography transformation matrix is ​​a 3x3 matrix, denoted as H(h11-h33). The calculation method is shown below. In the homography matrix, h33 is always equal to 1.

[0052]

[0053] Using SVD decomposition, the eight unknowns from h11 to h32 can be solved to obtain the homography matrix. Decomposing this homography matrix yields the scale factor s. Specifically, the homography matrix H is left-multiplied by the inverse of the camera matrix and right-multiplied by the camera matrix to obtain matrix A. Then, SVD decomposition is performed on A to obtain unitary matrices U and V. Multiplying the determinant of the transpose of U by the determinant of V gives the scale factor s. Using this homography matrix, an affine transformation is performed on the camera image. The pixel coordinates (Pix, Piy) in the affine transformed image are multiplied by the scale factor s, i.e., the points (s*Pix, s*Piy) are converted from image pixel coordinates to true scale coordinates.

[0054] 2) Obtain the offset value between the image coordinate system and the handling robot coordinate system, that is: obtain the correspondence between the image coordinate system and the handling robot coordinate system, and then obtain the position of the goods in the handling robot coordinate system;

[0055] 2. The calibration between the image coordinate system and the transport robot coordinate system requires calculating the offsets in the X-axis direction (Delta_X), Y-axis direction (Delta_Y), and the sine (Sin_Theta) and cosine (Cos_Theta) values ​​of the angle offsets—a total of four offset values ​​to be calculated. This is done by selecting two points in the image after the affine transformation and recording their coordinates in the image coordinate system (C1_X, C2_Y and C2_X, C2_Y). The transport robot then travels to these two points, as shown below. Figure 3As shown, the coordinates (V1_X, V2_Y and V2_X, V2_Y) of these two points in the transport robot are recorded simultaneously to obtain the correspondence between the two points. The two points in the image coordinate system, the two points in the transport robot coordinate system, and the four offsets to be calculated mentioned above have the following calculation relationship:

[0056]

[0057] 3. The four offsets Delta_X, Delta_Y, Sin_Theta, and Cos_Theta can be calculated. The above formula also shows the correspondence between the image coordinate system and the robot coordinate system. By calculation, the position in the robot coordinate system can be obtained. Assuming that the position of the goods in the image coordinate system is point (Pcx, Pcy), the position of the goods in the robot coordinate system (Prx, Pry) can be calculated by the following formula:

[0058]

[0059] 3) Based on the position of the goods in the coordinate system of the handling robot, the bounding box of the goods is obtained through image detection algorithm. The sub-images in the bounding box are processed to obtain the angle of the goods.

[0060] 4. Finally, angle calculations are performed. The bounding box of the goods is obtained through image detection algorithms. Sub-images within this bounding box are extracted, and LSD line detection is performed, recording the length and angle of each line. Lines longer than 0.5 times the longest side of the bounding box are extracted from this set of lines, and their angles are recorded. Since goods are mostly supported by pallets, and pallets and their loads are typically placed in rectangular spaces, the following formula can be used to reduce the line angle from a 360° interval to a 90° interval, where x is the angle value within the 360° interval:

[0061] abgle = atan(tan(atan(x)×2))

[0062] Arrange the angle values ​​within the obtained 90° interval in ascending order, then set an appropriate sliding window to traverse the range, recording the number of straight lines within each window. After the traversal is complete, extract the window position with the largest number of lines; the average of all angle values ​​within that window is the cargo angle. Figure 4 As shown.

[0063] 4) Based on steps 1) to 3), obtain the position and angle of the goods in the coordinate system of the handling robot.

[0064] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A warehouse location detection method capable of identifying position angles, characterized in that, Includes the following steps: 1) Perform camera angle and scale calibration on the camera on the handling robot to convert the image pixel coordinates into real scale coordinates; Step 1) specifically refers to: 1-1) Select four points on the ground that are not on the same straight line within the camera's field of view, and measure the relative positions of the true scale values ​​of the four points. That is, the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4). 1-2) Find the positions of the four corresponding points in the image coordinate system from the camera image, and calculate the homography transformation matrix of the camera relative to the ground using these four sets of corresponding points; 1-3) Decompose this homography transformation matrix to obtain the scale factor; 1-4) Perform affine transformation of the camera image using the homography transformation matrix, and combine it with the scale factor to convert the image pixel coordinates into real scale coordinates; 2) Obtain the offset value between the image coordinate system and the handling robot coordinate system, that is: obtain the correspondence between the image coordinate system and the handling robot coordinate system, and then obtain the position of the goods in the handling robot coordinate system; The offset values ​​between the image coordinate system and the handling robot coordinate system include: the offset Delta_X in the X-axis direction, the offset Delta_Y in the Y-axis direction, and the sine value Sin_Theta and the cosine value Cos_Theta of the angle offset. Step 2) includes the following steps: 2-1) Select two points in the image after affine transformation in step 1-4) and record the coordinates (C1_X, C2_Y) and (C2_X, C2_Y) in the image coordinate system. 2-2) The transport robot travels to these two points and records their coordinates within the robot as (V1_X, V2_Y) and (V2_X, V2_Y), thus obtaining the correspondence between the two points: ； 2-3) Calculate the four offsets to be calculated: Delta_X, Delta_Y, Sin_Theta, and Cos_Theta; 2-4) Let the position of the cargo in the image coordinate system be point (Pcx, Pcy). The position of the cargo in the robot coordinate system (Prx, Pry) is obtained by the following formula: ； 3) Based on the position of the goods in the coordinate system of the handling robot, the bounding box of the goods is obtained through image detection algorithm. The sub-images in the bounding box are processed to obtain the angle of the goods. 4) Based on steps 1) to 3), obtain the position and angle of the goods in the coordinate system of the handling robot.

2. The warehouse location detection method with identifiable position angle according to claim 1, characterized in that, Steps 1-2) are specifically as follows: Using these four corresponding points, calculate the homography matrix of the camera relative to the ground. The homography matrix is ​​a 3x3 matrix, denoted as H(h11-h33), where h33 is always equal to 1. 。 3. The warehouse location detection method with identifiable position angle according to claim 1, characterized in that, Steps 1-3 are specifically as follows: a. Multiply the homography transformation matrix H on the left by the inverse of the camera matrix and then on the right by the camera matrix to obtain matrix A; b. Perform SVD decomposition on A to obtain unitary matrices U and V; c. Multiply the determinant of the transpose of U by the determinant of V to obtain the scaling factor s.

4. The warehouse location detection method with identifiable position angle according to claim 3, characterized in that, Steps 1-4 are specifically as follows: Using this homography transformation matrix, an affine transformation is performed on the camera image. The pixel coordinates (Pix, Piy) in the image after the affine transformation are multiplied by the scale factor s, that is: the point (s*Pix, s*Piy) is converted from the image pixel coordinates to the true scale coordinates.

5. The warehouse location detection method with identifiable position angle according to claim 1, characterized in that, Step 3) includes the following steps: 3-1) Obtain the bounding box of the goods through an image detection algorithm, extract the sub-images within this bounding box, perform LSD line detection, and record the length and angle of the lines; 3-2) Extract the lines from this set whose length is greater than 0.5 times the length of the longest side of the Bounding Box, and record the angles of these lines; 3-3) Since the goods rely on the pallet, and the pallet and its load are placed in a rectangular space, the straight line is reduced from a 360° interval to a 90° interval, that is: ； Where x is the angle value in the interval (0, 360°); 3-4) Sort the angle values ​​in the (0, 90°) interval from smallest to largest, then set a sliding window to traverse them, and record the number of lines in the window; 3-5) After the traversal is completed, extract the window positions with the maximum number of positions, and take the average of all the angle values ​​in them, which is the angle of the goods in the coordinate system of the handling robot.

Images