Warehouse goods location detection method capable of identifying position angle

Abstract

The invention belongs to the field of warehouse goods allocation identification, and particularly relates to a warehouse goods allocation detection method capable of identifying a position angle, which comprises the following steps of: 1) carrying out camera angle and scale calibration on a camera on a transfer robot so as to convert image pixel coordinates into real scale coordinates; 2) obtaining an offset value between the image coordinate system and a transfer robot coordinate system, namely obtaining a corresponding relation between the image coordinate system and the transfer robot coordinate system, and further obtaining a cargo position under the transfer robot coordinate system; 3) according to the cargo position under the coordinate system of the transfer robot, obtaining a boundary frame of the cargo through an image detection algorithm, and processing a sub-image in the boundary frame to obtain a cargo angle; according to the method, the flexibility of warehouse management can be greatly improved when goods allocation detection is carried out.

Description

Technical Field

[0001] The invention belongs to the field of warehouse cargo position identification, in particular to a warehouse cargo position detection method capable of identifying position angles. Background Art

[0002] In unmanned logistics warehouses, a cargo location detection system is needed to provide warehouse status information. The cargo location detection system obtains information within the field of view through a high-positioned overhead camera, and uses some feature recognition methods to determine the status of certain cargo locations. Currently, feature recognition methods include traditional image feature extraction and deep learning methods, and feature matching is performed on real-time detection results to calculate the type of goods.

[0003] The cargo location detection system can calculate the position of the goods in the image in real time. However, in order to establish a connection with the automatic handling robot that picks up the goods, the cargo location detection system needs to configure the area of ​​each cargo location in the image in advance, and the angle cannot be placed arbitrarily. It is impossible to effectively calculate all position information within the camera's field of view. Summary of the invention

[0004] The purpose of the present invention is to provide a warehouse cargo position detection method that can identify position angles, and a method for calculating the cargo position in the coordinate system of a handling robot, so that the overall placement of the cargo is more flexible.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a warehouse cargo location detection method capable of identifying position angles, comprising the following steps:

[0006] 1) Calibrate the camera angle and scale of the camera on the handling robot to convert the image pixel coordinates into real scale coordinates;

[0007] 2) Obtaining the offset value between the image coordinate system and the handling robot coordinate system, that is, obtaining the corresponding relationship between the image coordinate system and the handling robot coordinate system, and then obtaining the cargo position in the handling robot coordinate system;

[0008] 3) According to the cargo position in the handling robot coordinate system, the cargo boundary box is obtained through the image detection algorithm, and the sub-image in the boundary box is processed to obtain the cargo angle;

[0009] 4) According to steps 1) to 3), the position and angle of the cargo in the coordinate system of the handling robot are obtained.

[0010] The step 1) is specifically:

[0011] 1-1) Select four points on the ground within the camera's field of view that are not on the same straight line, and measure the relative positions of the true scale values ​​of the four points, that is: the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4);

[0012] 1-2) Find the positions of the four corresponding points in the image coordinate system from the camera image, and calculate the homography transformation matrix of the camera relative to the ground through these four sets of corresponding points;

[0013] 1-3) Decompose the homography transformation matrix to obtain the scale factor;

[0014] 1-4) Perform affine transformation of the camera image through the homography transformation matrix, and combine it with the scale factor to convert the image pixel coordinates into real scale coordinates.

[0015] The steps 1-2) are specifically:

[0016] Through these four sets of corresponding points, the homography transformation matrix of the camera relative to the ground is calculated. The homography transformation matrix is ​​a 3-row 3-column matrix, set to H(h11-h33). In the homography matrix, h33 is always equal to 1, that is:

[0017]

[0018] The steps 1-3) are specifically:

[0019] a. Multiply the homography matrix H by the inverse of the camera matrix on the left and multiply it by the camera matrix on the right to obtain the matrix A;

[0020] b. Perform SVD decomposition on A to obtain unitary matrices U and V;

[0021] c. Multiply the transposed determinant of U by the determinant of V to get the scale factor s.

[0022] The steps 1-4) are specifically:

[0023] Use this homography transformation matrix to perform affine transformation on the camera image. The pixel coordinate point (Pix, Piy) in the affine transformed image is multiplied by the scale factor s, that is, the point (s*Pix, s*Piy) is the image pixel coordinate converted to the real scale coordinate.

[0024] The offset value between the image coordinate system and the handling robot coordinate system includes: an X-axis direction offset Delta_X, a Y-axis direction offset Delta_Y, and a sine value Sin_Theta and a cosine value Cos_Theta of the angle offset.

[0025] The step 2) comprises the following steps:

[0026] 2-1) Select two points in the image after affine transformation in step 1-4) and record the coordinates (C1_X, C2_Y) and (C2_X, C2_Y) in the image coordinate system;

[0027] 2-2) The transport robot drives to these two points, and records the coordinates of these two points in the transport robot as (V1_X, V2_Y) and (V2_X, V2_Y), thereby obtaining the corresponding relationship between the two points, namely:

[0028]

[0029] 2-3) Calculate the four offsets to be calculated: Delta_X, Delta_Y, Sin_Theta and Cos_Theta;

[0030] 2-4) Assume that the cargo position in the image coordinate system is point (Pcx, Pcy), and the cargo position in the robot coordinate system (Prx, Pry) is obtained by the following formula, that is:

[0031]

[0032] The step 3) comprises the following steps:

[0033] 3-1) Obtain the bounding box of the goods through the image detection algorithm, extract the sub-image in the bounding box, perform LSD straight line detection, and record the length and angle of the straight line;

[0034] 3-2) Extract the straight lines whose length is greater than 0.5 times the long side of the Bounding Box from this set of straight lines, and record the angles of these straight lines;

[0035] 3-3) Since the goods are placed on the pallet, and the pallet and its load are placed in a rectangular space, the straight line is reduced from a 360° interval to a 90° interval, that is:

[0036] abgle=atan(tan(atan(x)×2))

[0037] Where x is an angle value in the interval (0, 360°);

[0038] 3-4) Arrange the obtained angle values ​​in the interval (0, 90°) from small to large, set a sliding window to traverse, and record the number of straight lines in the window;

[0039] 3-5) After the traversal is completed, the window positions with the largest number of windows are extracted, and the average of all the angle values ​​is taken, that is, the angle of the cargo in the coordinate system of the handling robot.

[0040] The present invention has the following beneficial effects and advantages:

[0041] 1. The flexibility of warehouse management can be greatly improved by adopting the method of the present invention to perform cargo location detection.

[0042] 2. The present invention does not need to configure the area of ​​each cargo location in the image in advance, and can effectively calculate all position information for goods placed at any angle.

[0043] 3. The present invention solves the position and angle of the cargo in the coordinate system of the handling robot respectively, and the detection method is simpler and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A method flow chart of a preferred embodiment of the present invention;

[0045] Figure 2 is an image before affine transformation in a preferred embodiment of the present invention;

[0046] Figure 3 This is an image after affine transformation in a preferred embodiment of the present invention, and the coordinate system is calibrated at this time;

[0047] Figure 4 This is the final calculation result in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0049] like Figure 1 FIG. 1 is a flowchart of a method according to a preferred embodiment of the present invention. The present invention provides a warehouse cargo location detection method capable of identifying position angles, comprising the following steps:

[0050] 1) Calibrate the camera angle and scale of the camera on the handling robot to convert the image pixel coordinates into real scale coordinates;

[0051] 1. First, calibrate the camera angle and scale. Select four points on the ground within the camera's field of view that are not on the same straight line, and measure the relative positions of the four points to obtain the true scale values. For example, the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4). Then find the positions of the corresponding four points in the image coordinate system from the camera image. For example, the coordinates of the first point in the image are (Pcx1, Pcy1), the second point is (Pcx2, Pcy2), the third point is (Pcx3, Pcy3), and the fourth point is (Pcx4, Pcy4). Through these four groups of corresponding points, the homography transformation matrix of the camera relative to the ground can be calculated. The homography transformation matrix is ​​a 3-row 3-column matrix, set to H(h11-h33). The calculation method is as follows. In the homography matrix, h33 is always equal to 1:

[0052]

[0053] Using SVD decomposition, the eight unknowns in h11-h32 can be solved to obtain the homography transformation matrix. Decomposing this homography transformation matrix can obtain the scale factor s. Specifically, the homography matrix H is multiplied by the inverse of the camera matrix on the left and the camera matrix on the right to obtain the matrix A. Then, A is decomposed by SVD to obtain the unitary matrices U and V. The scale factor s is obtained by multiplying the transposed determinant of U by the determinant of V. This homography transformation matrix is ​​used to perform an affine transformation on the camera image. The pixel coordinate point (Pix, Piy) in the image after the affine transformation is multiplied by the scale factor s, that is, the point (s*Pix, s*Piy) is the image pixel coordinate converted to the real scale coordinate.

[0054] 2) Obtaining the offset value between the image coordinate system and the handling robot coordinate system, that is, obtaining the corresponding relationship between the image coordinate system and the handling robot coordinate system, and then obtaining the cargo position in the handling robot coordinate system;

[0055] 2. The calibration between the image coordinate system and the handling robot coordinate system requires the calculation of the X-axis offset (Delta_X), Y-axis offset (Delta_Y), and the sine value (Sin_Theta) and cosine value (Cos_Theta) of the angle offset, a total of four offset values ​​to be calculated. Select two points in the image after the above affine transformation, record the coordinates in the image coordinate system (C1_X, C2_Y and C2_X, C2_Y), and drive the handling robot to these two points, such as Figure 3As shown, the coordinates of the two points in the handling robot (V1_X, V2_Y and V2_X, V2_Y) are recorded at the same time, so as to obtain the corresponding relationship between the two points. Since the two points in the image coordinate system and the two points in the handling robot coordinate system and the above four offsets to be calculated have the following calculation relationship:

[0056]

[0057] 3. The calculation can obtain the four offsets to be calculated, namely Delta_X, Delta_Y, Sin_Theta and Cos_Theta. The above formula is also the correspondence between the image coordinate system and the handling robot coordinate system. The position in the handling robot coordinate system can be obtained by calculation. Assuming that the cargo position in the image coordinate system is point (Pcx, Pcy), the cargo position in the robot coordinate system (Prx, Pry) can be calculated by the following formula:

[0058]

[0059] 3) According to the cargo position in the handling robot coordinate system, the cargo boundary box is obtained through the image detection algorithm, and the sub-image in the boundary box is processed to obtain the cargo angle;

[0060] 4. Finally, calculate the angle. The bounding box of the goods can be obtained through the image detection algorithm. The sub-image in the bounding box is extracted, and LSD straight line detection is performed, and the length and angle of the straight line are recorded. The straight lines whose length is greater than 0.5 times the long side of the bounding box are extracted from this group of straight lines, and the angles of these straight lines are recorded. Since most goods rely on pallets, and pallets and their cargo are mostly placed in rectangular spaces, the following formula can be used to reduce the straight line from a 360° interval to a 90° interval, where x is the angle value within the 360° interval:

[0061] abgle=atan(tan(atan(x)×2))

[0062] Arrange the obtained angle values ​​in the 90° interval from small to large, then set a suitable sliding window for traversal, and record the number of straight lines in the window. When the traversal is completed, the window position with the largest number of straight lines will be extracted, and the average value of all the angle values ​​in it is the cargo angle, such as Figure 4 shown.

[0063] 4) According to steps 1) to 3), the position and angle of the cargo in the coordinate system of the handling robot are obtained.

[0064] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A warehouse cargo location detection method capable of identifying position angles, characterized in that: The following steps are involved: 1) Calibrate the camera angle and scale of the camera on the handling robot to convert the image pixel coordinates into real scale coordinates; 2) Obtaining the offset value between the image coordinate system and the handling robot coordinate system, that is, obtaining the corresponding relationship between the image coordinate system and the handling robot coordinate system, and then obtaining the cargo position in the handling robot coordinate system; 3) According to the cargo position in the handling robot coordinate system, the cargo boundary box is obtained through the image detection algorithm, and the sub-image in the boundary box is processed to obtain the cargo angle; 4) According to steps 1) to 3), the position and angle of the cargo in the coordinate system of the handling robot are obtained.

2. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The step 1) is specifically: 1-1) Select four points on the ground within the camera's field of view that are not on the same straight line, and measure the relative positions of the true scale values ​​of the four points, that is: the coordinates of the first point on the ground are (Pgx1, Pgy1), the second point is (Pgx2, Pgy2), the third point is (Pgx3, Pgy3), and the fourth point is (Pgx4, Pgy4); 1-2) Find the positions of the four corresponding points in the image coordinate system from the camera image, and calculate the homography transformation matrix of the camera relative to the ground through these four sets of corresponding points; 1-3) Decompose the homography transformation matrix to obtain the scale factor; 1-4) Perform affine transformation of the camera image through the homography transformation matrix, and combine it with the scale factor to convert the image pixel coordinates into real scale coordinates.

3. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The steps 1-2) are specifically: Through these four sets of corresponding points, the homography transformation matrix of the camera relative to the ground is calculated. The homography transformation matrix is ​​a 3-row 3-column matrix, set to H(h11-h33). In the homography matrix, h33 is always equal to 1, that is:

4. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The steps 1-3) are specifically: a. Multiply the homography matrix H by the inverse of the camera matrix on the left and multiply it by the camera matrix on the right to obtain the matrix A; b. Perform SVD decomposition on A to obtain unitary matrices U and V; c. Multiply the transposed determinant of U by the determinant of V to get the scale factor s.

5. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The steps 1-4) are specifically: Use this homography transformation matrix to perform affine transformation on the camera image. The pixel coordinate point (Pix, Piy) in the affine transformed image is multiplied by the scale factor s, that is, the point (s*Pix, s*Piy) is the image pixel coordinate converted to the real scale coordinate.

6. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The offset value between the image coordinate system and the handling robot coordinate system includes: an X-axis direction offset Delta_X, a Y-axis direction offset Delta_Y, and a sine value Sin_Theta and a cosine value Cos_Theta of the angle offset.

7. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The step 2) comprises the following steps: 2-1) Select two points in the image after affine transformation in step 1-4) and record the coordinates (C1_X, C2_Y) and (C2_X, C2_Y) in the image coordinate system; 2-2) The transport robot drives to these two points, and records the coordinates of these two points in the transport robot as (V1_X, V2_Y) and (V2_X, V2_Y), thereby obtaining the corresponding relationship between the two points, namely: 2-3) Calculate the four offsets to be calculated: Delta_X, Delta_Y, Sin_Theta and Cos_Theta; 2-4) Assume that the cargo position in the image coordinate system is point (Pcx, Pcy), and the cargo position in the robot coordinate system (Prx, Pry) is obtained by the following formula, that is:

8. A warehouse cargo location detection method capable of identifying position angles according to claim 1, characterized in that: The step 3) comprises the following steps: 3-1) Obtain the bounding box of the goods through the image detection algorithm, extract the sub-image in the bounding box, perform LSD straight line detection, and record the length and angle of the straight line; 3-2) Extract the straight lines whose length is greater than 0.5 times the long side of the Bounding Box from this set of straight lines, and record the angles of these straight lines; 3-3) Since the goods are placed on the pallet, and the pallet and its load are placed in a rectangular space, the straight line is reduced from a 360° interval to a 90° interval, that is: angle = atan(tan(atan(x)×2)) Where x is an angle value in the interval (0, 360°); 3-4) Arrange the obtained angle values ​​in the interval (0, 90°) from small to large, set a sliding window to traverse, and record the number of straight lines in the window; 3-5) After the traversal is completed, the window positions with the largest number of windows are extracted, and the average of all the angle values ​​is taken, that is, the angle of the cargo in the coordinate system of the handling robot.

Images