A method for determining the pose of a spreader of an intelligent gantry crane

Abstract

The application discloses a kind of gantry crane's lifting appliance pose determination method of intelligence, it includes constructing gantry crane digital model and constructing camera module, and then obtain the video sequence of gantry crane lifting appliance descent angle of view and construct gantry crane view data training set;To the angle column calibration of gantry crane view data training set, obtain calibration angle column value;View data training set is input into neural network, and angle column detection is carried out using loss function, and angle column position value is obtained;Segment angle column position value, and obtain angle column pixel space position;The depth value of lifting appliance is obtained using three-point model, and the length value of lifting appliance, width value is estimated by scale estimation, for adjusting lifting appliance pose;Through the comparison of angle column pixel space position, the depth value of lifting appliance, length value and width value of gantry crane view data training set, whether the four angle columns of lifting appliance are aligned container is judged.The present application reduces positioning error by angle column calibration and neural network training, and does not need to rely on artificial visual inspection to adjust lifting appliance.

Description

Technical Field

[0001] This invention relates to the field of intelligent gantry cranes, and more particularly to a method for determining the position and posture of the lifting device of an intelligent gantry crane. Background Technology

[0002] Currently, the placement of containers in container yards using gantry cranes relies entirely on manual operation by the drivers. This manual observation and judgment of container areas and locations is prone to errors, leading to inaccuracies in container stacking information and causing management problems. Furthermore, the placement of containers by gantry cranes depends entirely on the truck driver's visual assessment. Without accurate positioning information between the gantry crane's spreader and the containers in the yard, drivers often have to adjust the gantry crane's position by moving the wheels when they observe discrepancies between the lifted and stacked containers. This slows down loading and unloading speeds and impacts efficiency. Summary of the Invention

[0003] This invention provides a method for determining the position of the lifting device of an intelligent gantry crane, so as to overcome the technical problems such as relying on manual visual estimation for the stacking position of containers by gantry cranes.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] A method for determining the position and orientation of a lifting device for an intelligent gantry crane includes the following steps:

[0006] Step 1: Construct a digital model of the gantry crane;

[0007] Step 2: Using the digital model of the gantry crane, install a virtual camera in the digital model of the gantry crane, acquire video sequences of the descent view of the gantry crane spreader through the virtual camera, and construct a training set of gantry crane view data;

[0008] Step 3: Perform corner column calibration on the gantry crane view data training set to obtain the calibration corner column values;

[0009] Step 4: Input the gantry crane view data training set with calibrated corner column values ​​into the neural network, and use the loss function to detect corner columns and obtain the corner column position values;

[0010] Step 5: Segment the corner pillar position values ​​to obtain the pixel spatial position of the corner pillars;

[0011] Step 6: Using the spatial position of the corner column pixels, obtain the depth value of the lifting device using a three-point model. Estimate the length and width values ​​of the lifting device through scale estimation to adjust the pose of the lifting device.

[0012] Step 7: By comparing the pixel spatial position of the diagonal columns, the depth value, length value, and width value of the spreader in the gantry crane view data training set, determine whether the four corner columns of the spreader are aligned with the container. If they are aligned with the container, the spreader pose determination is completed. If they are not aligned with the container, return to step 6.

[0013] Furthermore, the specific formula for calculating the corner prism position value obtained in step 4 is as follows:

[0014]

[0015] Where loss(object) represents the training set of gantry crane view data with calibrated corner column values, and K represents the number of grid cells. w represents the probability that a corner pillar has a mask. i h represents the calibrated pixel width of the i-th corner pillar. i x represents the calibrated pixel height of the i-th corner pillar. i The x-coordinate of the calibrated pixel of the i-th corner column. The predicted x-coordinate of the i-th corner prism, y i The ordinate of the i-th corner prism is represented by its ordinate. Represents the predicted ordinate of the i-th corner prism. This represents the predicted pixel width of the i-th corner peg. C represents the predicted image height of the i-th corner prism. i This represents the labeling type of the i-th corner prism. This represents the prediction type for the i-th corner prism. Let p represent the probability of no corner pillar occlusion, c represent the predicted probability of each class, classes represent the set of all classes, and p represents the probability of no corner pillar occlusion. i (c) represents the probability of classifying the corner prism. This represents the predicted probability of the corner column category.

[0016] Furthermore, the specific steps for obtaining the depth value of the lifting device using the three-point model in step 6 are as follows:

[0017] Step 6.1: Obtain the coordinates of any three corner pins in the pixel space. The formula for calculating the coordinates of any three corner pins in the pixel space is:

[0018]

[0019]

[0020]

[0021] There are four corner column coordinate points in the corner column pixel space; O, A and B are any three corner column coordinate points; X1, Y1 and Z1 are the coordinate components of point O; X2, Y2 and Z2 are the coordinate components of point A; X3, Y3 and Z3 are the coordinate components of point B;

[0022] Position(Z1) represents the coordinates of point O; Position(Z2) represents the coordinates of point A;

[0023] Position(Z3) represents the coordinates of point B.

[0024] Step 6.2: Obtain the Euclidean distance between the three corner prisms and the angle formed by the corner prisms based on their coordinate values. The formulas for calculating the Euclidean distance and angle are as follows:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] Where dis(Z1, Z2) is the Euclidean distance between point O and point A; dis(Z1, Z3) is the Euclidean distance between point O and point B; |OA| is the absolute distance between point O and point A; |OB| is the absolute distance between point O and point B; Let O be the vector between points A and O; Let O be the vector between points B; let cos∠AOB be the vector. with vector The angle value formed;

[0031] Step 6.3: Obtain the depth value of the lifting device based on the Euclidean distance between the corner posts and the angle formed by the corner posts. The formula for calculating the depth value of the lifting device is as follows:

[0032] angle(Z1, Z2, Z3)=cos∠AOB

[0033]

[0034] Where angle(Z1, Z2, Z3) is the cosine value of cos∠AOB, and g(Z1, Z2, Z3) is the depth value of the lifting device.

[0035] Furthermore, in step 4, the training set of gantry crane view data with calibrated corner column values ​​is input into the ResNet-50 neural network.

[0036] Furthermore, in step 2, a camera module is constructed below the spreader in the digital model of the gantry crane.

[0037] Beneficial effects: This invention acquires video sequences of the descent angle of the spreader using a virtual camera, performs corner column calibration, and uses a loss function trained by a neural network to detect the corner columns and obtain their position values, thereby reducing positioning errors and eliminating the need for manual visual adjustment of the spreader. Furthermore, by using a three-point model to locate the distance and position between the corner columns of the spreader, the accuracy of identification is further improved, thereby increasing loading and unloading efficiency. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of the method for determining the position of the lifting device according to the present invention;

[0040] Figure 2 This is a schematic diagram of the digital model of the gantry crane of the present invention;

[0041] Figure 3 This is a schematic diagram of a video sequence showing the descent angle of the lifting device of the present invention.

[0042] Among them, 1. Lifting gear. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This embodiment provides a method for determining the position and posture of the lifting device of an intelligent gantry crane, such as... Figure 1-3 This includes the following steps:

[0045] Step 1: Construct a digital model of the gantry crane using 3ds Max software;

[0046] Step 2: Using the digital model of the gantry crane, install a virtual camera in the digital model of the gantry crane, acquire video sequences of the descent view of the gantry crane spreader through the virtual camera, and construct a training set of gantry crane view data;

[0047] Step 3: Perform manual corner column calibration on the gantry crane view data training set to obtain the calibration corner column values;

[0048] Step 4: Input the gantry crane view data training set with calibrated corner column values ​​into the neural network, and use the loss function to detect corner columns and obtain the corner column position values;

[0049] Step 5: Segment the corner pillar position values ​​to obtain the pixel spatial position of the corner pillars;

[0050] Step 6: Using the spatial position of the corner column pixels, obtain the depth value of the lifting device using a three-point model. Estimate the length and width values ​​of the lifting device through scale estimation to adjust the pose of the lifting device.

[0051] Step 7: By comparing the pixel space values ​​of the diagonal columns, the depth, length, and width of the spreader in the gantry crane view data training set, determine whether the four corner columns of the spreader are aligned with the container. If aligned with the container, the spreader pose is determined; otherwise, return to Step 6. Specifically, since the virtual camera is fixed on the spreader, if the spreader is aligned with the container, the detected length, width, and depth of the container are relatively fixed in pixel space. When aligned, the center of the detected rectangle should be in the exact center of the field of view. The long distance of the rectangle is 100 pixels on each side, and the wide distance is 50 pixels vertically. Compare the measured values ​​with the values ​​in the database. If the error is within 10 units, it is determined that the spreader is aligned with the container; otherwise, it is determined that the spreader is not aligned with the container.

[0052] In a specific embodiment, the specific calculation formula for obtaining the corner column position value in step 4 is as follows:

[0053]

[0054]

[0055] Where loss(object) represents the training set of gantry crane view data with calibrated corner column values, and K represents the number of grid cells. w represents the probability of a corner pillar being masked (1 if there is a corner pillar in the region, 0 if there is no corner pillar). i h represents the calibrated pixel width of the i-th corner pillar. i x represents the calibrated pixel height of the i-th corner pillar. i The x-coordinate of the calibrated pixel of the i-th corner column. The predicted x-coordinate of the i-th corner prism, y i The ordinate of the i-th corner prism is represented by its ordinate. Represents the predicted ordinate of the i-th corner prism. This represents the predicted pixel width of the i-th corner peg. C represents the predicted image height of the i-th corner prism. i This represents the labeling type of the i-th corner prism. This represents the prediction type for the i-th corner prism. represents the probability of no corner pillar occlusion (1 when there is no corner pillar and 0 when there is a corner pillar), c represents the predicted probability of the class, classes represents the set of all classes, and p i (c) represents the probability of classifying the corner prism (0 or 1). This represents the predicted probability of the corner prism category (a real number between 0 and 1).

[0056] In a specific embodiment, the steps for obtaining the depth value of the lifting device using the three-point model in step 6 are as follows:

[0057] Step 6.1: Obtain the coordinates of any three corner pins in the pixel space. The formula for calculating the coordinates of any three corner pins in the pixel space is:

[0058]

[0059]

[0060]

[0061] There are four corner column coordinate points in the corner column pixel space; O, A and B are any three corner column coordinate points; X1, Y1 and Z1 are the coordinate components of point O; X2, Y2 and Z2 are the coordinate components of point A; X3, Y3 and Z3 are the coordinate components of point B;

[0062] Position(Z1) represents the coordinates of point O; Position(Z2) represents the coordinates of point A;

[0063] Position(Z3) represents the coordinates of point B.

[0064] Step 6.2: Obtain the Euclidean distance between the three corner prisms and the angle formed by the corner prisms based on their coordinate values. The formulas for calculating the Euclidean distance and angle are as follows:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Where dis(Z1, Z2) is the Euclidean distance between point O and point A; dis(Z1, Z3) is the Euclidean distance between point O and point B; |OA| is the absolute distance between point O and point A; |OB| is the absolute distance between point O and point B; Let O be the vector between points A and O; Let O be the vector between points B; let cos∠AOB be the vector. with vector The angle value formed;

[0071] Step 6.3: Obtain the depth value of the lifting device based on the Euclidean distance between the corner posts and the angle formed by the corner posts. The formula for calculating the depth value of the lifting device is as follows:

[0072] angle(Z1, Z2, Z3)=cos∠AOB

[0073]

[0074] Where angle(Z1, Z2, Z3) is the cosine value of cos∠AOB, and g(Z1, Z2, Z3) is the depth value of the lifting device.

[0075] In a specific embodiment, step 4 involves inputting the training set of gantry crane view data with calibrated corner column values ​​into the ResNet-50 neural network.

[0076] In a specific embodiment, step 2 involves constructing a camera module below the lifting device in the digital model of the gantry crane.

[0077] like Figure 3 As shown, if the coordinates of the three corner pillars are known as 0(76, 234), A(1007, 234), and B(74, 418), and since the pixel information is known, the coordinates of these three points are functions of Z1, Z2, and Z3, respectively. Substituting the corner pillar coordinates and the set focal length of 1200mm, we can obtain:

[0078]

[0079]

[0080]

[0081] The Euclidean distance between the three corner prisms and the angle formed by the corner prisms are obtained from their coordinate values. The formulas for calculating the Euclidean distance and angle are as follows:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] Relationship between spreader depth value and vector:

[0090]

[0091] Solve the above equations simultaneously:

[0092] The depth value of the lifting device can be calculated as follows:

[0093] Z = 0.10000m

[0094] Where dis(Z1, Z2) is the Euclidean distance between point 0 and point A; dis(Z1, Z3) is the Euclidean distance between point 0 and point B; The absolute distance between point 0 and point A; The absolute distance between point 0 and point B; Let A be the vector between point 0 and point A; Let be the vector between point 0 and point B; let cos∠AOB be the vector. with vector The angle value formed; Z is the depth value of the lifting device, i.e., angle(Z1, Z2, Z3);

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining the pose of a spreader of an intelligent gantry crane, characterized in that, Includes the following steps: Step 1: Construct a digital model of the gantry crane; Step 2: Construct a camera module in the digital model of the gantry crane, acquire video sequences of the gantry crane's descent angle through the camera module, and construct a training set of gantry crane view data; Step 3: Perform corner column calibration on the gantry crane view data training set to obtain the calibration corner column values; Step 4: Input the gantry crane view data training set with calibrated corner column values ​​into the neural network, and use the loss function to detect corner columns and obtain the corner column position values; Step 5: Segment the corner pillar position values ​​to obtain the pixel spatial position of the corner pillars; Step 6: Using the spatial position of the corner column pixels, obtain the depth value of the lifting device using a three-point model. Estimate the length and width values ​​of the lifting device through scale estimation to adjust the pose of the lifting device. Step 7: By comparing the pixel spatial position of the diagonal columns, the depth value, length value, and width value of the spreader in the gantry crane view data training set, determine whether the four corner columns of the spreader are aligned with the container. If they are aligned with the container, the spreader pose determination is completed. If they are not aligned with the container, return to step 6.

2. The method for determining the lifting device position of an intelligent gantry crane as described in claim 1, characterized in that, The specific calculation formula for obtaining the corner prism position value in step 4 is as follows: Where loss(object) represents the training set of gantry crane view data with calibrated corner column values, and K represents the number of grid cells. w represents the probability that a corner pillar has a mask. i h represents the calibrated pixel width of the i-th corner pillar. i x represents the calibrated pixel height of the i-th corner pillar. i The x-coordinate of the calibrated pixel of the i-th corner column. The predicted x-coordinate of the i-th corner prism, y i The ordinate of the i-th corner prism is represented by its ordinate. Represents the predicted ordinate of the i-th corner prism. This represents the predicted pixel width of the i-th corner peg. C represents the predicted image height of the i-th corner prism. i This represents the labeling type of the i-th corner prism. This represents the prediction type for the i-th corner prism. Let p represent the probability of no corner pillar occlusion, c represent the predicted probability of each class, classes represent the set of all classes, and p represents the probability of no corner pillar occlusion. i (c) represents the probability of classifying the corner prism. This represents the predicted probability of the corner column category.

3. The method for determining the lifting device position of an intelligent gantry crane as described in claim 1, characterized in that, The specific steps for obtaining the depth value of the lifting device using the three-point model in step 6 are as follows: Step 6.1: Obtain the coordinates of any three corner pins in the pixel space. The formula for calculating the coordinates of any three corner pins in the pixel space is: There are four corner column coordinate points in the corner column pixel space; O, A and B are any three corner column coordinate points; X1, Y1 and Z1 are the coordinate components of point O; X2, Y2 and Z2 are the coordinate components of point A; X3, Y3 and Z3 are the coordinate components of point B; Position(Z1) represents the coordinates of point O; Position(Z2) represents the coordinates of point A; Position(Z3) represents the coordinates of point B. Step 6.2: Obtain the Euclidean distance between the three corner prisms and the angle formed by the corner prisms based on their coordinate values. The formulas for calculating the Euclidean distance and angle are as follows: Where dis(Z1,Z2) is the Euclidean distance between point O and point A; dis(Z1,Z3) is the Euclidean distance between point O and point B; |OA| is the absolute distance between point O and point A; |OB| is the absolute distance between point O and point B; Let O be the vector between points A and O; Let O be the vector between points B; let cos∠AOB be the vector. with vector The angle value formed; Step 6.3: Obtain the depth value of the lifting device based on the Euclidean distance between the corner posts and the angle formed by the corner posts. The formula for calculating the depth value of the lifting device is as follows: angle(Z1,Z2,Z3)=cos∠AOB Where angle(Z1,Z2,Z3) is the cosine value of cos∠AOB, and g(Z1,Z2,Z3) is the depth value of the lifting device.

4. The method for determining the lifting device position of an intelligent gantry crane as described in claim 1, characterized in that: In step 4, the training set of gantry crane view data with calibrated corner column values ​​is input into the ResNet-50 neural network.

5. The method for determining the lifting device position of an intelligent gantry crane as described in claim 1, characterized in that: In step 2, a camera module is built below the spreader in the digital model of the gantry crane.

Images