Portal crane grab bucket positioning method based on machine vision and three-dimensional point cloud fusion
Through the integration of machine vision and three-dimensional point cloud, combined with camera and three-dimensional lidar, the precise positioning of the door machine grab is achieved, solving the problem of inaccurate positioning caused by manual observation, and improving the detection accuracy of the grab position.
Patent Information
- Application Number
- CN202410956532.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the grab positioning of the door machine is inaccurate, manual observation is prone to fatigue and there is a blind spot in the field of view, resulting in inaccurate detection of the grab position.
Using a method based on the fusion of machine vision and three-dimensional point clouds, images are collected through cameras and deep learning algorithms are used for real-time monitoring, combined with three-dimensional lidar scanning of the grab area, analyze the European-style distance of the three-dimensional point cloud, and detect and identify the three-dimensional position of the grab. And fuse the two to determine the exact position of the grab.
The precise positioning of the door machine grab is achieved, the inaccurate positioning problem caused by manual observation is solved, and the detection accuracy of the grab position is improved.
Smart Images

Figure CN118918178A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of machine vision, and in particular to a gantry crane grab positioning system based on the fusion of machine vision and three-dimensional point cloud. Background Art
[0002] Ports are distribution centers for goods, and gantry cranes are used for loading and unloading of goods. Gantry cranes in bulk cargo ports mainly use grab buckets to lift and move bulk goods such as ore and coal. Gantry cranes have high load capacity and a large operating range, and can efficiently complete the loading and unloading of goods.
[0003] In the process of crane operation, the grab bucket is an important grabbing device. The positioning of the grab bucket is very critical for the intelligent control and safety protection of the gantry crane. Therefore, the positioning of the grab bucket is very important. In the existing technology, the position of the grab bucket is mainly observed by the operator on site or through a camera. However, manual observation is prone to fatigue and there are blind spots in the field of vision, resulting in inaccurate positioning of the grab bucket. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a gantry crane grab positioning method based on machine vision and three-dimensional point cloud fusion in view of the deficiencies in the prior art.
[0005] The technical solution of the present invention is as follows:
[0006] A method for positioning a gantry crane grab bucket based on machine vision and three-dimensional point cloud fusion, comprising the following steps:
[0007] A1. Collect images through the camera, use the deep learning algorithm to monitor the grab bucket in the collected images in real time, and determine the grab bucket's position according to its position in the video;
[0008] A2 uses 3D laser radar to scan the grab area, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab;
[0009] A3 combines the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab;
[0010] A4 converts the grab area coordinates (x3, y3, z3) in the radar three-dimensional coordinate system into three-dimensional coordinates in the on-site ground coordinate system.
[0011] The method, in the step A1, comprises the following steps:
[0012] A11 uses the camera to capture images;
[0013] A12 uses deep learning methods to analyze video images in real time and determine the position of the grab in the image;
[0014] A13 determines the position area of the grab bucket in the three-dimensional space in the three-dimensional coordinate system of the camera according to the pixel position of the grab bucket in the image;
[0015] Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x , fy, then the position area of the object in three-dimensional space is (x1, y1, z1).
[0016]
[0017] The method, in the step A2, comprises the following steps:
[0018] A21 uses 3D LiDAR to scan the grab area;
[0019] A22 analyzes 3D point clouds in real time and uses adaptive spatial clustering methods to divide suspected grab areas;
[0020] Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T;
[0021] Scan all point cloud data, and form a temporary cluster with points greater than MinPoints within the radius R. This point cloud is called the core point.
[0022] For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster. Repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).
[0023] The method, in the step A3, comprises the following steps:
[0024] The position area (x1, y1, z1) of the grab in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera; the center point (x 2j ,y 2j ,z 2j ), is in the radar three-dimensional coordinate system; to achieve accurate positioning, the two coordinate systems need to be transformed;
[0025] This system converts the camera coordinate system into a three-dimensional point cloud coordinate system;
[0026] Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is
[0027]
[0028] Will Substitute (x1, y1, z1) into the formula
[0029] Get (x 1j ,y 1j ,z 1j );
[0030]
[0031] Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) with the shortest Euclidean distance and the smallest z2j is taken as the grab area (x3, y3, z3);
[0032] Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).
[0033] The method, in the step A4, comprises the following steps:
[0034] The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction;
[0035] Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω;
[0036] Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is
[0037]
[0038] Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:
[0039]
[0040] The beneficial effects of the present invention are as follows: if only a camera is used for grab identification, only the spatial state of the grab can be determined, but the specific position of the grab cannot be determined; if only three-dimensional laser radar scanning is used for grab identification, the grab cannot be accurately identified when the grab is in contact with the ground or material. The above-mentioned scheme of the present invention is adopted to determine the position of the gantry crane grab by adopting the fusion method of vision and three-dimensional point cloud. The fusion method of vision and three-dimensional point cloud not only solves the above problems, but also realizes the precise positioning of the gantry crane. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the system structure and method;
[0042] 1 upper arm, 2 trunk bridge; DETAILED DESCRIPTION
[0043] The present invention is described in detail below in conjunction with specific embodiments.
[0044] This embodiment provides a method for positioning a gantry crane grab bucket in a gantry crane grab bucket positioning system based on machine vision and three-dimensional point cloud fusion. The gantry crane has three states: standby state, operating state, and stop state. When the gantry crane is in the operating state, the grab bucket will be lifted, lowered, and translated. The grab bucket will transport cargo in the air or grab and release cargo close to the ground. When the gantry crane is in the standby state or the stop state, the grab bucket will be parked on the ground and integrated with the cargo on the ground. In summary, the grab bucket has two states: in the air and on the ground.
[0045] This system vertically installs a three-dimensional laser radar and a camera at the head platform of the elephant trunk bridge. The camera determines the orientation of the grab bucket, and then the three-dimensional laser radar is used to position the grab bucket in three dimensions, thereby achieving accurate three-dimensional spatial positioning of the grab bucket.
[0046] 1. Collect images through the camera, use the deep learning algorithm to monitor the grab in the collected images in real time, and determine the grab's position according to its position in the video.
[0047] 1) Use a camera to capture images.
[0048] 2) Through deep learning methods, the video images are analyzed in real time to determine the position of the grab in the image.
[0049] 3) According to the pixel position of the grab in the image, determine the position area of the grab in the three-dimensional space in the three-dimensional coordinate system of the camera.
[0050] Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x , fy, then the position area of the object in three-dimensional space is (x1, y1, z1).
[0051]
[0052] 2. Scan the grab area through 3D LiDAR, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab.
[0053] 1) Use 3D LiDAR to scan the grab area.
[0054] 2) Analyze the 3D point cloud in real time and use the adaptive spatial clustering method of the point cloud data to divide the suspected grab area.
[0055] This system adopts a clustering algorithm based on density space.
[0056] Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T.
[0057] Scan all point cloud data and form a temporary cluster with points within the radius R whose number is greater than MinPoints. This point cloud is called the core point.
[0058] For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster. Repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).
[0059] 3. Combine the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab.
[0060] The position area (x1, y1, z1) of the grab bucket in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera. The center point (x 2j ,y 2j ,z 2j ) is in the radar three-dimensional coordinate system. To achieve accurate positioning, the two coordinate systems need to be transformed.
[0061] This system converts the camera coordinate system into a three-dimensional point cloud coordinate system.
[0062] Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is
[0063]
[0064] Will Substitute (x1, y1, z1) into the formula
[0065] Get (x 1j ,y 1j ,z 1j ).
[0066]
[0067] Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) with the shortest Euclidean distance and the smallest z2j is used as the grab area (x3, y3, z3).
[0068] Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).
[0069] 4. Convert the grab area coordinates (x3, y3, z3) in the radar three-dimensional coordinate system to the three-dimensional coordinates in the on-site ground coordinate system.
[0070] The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction.
[0071] Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω.
[0072] Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is
[0073]
[0074] Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:
[0075]
[0076] The present invention integrates machine vision and three-dimensional point cloud, realizing real-time monitoring and video positioning of the gantry crane grab bucket through machine vision, and realizing accurate positioning of the grab bucket through three-dimensional point cloud data.
[0077] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A method for positioning a gantry crane grab bucket based on machine vision and three-dimensional point cloud fusion, characterized in that: The following steps are involved: A1. Collect images through the camera, use the deep learning algorithm to monitor the grab bucket in the collected images in real time, and determine the grab bucket's position according to its position in the video; A2 uses 3D laser radar to scan the grab area, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab; A3 combines the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab; A4 converts the grab area coordinates in the radar three-dimensional coordinate system into three-dimensional coordinates in the on-site ground coordinate system.
2. The method according to claim 1, characterized in that The step A1 comprises the following steps: A11 uses the camera to capture images; A12 uses deep learning methods to analyze video images in real time and determine the position of the grab in the image; A13 determines the position area of the grab bucket in the three-dimensional space in the three-dimensional coordinate system of the camera according to the pixel position of the grab bucket in the image; Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x 、f y , then the position area of the object in three-dimensional space is (x1, y1, z1); 3. The method according to claim 1, characterized in that: In the step A2, The following steps are involved: A21 uses 3D LiDAR to scan the grab area; A22 analyzes 3D point clouds in real time and uses adaptive spatial clustering methods to divide suspected grab areas; Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T; Scan all point cloud data, and form a temporary cluster with points greater than MinPoints within the radius R. This point cloud is called the core point. For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster; repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).
4. The method according to claim 1, characterized in that: The step A3 comprises the following steps: The position area (x1, y1, z1) of the grab in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera; the center point (x 2j ,y 2j ,z 2j ), is in the radar three-dimensional coordinate system; to achieve accurate positioning, the two coordinate systems need to be transformed; the camera coordinate system is transformed into a three-dimensional point cloud coordinate system; Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is Will Substitute (x1, y1, z1) into the formula Get (x 1j ,y 1j ,z 1j ); Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) are closest in Euclidean distance, and z 2j The smallest point is used as the grab area (x3, y3, z3); Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).
5. The method according to claim 1, characterized in that The step A4 comprises the following steps: The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction; Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω; Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:
Citation Information
Cited By
Gantry crane multi-source point cloud fusion method based on grab bucket positioning and grab bucket direct control method
CN121107261A
