Portal crane grab bucket positioning method based on machine vision and three-dimensional point cloud fusion

Through the integration of machine vision and three-dimensional point cloud, combined with camera and three-dimensional lidar, the precise positioning of the door machine grab is achieved, solving the problem of inaccurate positioning caused by manual observation, and improving the detection accuracy of the grab position.

CN118918178APending Publication Date: 2024-11-08滨沅国科(秦皇岛)智能科技股份有限公司
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410956532.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, the grab positioning of the door machine is inaccurate, manual observation is prone to fatigue and there is a blind spot in the field of view, resulting in inaccurate detection of the grab position.

Method used

Using a method based on the fusion of machine vision and three-dimensional point clouds, images are collected through cameras and deep learning algorithms are used for real-time monitoring, combined with three-dimensional lidar scanning of the grab area, analyze the European-style distance of the three-dimensional point cloud, and detect and identify the three-dimensional position of the grab. And fuse the two to determine the exact position of the grab.

Benefits of technology

The precise positioning of the door machine grab is achieved, the inaccurate positioning problem caused by manual observation is solved, and the detection accuracy of the grab position is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918178A_ABST
    Figure CN118918178A_ABST
Patent Text Reader

Abstract

The invention discloses a portal crane grab bucket positioning method based on machine vision and three-dimensional point cloud fusion, and the method comprises the following steps: A1, collecting an image through a camera, carrying out the real-time monitoring of a grab bucket in the collected image through a deep learning algorithm, and determining the direction of the grab bucket according to the position of the grab bucket in a video; a2, the grab bucket area is scanned through a three-dimensional laser radar, the three-dimensional point cloud Euclidean distance is analyzed, and the three-dimensional position of the grab bucket is detected and recognized; a3, fusing the suspected grab bucket area determined by the three-dimensional point cloud with the grab bucket area determined by vision, and determining the accurate position of the grab bucket; and A4, converting the grab bucket area coordinates under the radar three-dimensional coordinate system into three-dimensional coordinates under the field ground coordinate system. According to the scheme, the position of the grab bucket of the portal crane is determined by adopting a visual and three-dimensional point cloud fusion mode, the problems are solved by adopting the visual and three-dimensional point cloud fusion mode, and accurate positioning of the portal crane is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of machine vision, and in particular to a gantry crane grab positioning system based on the fusion of machine vision and three-dimensional point cloud. Background Art

[0002] Ports are distribution centers for goods, and gantry cranes are used for loading and unloading of goods. Gantry cranes in bulk cargo ports mainly use grab buckets to lift and move bulk goods such as ore and coal. Gantry cranes have high load capacity and a large operating range, and can efficiently complete the loading and unloading of goods.

[0003] In the process of crane operation, the grab bucket is an important grabbing device. The positioning of the grab bucket is very critical for the intelligent control and safety protection of the gantry crane. Therefore, the positioning of the grab bucket is very important. In the existing technology, the position of the grab bucket is mainly observed by the operator on site or through a camera. However, manual observation is prone to fatigue and there are blind spots in the field of vision, resulting in inaccurate positioning of the grab bucket. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a gantry crane grab positioning method based on machine vision and three-dimensional point cloud fusion in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention is as follows:

[0006] A method for positioning a gantry crane grab bucket based on machine vision and three-dimensional point cloud fusion, comprising the following steps:

[0007] A1. Collect images through the camera, use the deep learning algorithm to monitor the grab bucket in the collected images in real time, and determine the grab bucket's position according to its position in the video;

[0008] A2 uses 3D laser radar to scan the grab area, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab;

[0009] A3 combines the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab;

[0010] A4 converts the grab area coordinates (x3, y3, z3) in the radar three-dimensional coordinate system into three-dimensional coordinates in the on-site ground coordinate system.

[0011] The method, in the step A1, comprises the following steps:

[0012] A11 uses the camera to capture images;

[0013] A12 uses deep learning methods to analyze video images in real time and determine the position of the grab in the image;

[0014] A13 determines the position area of ​​the grab bucket in the three-dimensional space in the three-dimensional coordinate system of the camera according to the pixel position of the grab bucket in the image;

[0015] Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x , fy, then the position area of ​​the object in three-dimensional space is (x1, y1, z1).

[0016]

[0017] The method, in the step A2, comprises the following steps:

[0018] A21 uses 3D LiDAR to scan the grab area;

[0019] A22 analyzes 3D point clouds in real time and uses adaptive spatial clustering methods to divide suspected grab areas;

[0020] Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T;

[0021] Scan all point cloud data, and form a temporary cluster with points greater than MinPoints within the radius R. This point cloud is called the core point.

[0022] For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster. Repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).

[0023] The method, in the step A3, comprises the following steps:

[0024] The position area (x1, y1, z1) of the grab in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera; the center point (x 2j ,y 2j ,z 2j ), is in the radar three-dimensional coordinate system; to achieve accurate positioning, the two coordinate systems need to be transformed;

[0025] This system converts the camera coordinate system into a three-dimensional point cloud coordinate system;

[0026] Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is

[0027]

[0028] Will Substitute (x1, y1, z1) into the formula

[0029] Get (x 1j ,y 1j ,z 1j );

[0030]

[0031] Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) with the shortest Euclidean distance and the smallest z2j is taken as the grab area (x3, y3, z3);

[0032] Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).

[0033] The method, in the step A4, comprises the following steps:

[0034] The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction;

[0035] Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω;

[0036] Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is

[0037]

[0038] Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:

[0039]

[0040] The beneficial effects of the present invention are as follows: if only a camera is used for grab identification, only the spatial state of the grab can be determined, but the specific position of the grab cannot be determined; if only three-dimensional laser radar scanning is used for grab identification, the grab cannot be accurately identified when the grab is in contact with the ground or material. The above-mentioned scheme of the present invention is adopted to determine the position of the gantry crane grab by adopting the fusion method of vision and three-dimensional point cloud. The fusion method of vision and three-dimensional point cloud not only solves the above problems, but also realizes the precise positioning of the gantry crane. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the system structure and method;

[0042] 1 upper arm, 2 trunk bridge; DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments.

[0044] This embodiment provides a method for positioning a gantry crane grab bucket in a gantry crane grab bucket positioning system based on machine vision and three-dimensional point cloud fusion. The gantry crane has three states: standby state, operating state, and stop state. When the gantry crane is in the operating state, the grab bucket will be lifted, lowered, and translated. The grab bucket will transport cargo in the air or grab and release cargo close to the ground. When the gantry crane is in the standby state or the stop state, the grab bucket will be parked on the ground and integrated with the cargo on the ground. In summary, the grab bucket has two states: in the air and on the ground.

[0045] This system vertically installs a three-dimensional laser radar and a camera at the head platform of the elephant trunk bridge. The camera determines the orientation of the grab bucket, and then the three-dimensional laser radar is used to position the grab bucket in three dimensions, thereby achieving accurate three-dimensional spatial positioning of the grab bucket.

[0046] 1. Collect images through the camera, use the deep learning algorithm to monitor the grab in the collected images in real time, and determine the grab's position according to its position in the video.

[0047] 1) Use a camera to capture images.

[0048] 2) Through deep learning methods, the video images are analyzed in real time to determine the position of the grab in the image.

[0049] 3) According to the pixel position of the grab in the image, determine the position area of ​​the grab in the three-dimensional space in the three-dimensional coordinate system of the camera.

[0050] Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x , fy, then the position area of ​​the object in three-dimensional space is (x1, y1, z1).

[0051]

[0052] 2. Scan the grab area through 3D LiDAR, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab.

[0053] 1) Use 3D LiDAR to scan the grab area.

[0054] 2) Analyze the 3D point cloud in real time and use the adaptive spatial clustering method of the point cloud data to divide the suspected grab area.

[0055] This system adopts a clustering algorithm based on density space.

[0056] Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T.

[0057] Scan all point cloud data and form a temporary cluster with points within the radius R whose number is greater than MinPoints. This point cloud is called the core point.

[0058] For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster. Repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).

[0059] 3. Combine the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab.

[0060] The position area (x1, y1, z1) of the grab bucket in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera. The center point (x 2j ,y 2j ,z 2j ) is in the radar three-dimensional coordinate system. To achieve accurate positioning, the two coordinate systems need to be transformed.

[0061] This system converts the camera coordinate system into a three-dimensional point cloud coordinate system.

[0062] Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is

[0063]

[0064] Will Substitute (x1, y1, z1) into the formula

[0065] Get (x 1j ,y 1j ,z 1j ).

[0066]

[0067] Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) with the shortest Euclidean distance and the smallest z2j is used as the grab area (x3, y3, z3).

[0068] Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).

[0069] 4. Convert the grab area coordinates (x3, y3, z3) in the radar three-dimensional coordinate system to the three-dimensional coordinates in the on-site ground coordinate system.

[0070] The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction.

[0071] Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω.

[0072] Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is

[0073]

[0074] Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:

[0075]

[0076] The present invention integrates machine vision and three-dimensional point cloud, realizing real-time monitoring and video positioning of the gantry crane grab bucket through machine vision, and realizing accurate positioning of the grab bucket through three-dimensional point cloud data.

[0077] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for positioning a gantry crane grab bucket based on machine vision and three-dimensional point cloud fusion, characterized in that: The following steps are involved: A1. Collect images through the camera, use the deep learning algorithm to monitor the grab bucket in the collected images in real time, and determine the grab bucket's position according to its position in the video; A2 uses 3D laser radar to scan the grab area, analyze the Euclidean distance of the 3D point cloud, and detect and identify the 3D position of the grab; A3 combines the suspected grab area determined by the 3D point cloud with the grab area determined by vision to determine the exact position of the grab; A4 converts the grab area coordinates in the radar three-dimensional coordinate system into three-dimensional coordinates in the on-site ground coordinate system.

2. The method according to claim 1, characterized in that The step A1 comprises the following steps: A11 uses the camera to capture images; A12 uses deep learning methods to analyze video images in real time and determine the position of the grab in the image; A13 determines the position area of ​​the grab bucket in the three-dimensional space in the three-dimensional coordinate system of the camera according to the pixel position of the grab bucket in the image; Assume that the pixel size in the image is (u,v) and the pixel point at the center of the image is The pixel position of the grab bucket on the image is (u1, v1), and the focal length of the camera in the x and y directions is f x 、f y , then the position area of ​​the object in three-dimensional space is (x1, y1, z1); 3. The method according to claim 1, characterized in that: In the step A2, The following steps are involved: A21 uses 3D LiDAR to scan the grab area; A22 analyzes 3D point clouds in real time and uses adaptive spatial clustering methods to divide suspected grab areas; Set the neighborhood radius R and the minimum number of points MinPoints. Assuming that the distance between the gantry crane's trunk and the grab bucket is S and the grab bucket size threshold is T, the neighborhood radius R = S + T; Scan all point cloud data, and form a temporary cluster with points greater than MinPoints within the radius R. This point cloud is called the core point. For each temporary cluster, check whether the core points are within the same R range, and cluster to form a new temporary cluster; repeat the scan until all temporary clusters are clustered and classified, obtain the suspected grab area, and determine the center point of the area (x 2j ,y 2j ,z 2j ).

4. The method according to claim 1, characterized in that: The step A3 comprises the following steps: The position area (x1, y1, z1) of the grab in the three-dimensional space obtained by the camera is in the three-dimensional coordinate system of the camera; the center point (x 2j ,y 2j ,z 2j ), is in the radar three-dimensional coordinate system; to achieve accurate positioning, the two coordinate systems need to be transformed; the camera coordinate system is transformed into a three-dimensional point cloud coordinate system; Assume that the angle of rotation of the two coordinate systems around the x-axis is α1, the angle of rotation around the y-axis is β1, and the angle of rotation around the z-axis is γ1. The translation distance is Will Substitute (x1, y1, z1) into the formula Get (x 1j ,y 1j ,z 1j ); Set the threshold d, select (x 2j ,y 2j ,z 2j ), (x 1j ,y 1j ,z 1j ) and (x 2j ,y 2j ,z 2j ) are closest in Euclidean distance, and z 2j The smallest point is used as the grab area (x3, y3, z3); Assuming that the real grab area is the i-th grab suspected area, then (x3, y3, z3) = (x 2i ,y 2i ,z 2i ).

5. The method according to claim 1, characterized in that The step A4 comprises the following steps: The on-site ground coordinate system takes the driver's cab as the coordinate origin, the direction perpendicular to the shore is the x-axis direction, the direction parallel to the shore is the y-axis direction, and the vertical direction is the z-axis direction; Assume that the height of the gantry crane's rotation center is h, the length of the boom is l1, the length of the trunk is l2, the rotation angle is η, the pitch angle is θ, and the angle between the trunk and the boom is ω; Then the angle of rotation around the x-axis between the 3D radar coordinate system and the ground coordinate system is α2, the angle of rotation around the y-axis is β2, the angle of rotation around the z-axis is γ2, and the translation distance is Then in the ground coordinate system, the coordinates of the grab (x4, y4, z4) are:

Citation Information

Cited By

  • Gantry crane multi-source point cloud fusion method based on grab bucket positioning and grab bucket direct control method

    CN121107261A