Visual box grabbing method for gantry crane

Abstract

The invention relates to the technical field of machine vision, in particular to a visual box grabbing method for a gantry crane, which comprises the following steps: mounting four groups of high-definition cameras on the same plane of the lifting end of a lifting appliance, and enabling the four groups of cameras to acquire images in real time in up-down and left-right symmetry; based on images shot by a high-precision calibration plate in an operation area, internal parameters and external parameters of four groups of cameras are calculated by applying a calibration algorithm; analyzing container corner image features, and determining corner three-dimensional coordinates and image pixel coordinates based on a measurement tool; a multi-task learning model is established, angular point positioning is used as a main task, the height change of the lifting appliance is simulated by cutting pictures, data enhancement training is carried out, and the method has the advantages of being easy to install, high in detection speed, high in precision, high in robustness and the like.

Description

Technical Field

[0001] The invention relates to the technical field of machine vision, and in particular to a method for visually grabbing a box for a gantry crane. Background Art

[0002] In modern port logistics, gantry cranes are key equipment used to move containers from ships to docks or from docks to ships. However, traditional operation methods rely on manual labor, which is inefficient and has large errors. In order to improve efficiency and accuracy, automatic container grabbing systems have emerged and have gradually become an important part of smart port construction.

[0003] The invention patent with application number 202110566120.8 discloses a method for automatically grabbing a box with a gantry crane, which is characterized by comprising the following steps: S100: The sidelines of the gantry crane's spreader and the container include long sides and short sides arranged in pairs; a first laser radar is respectively arranged at the two ends of the two long sides of the gantry crane's spreader; a second laser radar is respectively arranged in the middle of the two short sides of the gantry crane's spreader; the scanning directions of the first laser radar and the second laser radar point to the bottom of the spreader; each first laser radar and the second laser radar are used to obtain point cloud data below the spreader; S200: The horizontal movement direction of the gantry crane is taken as the Y-axis, the direction perpendicular to the Y-axis in the horizontal plane is the X-axis, the vertical direction is the Z-axis, and the world coordinate system is constructed according to the right-hand rule; S300: After the gantry crane's spreader moves to a certain height above the container to be grabbed or stacked, each first laser radar and the second laser radar obtains the point cloud data below the spreader, and each first laser radar and the second laser radar calibrate all the point cloud data to the world coordinate system respectively, The calibrated point cloud data is stitched, and duplicate point cloud data is removed to obtain the point cloud data of each edge of the spreader, the containers loaded on the spreader, and the edge of the containers to be grabbed or stacked on the ground, and their spatial coordinates in the world coordinate system; S400: According to the spatial coordinates of the endpoints of each edge of the spreader in the world coordinate system, and the spatial coordinates of each edge of the container loaded on the spreader and the edge of the container to be grabbed or stacked on the ground in the world coordinate system, calculate the offset distance between each edge of the spreader and the edge of the container to be grabbed, or the offset distance between the edge of the container loaded on the spreader and the edge of the container already in the yard; S500: The spreader is translated and rotated by the offset obtained in the previous step to adjust the spreader posture. When the current translation and rotation of the spreader reach the set threshold, the spreader posture adjustment is stopped, and the spreader is lowered to grab or release, and the container is grabbed or stacked.

[0004] The application aims to solve the problem that "although some ports have adopted the RTG yard automation solution, there are still shortcomings such as difficulty in calibration, easy deviation of the RTG truck benchmark, inconvenience in detecting the current posture of the crane, and time-consuming posture adjustment, and the operating efficiency cannot be effectively improved."

[0005] However, existing technologies all use lasers to determine the deviation between the spreader and the target box, and when on the lane side, the driver needs to be guided to move the container truck into position and grab the container; To this end, we proposed a method for gantry crane visual box grabbing. Summary of the invention

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for visually grasping a box for a gantry crane, which solves the technical problems raised in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for visually grabbing a box by a gantry crane, comprising: Four sets of high-definition cameras are installed on the same plane at the lifting end of the spreader, so that the real-time images collected by the four sets of cameras are symmetrical from top to bottom and from left to right; images are taken in the working area based on a high-precision calibration plate, and the calibration algorithm is used to calculate the internal and external parameters of the four sets of cameras; the image features of the corner points of the container are analyzed, and the three-dimensional coordinates of the corner points and the image pixel coordinates are determined based on the measuring tool; a multi-task learning model is built, with corner point positioning as the main task, and the height change of the spreader is simulated by cropping pictures to perform data enhancement training; the top laser scanning system is started before the AGV car arrives at the work row, and the height of the work row is scanned after setting the parameters, and the data is processed using a filtering algorithm, calibrated and converted to the gantry crane coordinate system; after the car arrives at the work row, the distance from the camera to the box surface is calculated based on the height information of the laser anti-bowling spreader, and the world coordinates of the box corner points are calculated based on the monocular triangulation principle and camera parameters, the deviation is analyzed by comparing the target corner points, the car and the spreader are controlled to reduce the deviation based on the deviation rotation direction, and the box grabbing operation is performed in the process of reducing the deviation.

[0008] Furthermore, during the installation phase, the four groups of high-definition cameras measure the points of the four groups in the same plane by means of a level meter, and the high-definition cameras are installed using the measured points. The four groups of high-definition cameras operate synchronously, and the four groups of high-definition cameras operate in conjunction with the lifting end of the sling; Among them, the closed image obtained by connecting the four groups of points measured by the level is a rectangle or a positive direction. When the closed image is a rectangle, the aspect ratio of the rectangle is greater than 2 / 1, and the area of ​​the closed image is greater than the cross-sectional area of ​​the lifting end intercepted by three quarters of the plane where the closed figure is located.

[0009] Furthermore, the calibration algorithm is applied to calculate the four sets of camera intrinsic and extrinsic parameters in the execution phase: Set a specified pattern on the calibration plate so that the pattern size and accuracy meet the camera calibration requirements, and further place the calibration plate at any position and angle within the gantry crane operation area to simulate the container posture. Use four sets of high-definition cameras to simultaneously shoot the calibration plate, and input the captured calibration plate image data into the calibration algorithm software. The algorithm first identifies the feature points of the calibration plate in the image, and establishes the relationship equation between the image pixel coordinates and the world coordinates based on the position of the feature points in the image and the known actual geometric size of the calibration plate; During the shooting process, it is necessary to ensure that the calibration board performs a focus process in real time on each high-definition camera screen. Before the calculation operation begins, the system user traverses the calibration data and manually cleans the calibration data.

[0010] Furthermore, the four sets of camera internal parameters and external parameters are calculated by the following formula: , ; Where: is the coordinate on the image plane; is the focal length in the X and Y directions; is the coordinate of the center of the image; is a point in the camera coordinate system; is a point in the world coordinate system; is the rotation matrix; is the translation vector.

[0011] Furthermore, in formula (1), the known world coordinate points and the corresponding image coordinate points are combined with the linear regression algorithm to , To solve, in formula (2), a set of equations is established based on the corresponding points in each set of world coordinate system and camera coordinate system, and the SVD algorithm is used to solve the rotation and translation parameters to obtain the external parameters.

[0012] Furthermore, after using the calibration board to jointly calibrate the intrinsic parameters and positions of the four cameras, we start to calibrate the positions of the corner points of the grab box: A standard container model with known corner points is placed in multiple typical positions and postures within the gantry crane operation area. Four calibrated cameras are started to shoot the container model simultaneously to collect image data containing corner points. The pixel coordinates of the corner points in the image are identified through the corner point detection algorithm based on the features of the corner points at the image edge, texture, and grayscale changes. The calibrated camera intrinsic and extrinsic parameters are further used to convert the pixel coordinates of the corner points into world coordinates, and the associated coordinates of the image corner points and the actual space corner points are established.

[0013] Furthermore, the measuring tools used in the stage of determining the three-dimensional coordinates of the corner points and the image pixel coordinates include: a total station, a laser tracker, a three-dimensional laser scanner, a differential GPS, and an altitude IMU.

[0014] Furthermore, the analysis logic of the container corner image features is expressed as: The collected original image is gray-scaled, and the color image is converted into a gray-scale image. Further filtering operations are performed to remove noise in the image. The local autocorrelation matrix of each pixel in the image is calculated through the Harris corner detection algorithm. The pixel is judged to be a corner point based on the eigenvalue of the matrix. For the detected corner points, the gradient information of the surrounding area is calculated based on the Sobel operator. The corner point features are screened according to the uniqueness of the features to remove false corner point features caused by noise or other interference factors.

[0015] Furthermore, the multi-task learning model selects any one of the multi-task CNN model or the multi-task relational network model based on shared features, and the cropping scales include: 1 / 2, 1 / 3, 1 / 4, 1 / 8. During the data enhancement training process, Gaussian noise or salt and pepper noise is added to the selected learning model for training.

[0016] Furthermore, the scanning density of the laser scanning system is such that the smaller the volume of the box to be grabbed by the sling is, the higher the scanning density is, and vice versa.

[0017] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects: The present invention provides a method for visual box grabbing of a gantry crane. During the execution of the method, the method has the advantages of easy installation, fast detection speed, high precision and strong robustness. In addition, the method provides accurate images and reliable data for the operation through reasonable installation, calibration, corner point calibration and data processing of the camera. Model training improves the model performance to adapt to the changes in the operation. The height is accurately acquired and the coordinate system is converted before the operation. During the operation, the deviation can be accurately calculated, the box grabbing can be controlled and adjusted in real time, ensuring that the entire operation process is stable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 The figure is a flow chart of a method for visually grabbing boxes by a gantry crane. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention will be further described below in conjunction with the embodiments.

[0022] Embodiment 1: A method for visually grabbing a box for a gantry crane in this embodiment is as follows: Figure 1 As shown, including: Step 1: Install four sets of high-definition cameras on the same plane at the lifting end of the hoisting device, so that the real-time collection images of the four sets of cameras are symmetrical on the left and right sides; Step 2: Take images in the working area based on the high-precision calibration plate, and use the calibration algorithm to calculate the four sets of camera intrinsic and extrinsic parameters; The operation execution phase of applying the calibration algorithm to calculate the four sets of camera intrinsic and extrinsic parameters: Set a specified pattern on the calibration plate so that the pattern size and accuracy meet the camera calibration requirements, and further place the calibration plate at any position and angle within the gantry crane operation area to simulate the container posture. Use four sets of high-definition cameras to simultaneously shoot the calibration plate, and input the captured calibration plate image data into the calibration algorithm software. The algorithm first identifies the feature points of the calibration plate in the image, and establishes the relationship equation between the image pixel coordinates and the world coordinates based on the position of the feature points in the image and the known actual geometric size of the calibration plate; During the shooting process, it is necessary to ensure that the calibration board performs a focus process in real time on each high-definition camera screen. Before the calculation operation begins, the system end user traverses the calibration data and manually cleans the calibration data; The four sets of camera internal and external parameters are calculated using the following formula: , ; Where: is the coordinate on the image plane; is the focal length in the X and Y directions; is the coordinate of the center of the image; is a point in the camera coordinate system; is a point in the world coordinate system; is the rotation matrix; is the translation vector; In formula (1), the known world coordinate points and the corresponding image coordinate points are combined with the linear regression algorithm to , To solve, in formula (2), a set of equations is established based on the corresponding points in each set of world coordinate system and camera coordinate system, and the rotation and translation parameters are solved using the SVD algorithm to obtain the external parameters; After using the calibration board to jointly calibrate the intrinsic parameters and positions of the four cameras, start calibrating the positions of the corner points of the grab box: A standard container model with known corner points is placed at multiple typical positions and postures in the gantry crane operation area. Four calibrated cameras are started to shoot the container model at the same time to collect image data containing corner points. The pixel coordinates of the corner points in the image are identified by the corner point detection algorithm based on the corner point's edge width, texture, and grayscale changes. The calibrated camera intrinsic and extrinsic parameters are further used to convert the pixel coordinates of the corner points into world coordinates, and the associated coordinates of the image corner points and the actual space corner points are established. Step 3: Analyze the image features of the container corner points and determine the three-dimensional coordinates of the corner points and the image pixel coordinates based on the measurement tool; Step 4: Build a multi-task learning model, take corner point positioning as the main task, and simulate the height change of the hoist by cropping pictures to perform data enhancement training; Step 5: The AGV starts the top laser scanning system before arriving at the work row, sets the parameters and scans the height of the work row, processes the data using a filtering algorithm, calibrates and converts it to the gantry crane coordinate system; Step 6: After the trolley arrives at the work row, the distance from the camera to the box surface is calculated based on the height information of the laser anti-bowling hoist. The world coordinates of the box corner points are calculated based on the monocular triangulation principle and camera parameters. The deviation is analyzed by comparing the target corner points. The trolley and hoist are controlled to reduce the deviation based on the deviation rotation direction, and the box grabbing operation is performed in the process of reducing the deviation.

[0023] In this embodiment, by sequentially executing the methods in the above embodiments, the accuracy and reliability of the gantry crane's visual box grabbing are effectively improved, making the application of the gantry crane's visual box grabbing technology more intelligent and more suitable for promotion, while reducing the labor cost expenditure during the gantry crane's box grabbing operation.

[0024] Embodiment 2: In terms of specific implementation, based on Example 1, this example refers to Figure 1 The method for visually grabbing a box for a gantry crane in Example 1 is further specifically described as follows: During the installation phase, the four sets of high-definition cameras are measured with a level meter at four points on the same plane, and the high-definition cameras are installed using the measured points. The four sets of high-definition cameras run synchronously and follow the lifting end of the hoisting device. The closed image obtained by connecting the four groups of points measured by the level meter is a rectangle or a positive direction. When the closed image is a rectangle, the aspect ratio of the rectangle is greater than 2 / 1, and the area of ​​the closed image is greater than the cross-sectional area of ​​the lifting end intercepted by three quarters of the plane where the closed figure is located; The measurement tools used in the phase of determining the three-dimensional coordinates of corner points and the image pixel coordinates include: total station, laser tracker, three-dimensional laser scanner, differential GPS, and height IMU; The analysis logic of container corner image features is expressed as: Grayscale the collected original image, convert the color image into a grayscale image, and further perform filtering operations to remove noise in the image. The Harris corner detection algorithm is used to calculate the local autocorrelation matrix of each pixel in the image. The eigenvalue of the matrix is ​​used to determine whether the pixel is a corner point. For the detected corner points, the gradient information of the surrounding area is calculated based on the Sobel operator. The corner point features are screened according to the uniqueness of the features to remove false corner point features caused by noise or other interference factors. The multi-task learning model uses any of the multi-task CNN models or multi-task relational network models based on shared features. The cropping scales include: 1 / 2, 1 / 3, 1 / 4, and 1 / 8. During the data enhancement training process, Gaussian noise or salt and pepper noise is added to the selected learning model for training; The scanning density of the laser scanning system follows the rule that the smaller the volume of the box that the sling grabs, the higher the scanning density, and vice versa.

[0025] In this embodiment, through the above settings, further step execution logic support is provided for the system in Example 1, ensuring that the method in Example 1 can be stably applied to the box grabbing operation of the gantry crane.

[0026] In summary, the method in the above embodiment has the advantages of easy installation, fast detection speed, high accuracy and strong robustness during execution. In addition, the method provides accurate images and reliable data for the operation through reasonable installation, calibration, corner point calibration and data processing of the camera. Model training improves model performance to adapt to changes in the operation. The height is accurately acquired and the coordinate system is converted before the operation. During the operation, the deviation can be accurately calculated, the box grabbing can be controlled and adjusted in real time to ensure that the entire operation process is stable and accurate.

[0027] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visually grabbing a box by a gantry crane, characterized in that: include: Step 1: Install four sets of high-definition cameras on the same plane at the lifting end of the hoisting device, so that the real-time collection images of the four sets of cameras are symmetrical on the left and right sides; Step 2: Take images in the working area based on the high-precision calibration plate, and use the calibration algorithm to calculate the four sets of camera intrinsic and extrinsic parameters; Step 3: Analyze the image features of the container corner points and determine the three-dimensional coordinates of the corner points and the image pixel coordinates based on the measurement tool; Step 4: Build a multi-task learning model, take corner point positioning as the main task, and simulate the height change of the hoist by cropping pictures to perform data enhancement training; Step 5: The AGV starts the top laser scanning system before arriving at the work row, sets the parameters and scans the height of the work row, processes the data using a filtering algorithm, calibrates and converts it to the gantry crane coordinate system; Step 6: After the trolley arrives at the work row, the distance from the camera to the box surface is calculated based on the height information of the laser anti-bowling hoist. The world coordinates of the box corner points are calculated based on the monocular triangulation principle and camera parameters. The deviation is analyzed by comparing the target corner points. The trolley and hoist are controlled to reduce the deviation based on the deviation rotation direction, and the box grabbing operation is performed in the process of reducing the deviation.

2. A method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: During the installation phase, the four groups of high-definition cameras are measured by a level meter at four points on the same plane, and the high-definition cameras are installed using the measured points. The four groups of high-definition cameras are operated synchronously, and the four groups of high-definition cameras are operated in conjunction with the lifting end of the lifting device; Among them, the closed image obtained by connecting the four groups of points measured by the level is a rectangle or a positive direction. When the closed image is a rectangle, the aspect ratio of the rectangle is greater than 2 / 1, and the area of ​​the closed image is greater than the cross-sectional area of ​​the lifting end intercepted by three quarters of the plane where the closed figure is located.

3. The method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: The operation execution phase of applying the calibration algorithm to calculate the four sets of camera intrinsic and extrinsic parameters: Set a specified pattern on the calibration plate so that the pattern size and accuracy meet the camera calibration requirements, and further place the calibration plate at any position and angle within the gantry crane operation area to simulate the container posture. Use four sets of high-definition cameras to simultaneously shoot the calibration plate, and input the captured calibration plate image data into the calibration algorithm software. The algorithm first identifies the feature points of the calibration plate in the image, and establishes the relationship equation between the image pixel coordinates and the world coordinates based on the position of the feature points in the image and the known actual geometric size of the calibration plate; During the shooting process, it is necessary to ensure that the calibration board performs a focus process in real time on each high-definition camera screen. Before the calculation operation begins, the system user traverses the calibration data and manually cleans the calibration data.

4. A method for visually grabbing a box by a gantry crane according to claim 3, characterized in that: The four sets of camera internal and external parameters are calculated by the following formula: 、 ； Where: is the coordinate on the image plane; is the focal length in the X and Y directions; is the coordinate of the center of the image; is a point in the camera coordinate system; is a point in the world coordinate system; is the rotation matrix; is the translation vector.

5. The method for visually grabbing a box by a gantry crane according to claim 3, characterized in that: In formula (1), the known world coordinate points and the corresponding image coordinate points are combined with the linear regression algorithm to , To solve, in formula (2), a set of equations is established based on the corresponding points in each set of world coordinate system and camera coordinate system, and the SVD algorithm is used to solve the rotation and translation parameters to obtain the external parameters.

6. The method for visually grabbing a box by a gantry crane according to claim 3, characterized in that: After using the calibration board to jointly calibrate the intrinsic parameters and positions of the four cameras, start calibrating the positions of the corner points of the grab box: A standard container model with known corner points is placed in multiple typical positions and postures within the gantry crane operation area. Four calibrated cameras are started to shoot the container model simultaneously to collect image data containing corner points. The pixel coordinates of the corner points in the image are identified through the corner point detection algorithm based on the features of the corner points at the image edge, texture, and grayscale changes. The calibrated camera intrinsic and extrinsic parameters are further used to convert the pixel coordinates of the corner points into world coordinates, and the associated coordinates of the image corner points and the actual space corner points are established.

7. The method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: The measuring tools used in the stage of determining the three-dimensional coordinates of corner points and the image pixel coordinates include: a total station, a laser tracker, a three-dimensional laser scanner, a differential GPS, and an altitude IMU.

8. The method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: The analysis logic of the container corner image features is expressed as: The collected original image is gray-scaled, and the color image is converted into a gray-scale image. Further filtering operations are performed to remove noise in the image. The local autocorrelation matrix of each pixel in the image is calculated through the Harris corner detection algorithm. The pixel is judged to be a corner point based on the eigenvalue of the matrix. For the detected corner points, the gradient information of the surrounding area is calculated based on the Sobel operator. The corner point features are screened according to the uniqueness of the features to remove false corner point features caused by noise or other interference factors.

9. The method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: The multi-task learning model selects any one of a multi-task CNN model or a multi-task relational network model based on shared features, and the cropping scales include: 1 / 2, 1 / 3, 1 / 4, 1 / 8. During the data enhancement training process, Gaussian noise or salt and pepper noise is added to the selected learning model for training.

10. The method for visually grabbing a box by a gantry crane according to claim 1, characterized in that: The scanning density of the laser scanning system is such that the smaller the volume of the box to be grabbed by the sling is, the higher the scanning density is, and vice versa.

Images