A method for identifying the pose of a spreader based on visual detection

Abstract

This application relates to the technical field of intelligent gantry cranes, and discloses a method for identifying the position and pose of a spreader based on visual detection. The actual position and pose of the spreader during operation are dynamically identified through visual recognition, and compared and analyzed with the theoretical position and pose data. Finally, the amount of position and pose adjustment of the spreader is obtained, and the position and pose of the spreader are adjusted in real time according to the adjustment amount. Therefore, for objects with high positioning accuracy requirements, the method of this application has a better lifting effect, can achieve more precise lifting, and can also improve the handling efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent gantry cranes, and particularly to a method for identifying the position and pose of a spreader based on visual detection. Background Art

[0002] Currently, in the container yard of a gantry crane, the identification of container blocks and positions entirely depends on the manual operation of the driver. However, it is easy to make mistakes in identifying container blocks and positions by manual observation, resulting in information errors in container stacking and bringing trouble to the management of containers. At the same time, the position where the gantry crane stacks containers completely depends on the manual visual inspection of the truck driver. When using the gantry crane to lift an object with high positioning accuracy requirements, the alignment of the lifting interface is achieved by the driver or camera observing the operation of the spreader. The actual operation is difficult, and the lifting efficiency is also low. Summary of the Invention

[0003] In order to solve the problems and deficiencies existing in the above-mentioned prior art, the present application proposes a method for identifying the position and pose of a gantry crane spreader based on visual detection, which can quickly identify and locate the position of the spreader and obtain the amount of position and pose adjustment of the spreader, thereby improving the loading and unloading efficiency.

[0004] In order to achieve the above-mentioned invention purpose, the technical solution of the present application is specifically as follows:

[0005] A method for identifying the position and pose of a spreader based on visual detection specifically includes the following steps:

[0006] Fix four targets in a cross shape on the spreader, and install two binocular cameras at the bottom of the trolley, namely the first binocular camera and the second binocular camera. The two binocular cameras are located above the targets, and the optical axes of the cameras are parallel to each other and perpendicular to the ground;

[0007] Establish a reference coordinate system with the center point of the trolley platform as the reference point, establish a camera coordinate system with the optical center of the first binocular camera as the reference point, calibrate the two binocular cameras, and obtain the internal parameters of the two cameras and the position parameter relationship between the two cameras;

[0008] Solve the transformation matrix between the camera coordinate system and the reference coordinate system according to the coordinates of the target center point of the target in the camera coordinate system and the coordinate value of the target center point in the reference coordinate system;

[0009] The binocular cameras take pictures and identify the targets on the spreader, transmit the obtained target images to the computer, and the computer performs image processing to calculate the offset and rotation amount of the spreader at this time, and finally realizes the identification of the position and pose of the spreader.

[0010] The solving of the transformation matrix between the camera coordinate system and the reference coordinate system according to the coordinates of the target center point of the target in the camera coordinate system and the coordinate value of the target center point in the reference coordinate system includes:

[0011] The first binocular camera and the second binocular camera respectively take pictures and identify the targets on the spreader. The computer processes the images transmitted by the two cameras, and obtains the coordinates of the target center of each target in the camera coordinate system through the binocular positioning algorithm ( c x i , c y i , c z i , i = A, B, C, D);

[0012] Use a total station or a laser tracker to measure the coordinate values of the target center positions on the four targets, and obtain the coordinate values of the target center of each target in the reference coordinate system ( b x i , b y i , b z i , i = A, B, C, D);

[0013] According to the coordinate values of the target center in the two coordinate systems, obtain the transformation matrix between the two coordinate systems through the following calculation expression b T c

[0014]

[0015] Furthermore, the binocular camera takes pictures and identifies the targets on the spreader, transmits the obtained target images to the computer, and the computer performs image processing, calculates the offset and rotation amount of the spreader at this time, and finally realizes the recognition and positioning of the spreader pose, including:

[0016] The computer calculates the coordinates of the target center of each target in the camera coordinate system at this time according to the target images taken by the two binocular cameras ( c x i ’, c y i ’, c z i ’, i = A, B, C, D);

[0017] According to the transformation matrix b T c between the camera coordinate system and the reference coordinate system, calculate the coordinates of the target center of each target in the reference coordinate system at this time ( b x i ’, b y i ’, b z i ’, i = A, B, C, D);

[0018] According to the coordinates of the target center in the reference coordinate system, solve for the coordinates (x0, y0, z0) of the center point of the cross-shaped structure enclosed by the four targets in the reference coordinate system;

[0019] According to the theoretical pose coordinates (x, y, z) of the spreader center in the reference coordinate system and the coordinate values of the center point of the cross-shaped structure in the reference coordinate system, solve for the spreader offset (x s , y s , z s )

[0020]

[0021] According to the coordinates of the target centers of the four targets in the reference coordinate system, solve for the rotation amount of the spreader according to the following calculation expression

[0022]

[0023] where s is the deflection angle of the spreader, l is the pitch angle of the spreader, and t is the roll angle of the spreader.

[0024] Furthermore, a light source target and a reflection target are arranged on the target.

[0025] Advantages of the present application:

[0026] (1) The present application identifies the actual pose of the spreader of the gantry crane during operation through visual recognition, compares and analyzes it with the theoretical pose data, and finally obtains the pose adjustment amount of the spreader, and adjusts the pose of the spreader in real time according to the adjustment amount. Therefore, for objects with high positioning accuracy requirements, the method of the present application has a better lifting effect, can achieve more precise lifting, and can also improve the loading and unloading efficiency.

[0027] (2) The target of the present application adopts a combination of a light source target with active illumination and a reflection target, which can better adapt to the identification and detection of spreaders in different environments such as strong light, rainy days, foggy days, and cloudy days. Description of the Drawings

[0028] The foregoing and following specific descriptions of the present application become clearer when read in conjunction with the following drawings, in which:

[0029] Figure 1 is the flowchart of the method of the present application;

[0030] Figure 2 is the schematic diagram of the target and camera installation structure of the present application;

[0031] Figure 3 is the schematic diagram of the spreader offset of the present application.

[0032] In the accompanying drawings:

[0033] 1. Target; 2. Trolley; 3. First camera; 4. Second camera; 5. Sling; 6. Container. Detailed implementation manners

[0034] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will further illustrate the technical solutions for achieving the invention purpose of this application through several specific embodiments. It should be noted that the technical solutions claimed in this application include but are not limited to the following embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of this application.

[0035] Currently, when using a gantry crane to hoist an object with high positioning accuracy requirements, the alignment of the hoisting interface is usually achieved by the driver or camera observing the operation of the sling. This not only has the problem of great actual operation difficulty but also has a low hoisting efficiency.

[0036] Based on this, this embodiment discloses a method for identifying the pose of a sling based on visual detection. This method can quickly identify the current pose of the sling and the offset and rotation amounts of the current pose of the sling relative to the theoretical pose of the sling, so as to quickly adjust the pose of the sling according to the offset and rotation amounts, making the pose of the sling coincide with the theoretical pose and successfully completing the grasping and hoisting of the lifted object.

[0037] At the initial stage, the theoretical pose is manually input into the computer by workers. The theoretical pose is the best ideal position of the sling when grasping the lifted object.

[0038] When this method is specifically implemented, it includes two processes. One is the camera calibration and coordinate system conversion process before formally identifying the pose of the sling, and the other is to formally start identifying the pose of the sling and output the offset and rotation amounts of the sling at this time.

[0039] The camera calibration and coordinate system conversion process is mainly as follows:

[0040] Referring to the attached drawings of the specification Figure 2 , first, install the target 1 on the upper surface of the sling 5, and install a binocular camera at the bottom of the trolley 2. The target 1 installed on the upper surface of the sling 5 is recognized by the camera. There are several target points on the target 1. The binocular camera is connected to the computer, and the image of the target 1 obtained by shooting is transmitted to the computer. Then, a camera coordinate system O c with the optical center of the camera as the coordinate origin and a reference coordinate system O b; During calibration, the computer recognizes the image of target 1, obtains the coordinates of the center of the target point on target 1 in the camera coordinate system. Then, a total station or laser tracker is used to measure and track the target point on target 1 to obtain the coordinates of the center of the target point in the reference coordinate system. Then, based on the coordinates of the center of the target point in the camera coordinate system and the coordinates of the center of the target point in the reference coordinate system, the transformation matrix between the camera coordinate system and the reference coordinate system is jointly solved.

[0041] The process of recognizing the pose of the spreader is mainly as follows:

[0042] When actually recognizing the pose of the spreader, the camera obtains the coordinates of the center of the target point of target 1 at this time in the camera coordinate system. Through the transformation matrix obtained during calibration, this coordinate is transformed into the reference coordinate system, so as to obtain the base coordinates of the center of the target point. Using the coordinates of multiple center of target points, the actual spatial position of the spreader 5 during operation can be obtained. According to the actual position of the object to be lifted and other objects to be grasped, the theoretical spatial position of the spreader 5 that needs to run to the clamping position plane can be known. Through the actual spatial position and the theoretical spatial position, the offset and rotation amount of the spreader 5 can be obtained.

[0043] The theoretical spatial position is the theoretical pose of the spreader, and the actual spatial position is the actual pose of the spreader at this time.

[0044] Refer to the attached Figure 1 , The specific steps of this method are as follows:

[0045] Step S1. Fix four targets 1 in a cross shape on the upper surface of the spreader 5. The four targets 1 are in the same plane, and when the targets 1 are installed, they are respectively located on the symmetry axes in the vertical and horizontal directions of the upper surface of the spreader 5. The plane where the targets 1 are located is parallel to the upper surface of the spreader 5, and when viewed from above, the cross center of the four targets 1 coincides with the center of the spreader 5. The four targets 1 are respectively denoted as A, B, C, and D. Among them, the distance between target A and target C is L AC , the distance between target B and target D is L BD , Install two binocular cameras at the bottom of the trolley, namely the first binocular camera 3 and the second binocular camera 4. Both binocular cameras are located above the target 1. The optical axes of the two cameras are parallel to each other and perpendicular to the ground. The distance between the two cameras is L.

[0046] Step S2. Establish a reference coordinate system O with the center point of the platform of the trolley 2 as the reference point b , the Z-axis points vertically upward, the X-axis points in the direction of the trolley movement, and the Y-axis is determined according to the right-hand rule; then establish a camera coordinate system O with the optical center of the first binocular camera 3 as the reference point c, the Z-axis points vertically downward, the X-axis points in the moving direction of the trolley, and the Y-axis is determined according to the right-hand rule. Calibrate the two binocular cameras to obtain the internal parameters of the two cameras and the positional parameter relationship between the two cameras.

[0047] In this embodiment, it should be noted that the calibration method for the two cameras is a conventional means well-known to those skilled in the art, such as the Zhang Zhengyou calibration method.

[0048] Step S3. According to the coordinates of the target center of each target 1 in the camera coordinate system and the coordinate values of the target center in the reference coordinate system, solve the transformation matrix between the camera coordinate system and the reference coordinate system.

[0049] Step S4. The spreader starts to operate under the action of the trolley. The two binocular cameras respectively take pictures and identify the four targets (A, B, C, D) on the spreader 1, transmit the obtained images of the target 1 to the computer, and the computer performs image processing to calculate the offset and rotation amount of the spreader 1 at this time, and finally realizes the pose recognition of the spreader 1.

[0050] Further, the specific steps of step S3 include:

[0051] The first binocular camera and the second binocular camera respectively take pictures and identify the target 1 on the spreader, transmit the obtained images to the computer, and the computer finally obtains the coordinate values of the target centers of the four targets 1 in the camera coordinate system through the binocular positioning algorithm ( c x i ’, c y i ’, c z i ’, i = A, B, C, D);

[0052] Use a total station or a laser tracker to measure the position coordinates of the target centers of the four targets 1 to obtain the coordinate values of the target centers of the four targets 1 in the reference coordinate system respectively ( b x i , b y i , b z i , i = A, B, C, D);

[0053] According to the coordinate values of the target center of the target 1 in the two coordinate systems, obtain the transformation matrix of the two coordinate systems through the following calculation expression b T c

[0054]

[0055] where i = A, B, C, D.

[0056] Further, the step S4 specifically includes:

[0057] Based on the images of target 1 captured by two binocular cameras, the computer calculates, through the binocular positioning algorithm, the coordinates of the target centers of each target 1 on the spreader 5 in the camera coordinate system at this time ( c x i ’, c y i ’, c z i ’, i = A, B, C, D);

[0058] Based on the transformation matrix b T c between the camera coordinate system and the reference coordinate system, the computer calculates the coordinates of the target centers of each target 1 in the reference coordinate system at this time ( b x i ’, b y i ’, b z i ’, i = A, B, C, D);

[0059] Based on the coordinates of the target centers of each target 1 in the reference coordinate system, the computer solves for the coordinates (x0, y0, z0) of the center point of the cross-shaped structure formed by the four targets 1 in the reference coordinate system;

[0060] Based on the theoretical position coordinates (x, y, z) of the spreader 1 center in the reference coordinate system and the coordinate values (x0, y0, z0) of the center point of the cross-shaped structure in the reference coordinate system, the computer solves for the spreader offset (x s , y s , z s )

[0061]

[0062] Based on the coordinates of the target centers of the four targets 1 in the reference coordinate system, the computer solves for the rotation amount of the spreader according to the following calculation expression

[0063]

[0064] where s is the deflection angle of the spreader, l is the pitching angle of the spreader, and t is the roll angle of the spreader.

[0065] This method can dynamically identify the pose of the spreader at this time, and then obtain the difference between the current spatial position of the spreader and the theoretical spatial position. Finally, the control system controls the spreader to adjust its pose according to the offset and rotation amount of the spreader at this time, so that the spreader coincides with the theoretical grasping position, and finally successfully completes the grasping and lifting of the lifted object.

[0066] Further, in this embodiment, a light source target point and a reflection target point are provided on the target. To better apply the recognition of the spreader pose in different weather conditions, the target in this embodiment combines a light source target point that actively emits light outward and a conventional reflection target point, and they are jointly set on the target. When the light is relatively dim, the light source target point on the target is turned on. This target point is equivalent to a light source and actively emits light outward. Therefore, at this time, the recognition of the spreader pose is mainly achieved through the light source target point on the target. Under the weather conditions with sufficient light, the light source target does not actively emit light outward.

[0067] In this embodiment, it should be noted that the method for the coordinates of the center point of the cross-shaped structure formed by the four targets 1 in the reference coordinate system is a conventional technical means well-known to those skilled in the art.

[0068] In this embodiment, it should also be noted that the theoretical position coordinates of the center of the spreader 1 in the reference coordinate system are known and are input into the computer manually, which is the theoretical spatial position for the spreader to grasp the suspended object.

[0069] In this embodiment, it should be further noted that the binocular positioning algorithm belongs to the technical means well-known to those skilled in the art. Binocular positioning mainly performs positioning based on the imaging parallax of two cameras.

[0070] In the description of this application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the protection scope of this application.

[0071] In the description of this application, it should also be noted that unless otherwise clearly specified and limited, the terms "set", "installed", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific situations.

[0072] The above are only the preferred embodiments of this application, and do not impose any formal limitations on this application. Any simple modifications and equivalent changes made to the above embodiments based on the technical essence of this application all fall within the protection scope of this application.

Claims

1. A method for identifying the pose of the spreader of a gantry crane based on visual detection, characterized in that, Including: Fix four targets in a cross shape on the spreader. Install two binocular cameras at the bottom of the trolley, namely the first binocular camera and the second binocular camera. The two binocular cameras are located above the targets, and the camera optical axes are parallel to each other and perpendicular to the ground. Establish a reference coordinate system with the center point of the trolley platform as the reference point, and establish a camera coordinate system with the optical center of the first binocular camera as the reference point. Calibrate the two binocular cameras to obtain the internal parameters of the two cameras and the position parameter relationship between the two cameras. According to the coordinates of the target center point of the target in the camera coordinate system and the coordinate value of the target center point in the reference coordinate system, solve the transformation matrix between the camera coordinate system and the reference coordinate system. The binocular cameras take pictures and recognize the targets on the spreader, and transmit the obtained target images to the computer. The computer performs image processing, calculates the offset and rotation amount of the spreader at this time, and finally realizes the recognition of the spreader pose. Specifically, Based on the target images captured by two binocular cameras, the computer calculates the coordinates of the target centers of each target in the camera coordinate system at this time ( c x i ', c y i ', c z i ', i = A, B, C, D); According to the transformation matrix between the camera coordinate system and the reference coordinate system b T c , the coordinates of the target centers of each target in the reference coordinate system at this time are calculated as ([[]] b x i ', by i ', b z i ', i = A, B, C, D); According to the coordinates of the target center point in the reference coordinate system, solve the coordinates (x0, y0, z0) of the center point of the cross-shaped structure surrounded by the four targets in the reference coordinate system. According to the theoretical pose coordinates (x, y, z) of the spreader center in the reference coordinate system and the coordinate values (x0, y0, z0) of the center point of the cross-shaped structure in the reference coordinate system, solve the spreader offset (x s , y s , z s ) through the following calculation expressions: According to the coordinates of the target center points of the four targets in the reference coordinate system, solve the rotation amount of the spreader according to the following calculation expression: Where, s is the deflection angle of the spreader, l is the pitching angle of the spreader, and t is the roll angle of the spreader.

2. The method for identifying the position and pose of the spreader of the gantry crane based on visual detection according to claim 1, wherein, The solving of the transformation matrix between the camera coordinate system and the reference coordinate system according to the coordinate value of the target center point of the target in the camera coordinate system and the coordinate value of the target center point in the reference coordinate system includes: The first binocular camera and the second binocular camera respectively take pictures and identify the targets on the spreader, and the computer processes the images transmitted by the two cameras, and obtains the coordinates of the target center of each target in the camera coordinate system through the binocular positioning algorithm ( c x i , c y i , c z i , i = A, B, C, D); Measure the coordinate values of the center positions of the target points on the four targets using a total station or a laser tracker to obtain the coordinate values of the center of each target point in the reference coordinate system ( b x i , b y i , b z i , i = A, B, C, D); According to the coordinate values of the target center in two coordinate systems, the transformation matrix between the two coordinate systems is obtained through the following calculation expressions b T c 3. A method for identifying the position and pose of a gantry crane spreader based on visual detection according to claim 1, characterized in that, Light source targets and reflection targets are arranged on the target.

Images