Unified Calibration Method and System for External Container Truck Positioning Cameras of Automated Cranes

Abstract

The present invention discloses a unified calibration method and system for the positioning cameras of an automated crane for external container trucks. The method includes: according to the positions of the locks on the truck bed of the external container truck under the standard operating conditions, four calibrated positioning cameras are correspondingly installed on the crane gantry legs; the video image data of the external container truck lane are respectively collected through the positioning cameras; the lock features in the video image data of the external container truck lane are respectively extracted by the target detection method to obtain the pixel coordinates of the corner points of the lock detection frame; the world coordinates of the corner points of the lock detection frame are calculated based on the internal parameters of the positioning camera and the actual size of the lock; the external parameters of each positioning camera are respectively calculated according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the internal parameters of each positioning camera. The present invention can unify the camera coordinate systems of multiple cameras into the world coordinate system without additionally setting up ground landmark points, reducing the calculation error, simplifying the calibration process, and having higher efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent port hoisting, and particularly relates to a unified calibration method and system for external container truck positioning cameras of an automated crane. Background Art

[0002] With the gradual advancement of the port automation process, in order to improve the operation efficiency, the requirements for the full automation of external container trucks in the port are also gradually increasing. At present, restricted by the problems of a large variety of external container trucks and significant feature differences, it is still difficult to obtain the position of the external container truck relative to the crane by using the traditional three-dimensional point cloud to extract the feature information of the external container truck, and the operation efficiency is low. The realization of the external container truck video positioning and guiding function depends on the unified calibration of multiple cameras, calculating the relative pose of the cameras, and then solving the relative position of the external container truck. To improve the accuracy of the automated operation, it is necessary to perform rapid and high-precision external parameter calibration work on the cameras. The traditional method for calibrating the external parameters of a camera requires placing a calibration board to solve the external parameters of the camera. However, the container truck positioning camera is installed on the crane gantry leg, and the angle between the camera optical axis and the ground is large. In this hardware installation situation, calibrating the external parameters requires manual holding or other fixed methods, which easily leads to large errors in the calibrated external parameter values and the process is relatively complex. The invention patent with the publication number CN113012235A discloses a hoisting tool attitude control system and its control method for a port crane, which installs four industrial cameras on the hoisting tool, requires multiple ground markings to cooperate to complete the calibration between the cameras, strictly controls the relative positions between different ground markings, and the calibration process is complicated, which easily brings calibration errors.

[0003] In summary, aiming at the application scenario of the external container truck positioning camera, it is necessary to design a reasonable multi-camera external parameter calibration method to achieve stable external parameter calibration efficiency and reduce calibration errors. Summary of the Invention

[0004] In view of this, the present invention proposes a unified calibration method and system for external container truck positioning cameras of an automated crane to solve the problem of large errors in the external parameter calibration of the external container truck positioning camera.

[0005] In the first aspect of the present invention, a unified calibration method for external container truck positioning cameras of an automated crane is disclosed, and the method includes:

[0006] According to the position of the lock on the container truck board under the standard operating condition of the external container truck, four calibrated positioning cameras are correspondingly installed on the crane gantry leg so that the lock on the container truck board is within the field of view of the corresponding positioning camera;

[0007] Collect the video image data of the external container truck lane through the positioning cameras respectively;

[0008] Extract the lock head features from the video image data of the outer container truck lane by the target detection method, and obtain the pixel coordinates of the corner points of the lock head detection frame;

[0009] Calculate the world coordinates of the corner points of the lock head detection frame based on the actual size of the lock head;

[0010] Calculate the external parameters of each positioning camera respectively according to the pixel coordinates of the corner points of the lock head detection frame, the world coordinates of the corner points of the lock head detection frame and the internal parameters of each positioning camera.

[0011] On the basis of the above technical solutions, preferably, the corresponding installation of four calibrated positioning cameras on the crane gantry leg according to the position of the lock head on the outer container truck board under the standard operation condition of the outer container truck, so that the lock head on the outer container truck board is within the field of view of the corresponding positioning camera specifically includes:

[0012] Taking the central plane where half of the long side dimensions of 20-foot and 40-foot containers are located as the symmetry plane respectively, symmetrically set four positioning cameras at a position 1.5 meters to 2 meters above the ground on the gantry leg on the side of the outer container truck operation lane of the crane, and ensure that for different containers and operation types under the standard operation condition of the outer container truck, each positioning camera can capture a pair of lock head features arranged oppositely on the outer container truck board within the field of view.

[0013] On the basis of the above technical solutions, preferably, the target detection method is implemented by a deep learning algorithm. After detecting and recognizing the lock head, solve the coordinates of the four corner points of the lock head detection frame.

[0014] On the basis of the above technical solutions, preferably, the specific steps of calculating the world coordinates of the corner points of the lock head detection frame based on the actual size of the lock head include:

[0015] Establish a world coordinate system with the center of the gantry leg on the loading lane side as the origin, and establish a camera coordinate system for each positioning camera respectively;

[0016] Measure the world coordinates of one corner point according to the overlapping relationship between the lock head and the corner points of the detection frame;

[0017] Based on the world coordinates of one corner point of the detection frame, combine the actual size of the lock head and the actual distance between the two lock heads in the trolley direction in the same image to calculate the world coordinates of other corner points of the detection frame, and obtain the world coordinates of 8 corner points of the two lock heads in the same image.

[0018] On the basis of the above technical solutions, preferably, the specific steps of calculating the world coordinates of other corner points of the detection frame based on the world coordinates of one corner point of the detection frame, combining the actual size of the lock head and the actual distance between the two lock heads in the trolley direction in the same image include:

[0019] Assume that the actual size of the lock head standard part is height h s, thickness t s , width w s and the distance d in the thickness direction of the corner point s . In the same image, the actual distance between the two lock heads in the trolley direction is D s ;

[0020] Suppose in the unified image collected by a certain positioning camera A, the corner pixel coordinates of the two lock head detection frames at the near end and the far end are: {P Ani1 , P Ani2 , P Ani3 , P Ani4}, {P Afi1 , P Afi2 , P Afi3 , P Afi4}. The world coordinates corresponding to each corner point are expressed as: {P Anw1 , P Anw2 , P Anw3 , P Anw4}, {P Afw1 , P Afw2 , P Afw3 , P Afw4}. Among them, P Anw1 , P Anw2 , P Anw3 , P Anw4 are the lower left, lower right, upper right and upper left points of the near-end lock head detection frame respectively, and P Afw1 , P Afw2 , P Afw3 , P Afw4 correspond to the lower left, lower right, upper right and upper left points of the far-end lock head detection frame;

[0021] According to the overlapping relationship between the lock head and the corner point of the detection frame, measure the world coordinates P Anw2 of a corner point: {X Anw2 , Y Anw2 , Z Anw2}. According to {X Anw2 , Y Anw2 , Z Anw2}, calculate the world coordinates of the remaining 7 corner points:

[0022] P Anw1 : {X Anw2 , Y Anw2 + w s , Z Anw2};

[0023] P Anw3 : {X Anw2 + d s , Y Anw2 , Z Anw2 + h s};

[0024] P Anw4 :{X Anw2 +d s ,Y Anw2 +w s ,Z Anw2 +h s};

[0025] P Afw1 :{X Anw2 +D s ,Y Anw2 +w s ,Z Anw2};

[0026] P Afw2 :{X Anw2 +D s ,Y Anw2 ,Z Anw2}

[0027] P Afw3 :{X Anw2 +d s +D s ,Y Anw2 ,Z Anw2 +h s};

[0028] P Afw4 :{X Anw2 +d s +D s ,Y Anw2 +w s ,Z Anw2 +h s}.

[0029] Based on the above technical solution, preferably, the step of calculating the external parameters of each positioning camera according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the internal parameters of each positioning camera specifically includes:

[0030] According to the camera pinhole model, the corresponding relationship equation between the pixel coordinates of each lock detection frame corner point and the world coordinates is established based on the positioning camera intrinsic parameters;

[0031] The corresponding relationship equations are combined to establish a system of equations, which are solved using the PnP algorithm in the image processing algorithm to obtain the rotation vector and translation vector of the camera coordinate system of each positioning camera relative to the world coordinate system.

[0032] On the basis of the above technical solution, preferably, the corresponding relationship equation between the pixel coordinates of each lock detection frame corner point and the world coordinates is established based on the camera pinhole model and the positioning camera intrinsic parameters, specifically including:

[0033] For any positioning camera A, the following corresponding relationship equation exists:

[0034]

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Z c is the conversion factor between the detection camera and the world coordinate system. (u Ani1 , v Ani1 ), (u Ani2 , v Ani2 ), (u Ani3 , v Ani3 ), (u Ani4 , v Ani4 ) are respectively the pixel coordinates corresponding to the corner points P Ani1 , P Ani2 , P Ani3 , P Ani4 of the lock detection frame. (u Afi1 , v Afi1 ), (u Afi2 , v Afi2 ), (u Afi3 , v Afi3 ), (u Afi4 , v Afi4 ) are respectively the pixel coordinates corresponding to the corner points P Afi1 , P Afi2 , P Afi3 , P Afi4 of the lock detection frame. K A is the camera internal parameter calibration result. R AWC and T AWC are respectively the rotation vector and translation vector of the camera coordinate system of the positioning camera A relative to the world coordinate system.

[0043] In the second aspect of the present invention, a unified calibration system for the external container truck positioning cameras of an automated crane is disclosed. The system includes:

[0044] Positioning camera: It is installed on the crane gantry leg, and the lock on the outer container truck board is within the field of view of the corresponding positioning camera, and is used to collect video image data of the outer container truck lane;

[0045] Feature extraction module: It is used to extract the lock features in the video image data of the outer container truck lane respectively by the target detection method, and obtain the pixel coordinates of the corner points of the lock detection frame;

[0046] Coordinate conversion module: It is used to calculate the world coordinates of the corner points of the lock detection frame based on the actual size of the lock;

[0047] External parameter calibration module: It is used to calculate the external parameters of each positioning camera respectively according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame and the internal parameters of each positioning camera.

[0048] The present invention has the following beneficial effects compared with the prior art:

[0049] 1) In the present invention, four calibrated positioning cameras are correspondingly installed on the crane gantry leg, which are respectively used to obtain the feature data of two locks in front of the 40-foot, in front of the 20-foot, behind the 20-foot and behind the 40-foot. Through the deep learning target detection method, the lock information of the outer container truck within the field of view is located, and the external parameters of the cameras are calibrated with the outer container truck lock as the unified feature identifier. Without additionally setting up ground landmark points, the camera coordinate systems of multiple cameras can be unified into the world coordinate system, reducing the calculation error caused by additionally setting up landmark points, and the calibration can be completed in the standard operation process, simplifying the calibration process and having higher efficiency;

[0050] 2) The present invention utilizes the overlapping relationship between the lock and the corner points of the detection frame. Based on the world coordinate system of a measured corner point of the detection frame, combined with the actual size of the lock and the actual distance between the two locks in the trolley direction within the same image, the world coordinate systems of other corner points of the detection frame are deduced. Finally, the external parameters of each positioning camera are calculated respectively according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame and the internal parameters of each positioning camera, reducing the complexity of external parameter calibration, having stronger stability, and more reasonably calculating the pose of the outer container truck positioning camera relative to the container truck lock. Furthermore, the accurate relative position of the outer container truck to the crane can be calculated more directly and quickly using this pose information, which is beneficial to realizing functions such as video container truck guidance required for outer container truck automation. Description of the Drawings

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0052] Figure 1 It is the top view of the installation positions of cameras A, B, C, and D for positioning.

[0053] Figure 2 It is the schematic diagram of the world coordinate system and the installation positions of the cameras.

[0054] Figure 3 It is the schematic diagram of the camera coordinate systems of cameras A, B, C, and D for positioning.

[0055] Figure 4 It is the top view of the distribution of the outer container lock heads.

[0056] Figure 5 It is the top view of the distribution of the outer container lock heads of 40-foot and 20-foot containers during calibration.

[0057] Figure 6 It is the schematic diagram of the imaging effect of the lock heads on cameras A, B, C, and D for positioning.

[0058] Figure 7 It is the schematic diagram of the lock head detection frame.

[0059] Figure 8 It is the schematic diagram of the three views of the lock head and the distribution of the corner points of the lock head detection frame. Specific implementation mode

[0060] Next, in combination with the implementation mode of the present invention, the technical solutions in the implementation mode of the present invention will be clearly and completely described. Obviously, the described implementation mode is only a part of the implementation modes of the present invention, rather than all the implementation modes. Based on the implementation modes in the present invention, all other implementation modes obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

[0061] The present invention provides a unified calibration method for the outer container positioning cameras of an automated crane, and the method includes:

[0062] S1. According to the positions of the lock heads on the outer container truck bed under the standard operating conditions of the outer container, install four calibrated positioning cameras on the crane gantry legs so that the lock heads on the outer container truck bed are within the field of view of the corresponding positioning cameras.

[0063] Specifically, taking the central planes where half of the long side dimensions of 20-foot and 40-foot containers are located as the symmetry planes, symmetrically set four positioning cameras at positions 1.5 meters to 2 meters above the ground on the gantry legs on the side of the outer container operation lane of the crane, and ensure that under the standard operating conditions of the outer container, for different containers and operation types, each positioning camera can capture a pair of lock head features arranged oppositely on the outer container truck bed within the field of view. Let the four positioning cameras be positioning cameras A, B, C, and D, and the top view of the installation positions is as Figure 1As shown in the figure, the positioning cameras A, B, C, and D are respectively used to obtain the characteristic data of the two locking heads at the front of the 40-foot container, the front of the 20-foot container, the rear of the 20-foot container, and the rear of the 40-foot container on the outer container truck bed.

[0064] Since a total of 4 positioning cameras are used and there are differences in the installation positions of each positioning camera, and each has an independent camera coordinate system, it is necessary to set a unified world coordinate system to solve the relative poses of each camera through the method of camera extrinsic calibration, so as to achieve unified calibration of multiple positioning cameras. To facilitate the positioning of the outer container truck, the center of the side door leg of the loading lane is set as the origin O of the world coordinate system w , with O w The forward side of the vertical door leg is the positive direction of the X axis, the left side of the parallel door leg is the positive direction of the Y axis, and the direction perpendicular to the ground passing through the point O w upward is the positive direction of the Z axis, and the world coordinate system C w is established, as Figure 2 shown.

[0065] All four positioning cameras are cameras that have undergone intrinsic calibration. The intrinsic calibration is achieved through the Zhang Dingyou calibration method. After the intrinsic calibration, the camera intrinsic parameter matrix determined by the camera hardware settings can be obtained: where f represents the focal length of the camera in millimeters; dx and dy respectively represent the widths of the image pixels in the x direction and y direction in millimeters; 1 / dx and 1 / dy can be understood as the number of pixels in 1 millimeter in the x direction and y direction; f / dx represents the length of the focal length in the x-axis direction described by pixels, and f / dy represents the length of the focal length in the y-axis direction described by pixels. u0 and v0 respectively represent the coordinates of the center of the photosensitive plate of the positioning camera in the pixel coordinate system. To distinguish the intrinsic parameter coefficients of the four positioning cameras, the name of the positioning camera is used as the subscript of the camera intrinsic parameter coefficient. Taking the positioning camera A as an example: f A , dx A , dy A , u 0A and v 0A are respectively the intrinsic parameter data of the positioning camera A obtained through the Zhang Dingyou calibration method.

[0066] The intrinsic calibration results of each positioning camera calculated by the above steps will be used for the unified calibration of the camera extrinsic parameters. When performing the unified calibration of the camera extrinsic parameters, in addition to establishing a unified world coordinate system C w , it is also necessary to establish the camera coordinate systems of the four cameras respectively, as well as the corresponding image coordinate systems and pixel coordinate systems of the cameras.

[0067] The camera coordinate systems of the four positioning cameras are established as Figure 3 shown, and are respectively C A , C B , C C and C D , where, O A , OB , O C and O D are the origins of their respective camera coordinate systems, and the positive directions of the X, Y, and Z axes of each camera are all w parallel to the X w Y w Z w axes of the C world coordinate system. Each positioning camera corresponds to its own pixel coordinate system and image coordinate system. The image coordinate system is based on the camera coordinate system and is a relative coordinate system. The X i axis and the Y i axis are respectively parallel to the X c and Z c axes of the camera coordinate system; the pixel coordinate system is based on the image coordinate system and is also a relative coordinate system. The X p axis and the Y p axis are respectively parallel to the X i axis and the Y i axis of the image coordinate system.

[0068] S2. Respectively collect the video image data of the external container truck lane through the positioning cameras.

[0069] After establishing the camera coordinate system, image coordinate system, pixel coordinate system, and world coordinate system, the feature point images in the field of view can be collected using each positioning camera.

[0070] The unified feature used in the unified external parameter calibration work of the present invention is the fixed lock on the external container truck board. Its distribution on the external container truck board is as Figure 4 shown. Generally, there are 12 locks on the external container truck board, evenly distributed on both sides of the truck board. In different operating states, the positions and quantities of the raised locks on the container truck board are inconsistent. When performing the unified calibration of the positioning cameras, the state of the locks that need to be raised, such as raising the 40-foot position lock and the middle 20-foot position lock, is as Figure 5 shown.

[0071] During unified calibration, within the field of view of each positioning camera, the distribution of the locks and their imaging positions are as Figure 6 shown. Each positioning camera can collect two locks at the near and far ends, and the imaging effect of each lock in the image is clear.

[0072] S3. Respectively extract the lock features in the video image data of the external container truck lane through the target detection method, and obtain the pixel coordinates of the corner points of the lock detection frame.

[0073] The target detection method of the present invention is implemented using a deep learning algorithm in deep learning. Part of the lock image data in the actual scene is collected in advance, labeled, and then the deep learning algorithm model is trained. After detecting and identifying the locks to be measured through the deep learning algorithm model, the four corner point coordinates of the lock detection frame are solved.

[0074] When performing the lock head detection, the selection state of the lock head detection frame is as follows Figure 7 shown. LU, LD, RU, and RD are the four corner points of the lock head detection frame respectively. Under the state where the model prediction is relatively accurate, the lower edge of the lock head detection frame coincides with the lower edge of the lock head imaged in the image, and the upper edge of the lock head detection frame coincides with the upper edge of the lock head imaged in the image. The distribution of the four corner points of the lock head detection frame in the three views of the lock head is as follows Figure 8 shown, where the dots represent the corner points of the detection frame.

[0075] S4. Calculate the world coordinates of the corner points of the lock head detection frame based on the actual size of the lock head.

[0076] First, measure the world coordinate system of a corner point of a detection frame, and then, based on the world coordinate system of a corner point of a detection frame, combined with the actual size of the lock head and the actual distance between the two lock heads in the trolley direction within the same image, deduce the world coordinate systems of other corner points of the detection frame to obtain the world coordinate systems of the 8 corner points of the two lock heads within the same image.

[0077] Specifically, let the actual size of the lock head standard part be height h s , thickness t s , width w s and the distance d in the thickness direction of the corner point s . Within the same image, the actual distance between the two lock heads in the trolley direction is D s . Suppose in the unified image collected by a certain positioning camera A, the pixel coordinates of the corner points of the two lock head detection frames at the near end and the far end are: {P Ani1 , P Ani2 , P Ani3 , P Ani4}, {P Afi1 , P Afi2 , P Afi3 , P Afi4}. The world coordinates corresponding to each corner point are expressed as: {P Anw1 , P Anw2 , P Anw3 , P Anw4}, {P Afw1 , P Afw2 , P Afw3 , P Afw4}, where P Anw1 , P Anw2 , P Anw3 , P Anw4 are the world coordinates of the lower left, lower right, upper right, and upper left corner points of the near-end lock head detection frame respectively, and P Afw1 , P Afw2 , P Afw3 , P Afw4The world coordinates corresponding to the lower left, lower right, upper right, and upper left points of the distal lock detection frame.

[0078] Based on the overlapping relationship between the lock head and the corner points of the detection frame, combined with the relative relationship between the external container lock head and the origin of the world coordinate system, measure the world coordinates of one of the corner points. Assume that the measured world coordinates of a corner point are P Anw2 :{X Anw2 ,Y Anw2 ,Z Anw2}, then the world coordinates of the remaining 7 corner points can be deduced based on {X Anw2 ,Y Anw2 ,Z Anw2}:

[0079] P Anw1 :{X Anw2 ,Y Anw2 +w s ,Z Anw2};

[0080] P Anw3 :{X Anw2 +d s ,Y Anw2 ,Z Anw2 +h s};

[0081] P Anw4 :{X Anw2 +d s ,Y Anw2 +w s ,Z Anw2 +h s};

[0082] P Afw1 :{X Anw2 +D s ,Y Anw2 +w s ,Z Anw2};

[0083] P Afw2 :{X Anw2 +D s ,Y Anw2 ,Z Anw2}

[0084] P Afw3 :{X Anw2 +d s +D s ,Y Anw2 ,Z Anw2 +h s};

[0085] P Afw4 :{XAnw2 +d s +D s ,Y Anw2 +w s ,Z Anw2 +h s}。

[0086] For each positioning camera, obtain the pixel coordinates and world coordinates of the 8 corner points of the 2 lock detection frames in the corresponding image.

[0087] S5. Calculate the external parameters of each positioning camera according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the internal parameters of each positioning camera.

[0088] According to the camera pinhole model and the principle of external parameter calibration, based on the pixel coordinates and world coordinates of the 8 corner points corresponding to each group of calibration cameras, and the internal parameter calibration results K A , K B , K C , K D , obtain the rotation vectors R w , R AWC , R BWC , R CWC , R DWC and the translation vectors T AWC , T BWC , T CWC , T DWC of the coordinate systems of each calibration camera relative to the world coordinate origin C.

[0089] Specifically, according to the camera pinhole model, establish the corresponding relationship equation between the pixel coordinates and world coordinates of the corner points of each lock detection frame based on the internal parameters of the positioning camera. For example, for positioning camera A, there is the following corresponding relationship equation between the pixel coordinates and world coordinates of the 8 corner points:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] Z c is the conversion factor between the positioning camera and the world coordinate system, (u Ani1 ,v Ani1 )、(u Ani2 ,v Ani2 )、(u Ani3 ,v Ani3 )、(u Ani4 ,v Ani4 ) are the corner points P of the lock detection frame respectively Ani1 , P Ani2 , P Ani3 , P Ani4 The corresponding pixel coordinates, (u Afi1 ,v Afi1 )、(u Afi2 ,v Afi2 )、(u Afi3 ,v Afi3 )、(u Afi4 ,v Afi4 ) are the corner points P of the lock detection frame respectively Afi1 , P Afi2 , P Afi3 , P Afi4 The corresponding pixel coordinates, K A is the internal calibration result of positioning camera A, R AWC and T AWC are the rotation vector and translation vector of the camera coordinate system of positioning camera A relative to the world coordinate system.

[0099] The above 8 corresponding relationship equations are combined to establish an equation group, and the equation group is solved by the PnP algorithm in the image processing algorithm to obtain the rotation vector R of the camera coordinate system of the positioning camera A relative to the world coordinate system. AWC and translation vector T AWC , that is, the camera external parameters are obtained.

[0100] The other positioning cameras use the same method to establish corresponding equations between the pixel coordinates of the eight corner points and the world coordinates. These eight corresponding equations form a system of equations, and solving this system yields the rotation vector and translation vector of the corresponding positioning camera's camera coordinate system relative to the world coordinate system. At this point, all four positioning cameras are calibrated to the same world coordinate system, achieving unified calibration for the automated crane's external container truck positioning cameras.

[0101] According to the positions of the locks on the decks of 20-foot and 40-foot external container trucks under the standard operating conditions of external container trucks, the present invention installs four positioning cameras on the crane gantry legs. Taking the fixed locks on the external container truck decks as the unified feature, two locks within their respective fields of view are respectively located through deep learning methods. According to the conversion relationship between the pixel coordinates and the world coordinates of the locks, combined with the external parameter solution algorithm, the camera coordinate systems of multiple positioning cameras are unified into the world coordinate system, realizing the unified calibration of the external container truck positioning cameras.

[0102] Compared with the existing multi-camera pose solution methods, the method of the present invention has stronger stability, higher efficiency, smaller calculation errors, and can more reasonably calculate the pose of the external container truck positioning camera relative to the container truck lock. Therefore, on the basis of realizing the unified calibration of the external container truck positioning cameras of the automated crane, the locks' features in the picture can be recognized by the four positioning cameras, and using the calibration results of the internal and external parameters of the cameras, the pixel coordinates of the locks in the picture are converted into the crane world coordinates, and the position of the external container truck relative to the crane is calculated to realize functions such as container truck video guidance.

[0103] Corresponding to the above method embodiment, the present invention also proposes a unified calibration system for the external container truck positioning cameras of an automated crane, and the system includes:

[0104] Positioning cameras: Installed on the crane gantry legs, and the locks on the external container truck deck are within the fields of view of the corresponding positioning cameras, for collecting video image data of the external container truck lane;

[0105] Feature extraction module: Used to respectively extract the lock features in the video image data of the external container truck lane through object detection methods, and obtain the pixel coordinates of the corner points of the lock detection frame;

[0106] Coordinate conversion module: Used to calculate the world coordinates of the corner points of the lock detection frame based on the actual size of the lock;

[0107] External parameter calibration module: Used to calculate the external parameters of each positioning camera respectively according to the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the internal parameters of each positioning camera.

[0108] The above system embodiment and method embodiment are in one-to-one correspondence. For the brief description of the system embodiment, please refer to the method embodiment.

[0109] The present invention also discloses an electronic device, including: at least one processor, at least one memory, a communication interface, and a bus; wherein, the processor, the memory, and the communication interface complete mutual communication through the bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the method described above of the present invention.

[0110] The present invention also discloses a computer-readable storage medium, which stores computer instructions that enable the computer to implement all or part of the steps of the method described in the embodiments of the present invention. The storage medium includes various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0111] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed over multiple network units. Those of ordinary skill in the art can, without creative effort, select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A unified calibration method for the external container yard truck positioning cameras of an automated crane, characterized in that, The method comprises: According to the position of the lock on the truck board under the standard operating conditions of the truck, four positioning cameras calibrated with internal parameters are installed on the crane door legs to ensure that the lock on the truck board is within the field of view of the corresponding positioning cameras; The video image data of the external truck lanes are collected by positioning cameras respectively; The lock features in the outbound truck lane video image data are extracted by the target detection method, and the pixel coordinates of the corner points of the lock detection frame are obtained; Calculate the world coordinates of the corner points of the lock detection frame based on the actual size of the lock, specifically including: Establish a world coordinate system with the center of the lane side door leg as the origin, and establish a camera coordinate system for each positioning camera; Measure the world coordinates of a corner point based on the overlap between the lock head and the corner points of the detection frame; Based on the world coordinates of one detection frame corner point, combined with the actual size of the padlock and the actual distance between the two padlocks in the same image in the direction of the vehicle, the world coordinates of the other detection frame corner points are calculated. The world coordinates of the eight detection frame corner points of the two padlocks in the same image are obtained, specifically: Let the actual dimensions of the lock standard parts be height h s , thickness t s , width w s and the distance d in the thickness direction of the corner point s . In the same image, the actual distance between the two locks in the trolley direction is D s ; In a unified image collected by a certain positioning camera A, the corner pixel coordinates of two lock detection frames at the near end and the far end are: {P Ani1 , P Ani2 , P Ani3 , P Ani4}, {P Afi1 , P Afi2 , P Afi3 , P Afi4}. The world coordinates corresponding to each corner point are represented as: {P Anw1 , P Anw2 , P Anw3 , P Anw4}, {P Afw1 , P Afw2 , P Afw3 , P Afw4}, where P Anw1 , P Anw2 , P Anw3 , P Anw4 are respectively the lower left, lower right, upper right and upper left points of the near-end lock detection frame, and P Afw1 , P Afw2 , P Afw3 , P Afw4 correspond to the lower left, lower right, upper right and upper left points of the far-end lock detection frame; Measure the world coordinates P of a corner point based on the overlapping relationship between the lock head and the corner points of the detection frame Anw2 :{X Anw2 ,Y Anw2 ,Z Anw2}, and deduce the world coordinates of the remaining 7 corner points based on {X Anw2 ,Y Anw2 ,Z Anw2}: P Anw1 : {X Anw2 , Y Anw2 + w s , Z Anw2}; P Anw3 : {X Anw2 +d s , Y Anw2 , Z Anw2 +h s}; P Anw4 : {X Anw2 +d s , Y Anw2 +w s , Z Anw2 +h s}; P Afw1 : {X Anw2 + D s , Y Anw2 + w s , Z Anw2}; P Afw2 : {X Anw2 + D s , Y Anw2 , Z Anw2} P Afw3 : {X Anw2 +d s +D s , Y Anw2 , Z Anw2 +h s}; P Afw4 : {X Anw2 +d s +D s , Y Anw2 +w s , Z Anw2 +h s} The extrinsic parameters of each positioning camera are calculated based on the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the intrinsic parameters of each positioning camera.

2. The unified calibration method for the external yard truck positioning cameras of the automated crane according to claim 1, characterized in that According to the position of the lock on the truck board under the standard operating conditions of the truck, four positioning cameras calibrated with internal parameters are installed on the crane door legs to ensure that the lock on the truck board is within the field of view of the corresponding positioning cameras. Specifically, With the center plane of half the long side of the 20-foot and 40-foot containers as the symmetry planes, four positioning cameras are symmetrically set at a height of 1.5 to 2 meters from the ground on the door legs on the side of the crane's external container truck operating lane. It is ensured that under standard operating conditions of the external container truck, for different containers and operation types, each positioning camera can capture a pair of lock features set opposite to each other on the external container truck plate within its field of view.

3. The unified calibration method of the external container crane positioning camera of the automated crane according to claim 1, characterized in that The target detection method is implemented using a deep learning algorithm. After detecting and identifying the lock head, the coordinates of the four corner points of the unlocking head detection frame are calculated.

4. The unified calibration method for the external container truck positioning cameras of the automated crane according to claim 1, characterized in that, The calculation of the external parameters of each positioning camera based on the pixel coordinates of the corner points of the lock detection frame, the world coordinates of the corner points of the lock detection frame, and the internal parameters of each positioning camera specifically includes: According to the camera pinhole model, the corresponding relationship equation between the pixel coordinates of each lock detection frame corner point and the world coordinates is established based on the positioning camera intrinsic parameters; The corresponding relationship equations are combined to establish a system of equations, which are solved using the PnP algorithm in the image processing algorithm to obtain the rotation vector and translation vector of the camera coordinate system of each positioning camera relative to the world coordinate system.

5. The unified calibration method for the external container crane positioning cameras of the automated crane according to claim 4, characterized in that, The corresponding relationship equation between the pixel coordinates of each lock detection frame corner point and the world coordinates is established based on the camera pinhole model and the positioning camera intrinsic parameters, specifically including: For any positioning camera A, there is the following corresponding equation: Z c To detect the conversion factor between the camera and the world coordinate system, (u Ani1 , v Ani1 ), (u Ani2 , v Ani2 ), (u Ani3 , v Ani3 ), (u Ani4 , v Ani4 ) are the pixel coordinates corresponding to the corner points P Ani1 , P Ani2 , P Ani3 , P Ani4 of the lock head detection frame respectively. (u Afi1 , v Afi1 ), (u Afi2 , v Afi2 ), (u Afi3 , v Afi3 ), (u Afi4 , v Afi4 ) are the pixel coordinates corresponding to the corner points P Afi1 , P Afi2 , P Afi3 , P Afi4 of the lock head detection frame respectively. K A is the calibration result of the camera internal parameters. P AWC and T AWC are the rotation vector and translation vector of the camera coordinate system of the positioning camera A relative to the world coordinate system respectively.

6. An external truck positioning camera unified calibration system for an automated crane, which is used to implement the calibration method described in any one of claims 1 to 5, and is characterized in that, The system comprises: Positioning camera: It is installed on the crane door leg, and the lock on the truck board is within the field of view of the corresponding positioning camera, which is used to collect video image data of the truck lane; Feature extraction module: used to extract the lock head features in the video image data of the outer container truck lane respectively through the object detection method, and obtain the pixel coordinates of the corner points of the lock head detection frame; Coordinate conversion module: used to calculate the world coordinates of the corner points of the lock head detection frame based on the actual size of the lock head; External parameter calibration module: used to calculate the external parameters of each positioning camera respectively according to the pixel coordinates of the corner points of the lock head detection frame, the world coordinates of the corner points of the lock head detection frame and the internal parameters of each positioning camera.

Images