A method and system for estimating the pose of containers and spreaders based on keypoint regression.
By using a keypoint regression-based method and geometric and temporal verification techniques, the problem of pose detection accuracy for spreaders and containers in complex environments was solved, achieving high-precision pose estimation and stable automated control.
Patent Information
- Application Number
- CN202611219603.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-10
AI Technical Summary
Existing methods for detecting the relative posture between spreaders and containers lack precision, especially under conditions of changing lighting, shading, and dirt, making it difficult to accurately identify key structural points of the container and spreaders, leading to difficulties in inserting locking pins or the risk of collision.
A keypoint regression-based approach is adopted to directly identify key structural points of containers and spreaders through a pre-trained keypoint regression model. Combined with geometric constraints and temporal continuity verification, the pose deviation between the container and the spreader is calculated, and a Kalman filter is used for filtering to output the pose deviation result of the spreader relative to the container.
In complex operating environments, high-precision alignment detection between spreader and container was achieved, improving detection accuracy and real-time control, reducing the risk of misjudgment due to lighting and occlusion, and ensuring the stability and safety of automated control.
Smart Images

Figure CN123071433A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of port automation control technology. Specifically, it relates to a method and system for estimating the pose of containers and spreaders based on key point regression. It can be used for alignment detection, attitude measurement and auxiliary control in container operations of quay cranes, yards, rail-mounted gantry cranes, rubber-tired gantry cranes and automated yard spreaders. Background Technology
[0002] In container loading and unloading operations, the spreader must be accurately aligned with the corner fittings on the top of the container before the locking pin can be inserted, locked, and lifted. If there is a large lateral deviation, longitudinal deviation, rotation angle deviation, or height and posture deviation between the spreader and the container, the locking pin may not be able to accurately enter the corner fitting hole, or even cause collisions, scratches, uneven load on the spreader, or interruption of operations.
[0003] In existing technologies, the main methods for detecting the relative attitude between the spreader and the container include:
[0004] (1) Manual observation method: relies on the driver's experience and is greatly affected by perspective, lighting, obstruction and fatigue.
[0005] (2) LiDAR or ranging sensor method: can obtain distance information, but the equipment cost is high and it is sensitive to reflection, rain and fog, installation location and calibration accuracy.
[0006] (3) Traditional image recognition methods, such as edge detection, template matching, Hough transform, etc., are usually more sensitive to changes in lighting, dirt on the box, occlusion, box number text, corrosion and background interference.
[0007] (4) Target detection frame method: It can detect the overall position of the container or spreader, but it is difficult to accurately calculate the attitude angle, corner position and fine-grained deviation between the spreader and the container by relying solely on the rectangular frame. Summary of the Invention
[0008] To address the aforementioned existing detection methods, this invention proposes a container and spreader pose estimation method and system based on key point regression. This method directly identifies key structural points of the container and spreader, and calculates the pose and deviation based on the coordinates of the key points, thereby improving the recognition accuracy, robustness, and automated control capabilities of the spreader during container handling.
[0009] The present invention is implemented using the following technical solutions:
[0010] A method for estimating the pose of containers and spreaders based on keypoint regression is proposed, including:
[0011] S1, acquire images of the container and the spreader;
[0012] S2, input the container image and the spreader image into the pre-trained keypoint regression model to obtain the container keypoints and the spreader keypoints;
[0013] S3, calculate container pose data based on container key points; the container pose data includes the container center point and container attitude angle;
[0014] S4, Calculate the spreader's pose data based on the key points of the spreader; the spreader's pose data includes the center point of the spreader and the spreader's attitude angle;
[0015] S5, transform the image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle;
[0016] S6 outputs the positional deviation of the spreader relative to the container to the control system.
[0017] Compared with existing technologies, the advantages and positive effects of this invention are as follows: The container and spreader pose estimation method based on keypoint regression proposed in this invention acquires images of the container and spreader and inputs them into a keypoint regression model, directly outputting the coordinates of key points with clear geometric meaning, such as container corner fittings and spreader locking pins. It does not rely on traditional image processing methods such as manually designed features or edge detection, and can adapt to changes in visual features under different lighting conditions, container colors, and soiling conditions, thus accurately identifying target structural points even in complex operating environments. Furthermore, this invention calculates the center point and attitude angle of the container and spreader based on these key points, and calculates positional and angular deviations by converting image coordinates into physical coordinates in the operating coordinate system. This gives the deviation results a true physical scale meaning rather than remaining at the image pixel level. The final output pose deviation results can be directly used as input to the automatic or semi-automatic assisted container alignment control system, achieving seamless integration from visual perception to control feedback, effectively improving the detection accuracy and real-time control performance during the alignment process between the spreader and the container.
[0018] In some embodiments of the present invention, the method further includes a step of performing geometric constraint verification on the container pose data, including:
[0019] Side length constraints: Calculate the length ratio of adjacent sides of the container based on key points, and eliminate key point combinations whose length ratios do not conform to the container's length-to-width ratio; and / or,
[0020] Parallel constraint: Calculate the direction vector of the opposite side of the container based on the key points of the container, and eliminate key point combinations whose direction vectors do not conform to the parallel relationship.
[0021] This embodiment further introduces side length constraints and parallelism constraints as geometric verification methods based on keypoint regression. By utilizing the inherent geometric properties of containers as standard industrial products, such as fixed aspect ratios and parallelism of opposite sides, the rationality of keypoint combinations output by the model is verified. When the deviation between the aspect ratio calculated from the keypoint combination and the standard aspect ratio of the container exceeds the allowable error range, or when the parallelism between the direction vectors of opposite sides is lower than a preset threshold, it indicates that there are false detections or large offsets in this set of keypoints. In this case, abnormal keypoint combinations are removed to prevent erroneous keypoints from participating in subsequent attitude calculations and deviation calculations. This geometric constraint verification does not rely on additional sensors or manual intervention; it can achieve anomaly detection entirely based on the prior geometric information of the container itself. This effectively avoids the transmission of misjudgments of keypoints caused by partial occlusion, container surface contamination, or model prediction deviations to the attitude calculation results, significantly improving the output reliability and anti-interference capability of the system under harsh visual conditions.
[0022] In some embodiments of the present invention, the method further includes a step of applying temporal continuity constraints to the container pose data, including:
[0023] If the coordinate difference of the same key point in adjacent frames is greater than a preset value, the coordinates of the key point are corrected or removed using historical data.
[0024] This embodiment further applies temporal continuity constraints to key points based on geometric constraint verification. Leveraging the physical continuity and smooth motion of pose changes between the spreader and container during continuous operation, the coordinate change of the same key point between adjacent frames is used as an anomaly criterion. When the positional change of a key point relative to the same point in the previous frame exceeds a preset threshold, it indicates that the point may have a jump due to instantaneous occlusion, motion blur, or model prediction jitter. In this case, coordinate data from historical frames can be used to correct the anomaly point in the current frame or directly remove the anomaly. This temporal constraint mechanism effectively suppresses sporadic false detections and jitter in single-frame detection, ensuring the smooth consistency of pose estimation results in the time dimension, and further improving the stability of the system's continuous output deviation during spreader movement.
[0025] In some embodiments of the present invention, the pre-trained keypoint regression model outputs keypoints via heatmaps:
[0026] ;in, For key point locations, The standard deviation of the Gaussian kernel. Represents pixels Belongs to the Thermal response values at key points;
[0027] The training loss uses mean squared error:
[0028] ;in, Predict heatmaps for the model;
[0029] Key points The position is obtained through the position of the maximum response: .
[0030] This embodiment uses heatmaps as the output of the keypoint regression model, generating a two-dimensional Gaussian response map for each keypoint. The peak positions correspond to the precise locations of the keypoints in the image. Compared to directly regressing coordinate values, heatmaps preserve accurate spatial location information. The model is trained by minimizing the mean squared error between the predicted heatmap and the Gaussian heatmap constructed centered on the actual keypoints. This allows the network to fully learn the response distribution patterns of pixels surrounding the keypoints rather than simply fitting single coordinate values, enhancing the model's ability to perceive contextual features near the keypoints. During the prediction phase, the keypoint coordinates are obtained by finding the location of the maximum response in the heatmap. This allows the model to maintain high positioning accuracy while possessing good noise resistance. Even if the input image has a certain degree of blurring or noise interference, the Gaussian response region in the heatmap can still effectively indicate the true location of the keypoints, thereby improving the robustness and positioning accuracy of keypoint detection.
[0031] In some embodiments of the present invention, S5 specifically includes:
[0032] by Representing image coordinates, in Represents the planar coordinates of container operations;
[0033] If satisfied or ,get After normalization, we get , ;
[0034] For the center point of the container and the center point of the lifting gear Calculate the plane deviation as , ;
[0035] in, It is a homography matrix. is the scale factor.
[0036] This embodiment utilizes a homography matrix to map the coordinates of key points on the image plane to the coordinate system of the working plane. A one-to-one correspondence between image pixel positions and physical positions on the working plane is established through the homography matrix H obtained through calibration. The coordinate transformation can be completed using only a monocular camera, resulting in simple hardware configuration, low cost, and minimal computational load. The calculated planar deviations ΔX and ΔY between the container center and the spreader center are no longer pixel distances on the image plane, but rather physical deviations with actual length units on the working plane. This scheme is suitable for working scenarios where the top of the container is considered a plane, efficiently completing the conversion from image coordinates to physical coordinates without requiring depth information, providing an accurate spatial reference for deviation calculation.
[0037] In some embodiments of the present invention, S5 specifically includes:
[0038] based on and Transform the image point coordinates to the camera coordinate system; where... The coordinates of the image points, Depth in camera coordinate system The pixel measure of the camera's focal length on the image plane. These are the pixel coordinates of the intersection of the optical axis and the imaging plane.
[0039] Calculate the three-dimensional deviation between the spreader center point and the container center point:
[0040] ;in The three-dimensional coordinates of the centerline point of the spreader. The coordinates of the container's center point are given.
[0041] This embodiment establishes a back-projection relationship from image pixel coordinates to 3D coordinates in the camera coordinate system. Utilizing known camera intrinsic parameters and the depth value Z at key points, it uniquely maps 2D image points to 3D spatial points in the camera coordinate system. Compared to planar homography transformation, this 3D back-projection method can recover the complete 3D coordinate information of key points. It can not only calculate lateral and longitudinal deviations but also directly obtain the height deviation ΔZ between the spreader and the container, thus achieving accurate perception of the relative position of the spreader and container in the vertical direction. This 3D coordinate information can be further used to fit the spreader plane and the container top plane, calculating the angle between the normal vectors of the two planes. This provides a data foundation for calculating attitude deviations, enabling the system to more comprehensively perceive the six-degree-of-freedom relative pose relationship between the spreader and the container.
[0042] In some embodiments of the present invention, S5 further includes:
[0043] Define the unit direction vector of the long side of the container. and the unit direction vector of the short side of the container ;in, Let be the long-side direction vector of the container in the operating coordinate system;
[0044] Calculate the longitudinal deviation of the plane Lateral deviation of the plane ;in, , and For planar deviation;
[0045] Output relative deviation ;in, This refers to the height deviation between the spreader and the container. This represents the rotational angle deviation of the spreader relative to the container in the operating coordinate system.
[0046] This embodiment further projects the planar deviation vector ΔP in the working coordinate system onto the long and short sides of the container itself. By defining a unit direction vector for the long side of the container in the working coordinate system and rotating it by 90° to obtain the unit direction vector for the short side, and then using vector dot product operations, the two-dimensional deviation in the global coordinate system is decomposed into two independent components along the length and width directions of the container. This decomposition method makes the output longitudinal and lateral deviations directly related to the container's own geometry, and the coordinate system, after coordinate transformation, becomes the working coordinate system with a clear physical meaning. Since both the longitudinal and lateral deviations are projections calculated based on physical coordinates, while ΔZ is the height difference calculated based on three-dimensional coordinates, and Δθ is the rotation angle deviation recalculated based on physical coordinates, these four together constitute a complete deviation vector defined along the container's own coordinate system with real physical dimensions. This can be directly used as the closed-loop feedback input of the spreader to the container control system, avoiding additional coordinate transformations or unit conversions by the control system, significantly reducing the interface complexity and real-time computational burden of the control system.
[0047] In some embodiments of the present invention, the method prior to S6 further includes filtering and outlier removal steps, including:
[0048] The position deviation and the angle deviation are filtered using a Kalman filter;
[0049] When the prediction bias exceeds a preset threshold relative to the filtering result of the previous frame, the bias observation value of the current frame is marked as a low confidence observation.
[0050] During the Kalman filter update process, reduce the weight of the low-confidence observations, or pause the observation update and retain only the state prediction results.
[0051] This embodiment employs a Kalman filter to perform temporal filtering on the deviation calculation results. By constructing a state vector containing the deviation amount and the rate of change of deviation, the deviation at the current moment is predicted using a state transition matrix. The prediction result is then weighted and updated in conjunction with the observations of the current frame, ensuring that the final output deviation result reflects both the detection information of the current frame and the changing trends of historical frames. When the detection system determines that the current frame is a low-confidence observation, the observation noise covariance is increased or the observation update is directly paused during the filtering update process. This makes the filter output primarily rely on the kinematic predictions of historical states, temporarily reducing the impact of the current abnormal observation. Once subsequent frame detection returns to normal, the normal observation update weights are automatically restored. This filtering mechanism enables the system to output a smooth and physically reasonable deviation amount even when dealing with instantaneous occlusion, occasional false detections, or brief jitter, avoiding malfunctions or oscillations in the control system caused by instantaneous jumps in visual detection, effectively ensuring the safety and smoothness of automated hoisting operations.
[0052] In some embodiments of the present invention, when the keypoint regression model outputs container keypoints and spreader keypoints, it also outputs the confidence level of each keypoint; the threshold of the confidence level is dynamically adjusted based on illumination, motion blur and historical stability.
[0053] This embodiment outputs a confidence score for each keypoint and dynamically adjusts the confidence score threshold based on lighting conditions, motion blur, and historical detection stability, achieving adaptive evaluation of model prediction quality. When ambient lighting is good, the image is clear, and the historical consistency of detection results is high, the system automatically adopts a high confidence score threshold to strictly screen keypoints, ensuring that all detection results participating in the bias calculation are high-quality. Conversely, when lighting is poor, the image has motion blur, or historical detection fluctuations are large, the system automatically lowers the confidence score threshold to adapt to the actual situation where the model's confidence score generally decreases under difficult conditions, avoiding the situation where no usable keypoints are output due to an excessively high threshold. This adaptive mechanism enables the system to automatically adjust the detection admission criteria under different operating environments and conditions, avoiding both the problem of introducing low-quality keypoints under excellent conditions with a fixed threshold and the problem of excessive rejection under poor conditions with a fixed threshold, effectively expanding the system's environmental adaptability and effective working time window.
[0054] This invention also proposes a container and spreader pose estimation system based on keypoint regression, comprising:
[0055] An image acquisition device is installed in the container and spreader operation area to acquire images of the container and spreader.
[0056] The keypoint regression module is used to input container images and spreader images into a pre-trained keypoint regression model to obtain container keypoints and spreader keypoints.
[0057] The pose estimation module is used to calculate container pose data based on container key points and spreader pose data based on spreader key points; wherein, the container pose data includes the container center point and container attitude angle; the spreader pose data includes the spreader center point and spreader attitude angle.
[0058] The deviation calculation module is used to transform the image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle.
[0059] The output module is used to output the positional deviation of the spreader relative to the container to the control system.
[0060] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description
[0061] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions are used to explain the invention but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments; those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0062] Figure 1 This diagram illustrates the execution steps of the container and spreader pose estimation method based on keypoint regression proposed in this invention.
[0063] Figure 2 This is a schematic diagram of the execution steps of the container and spreader pose estimation method based on keypoint regression according to an embodiment of the present invention;
[0064] Figure 3 This is a schematic diagram of the execution steps of the container and spreader pose estimation method based on keypoint regression according to an embodiment of the present invention;
[0065] Figure 4 This is a schematic diagram of the system structure of the container and spreader attitude estimation system based on key point regression proposed in this invention. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0067] The container and spreader pose estimation method based on keypoint regression proposed in this invention, such as... Figure 1 As shown, it includes the following steps:
[0068] S1: Acquire images of the container and the spreader.
[0069] Image acquisition equipment includes, but is not limited to, monocular / multi-view cameras (industrial cameras, wide-angle cameras, or depth cameras, etc.) installed in fixed positions on spreaders, quay crane trolleys, quay crane beams, or work areas.
[0070] In this invention, the acquired images of the spreader and container are first preprocessed to perform operations such as distortion correction, cropping, brightness normalization, noise reduction, and scale transformation.
[0071] In this invention, the input image is represented as ,in, Indicates the image height. Indicates the image width. Indicates the number of channels, for example, in an RGB image. =3.
[0072] S2: Input the container image and the spreader image into the pre-trained keypoint regression model to obtain the container keypoints and the spreader keypoints.
[0073] In this embodiment of the invention, the key points of the container are defined as follows, taking the four corners of the top of the container as an example:
[0074]
[0075] in ; The left front corner point or the center of the left front corner component of the container. The right front corner point or the center of the right front corner component of the container. The right rear corner of the container or the center of the right rear corner component. The left rear corner point or the center of the left rear corner component of the container.
[0076] Taking the four locking pin centers of the spreader as an example, the set of key points of the spreader is defined as follows:
[0077]
[0078] in ; The center of the left front locking pin of the spreader. The center of the right front locking pin of the spreader. The center of the right rear locking pin of the spreading device. The center of the left rear locking pin of the lifting device.
[0079] To enhance the robustness of the system when some key points are occluded, this invention also defines an extended set of key points:
[0080]
[0081] in: As the center point of the container, Center point of the spreader , The midpoint of the long side of the container. , This refers to the midpoint of the spreader's long side. Extending keypoints provides additional geometric constraint information when some of the four main corner points are not visible, improving the system's fault tolerance. For example, when the left rear corner of the container is obscured by the spreader body, the approximate location of the obscured corner can be inferred from the symmetry between the container's center point and the other three corner points.
[0082] The final set of key points is the union of the three sets of key points mentioned above:
[0083]
[0084] In this embodiment of the invention, the keypoint regression model is built upon a deep neural network to directly predict the coordinates of keypoints with clear geometric meaning on a target object from an input image. The model is represented as follows:
[0085]
[0086] in: For parameters Neural network model; The set of key points predicted by the model; Number of key points; For the first The positions of key points in the image coordinate system; For the first Confidence level of each key point.
[0087] In this embodiment of the invention, the model outputs key points using a heatmap method. Specifically, the model outputs a heatmap for each key point, with the key point location corresponding to a peak position in the heatmap. The actual heatmap of each key point is constructed as follows:
[0088]
[0089] in: This represents the actual location of key points; This is the Gaussian kernel standard deviation, used to control the degree of diffusion in the response region of the thermogram. A larger one... A larger value makes the response area of the heatmap wider, making it easier for the model to converge during training, but the localization accuracy may decrease; a smaller value... The value makes the response area more concentrated and the positioning accuracy higher, but it increases the training difficulty. Represents pixels Belongs to the The thermal response value of each key point is a key point. The larger the thermal response value, the more likely the pixel is to be near that key point, and the range is (0,1).
[0090] The model predicts the following heatmap:
[0091]
[0092] The training loss uses mean squared error:
[0093]
[0094] This loss function compares the difference between the predicted heatmap and the actual heatmap pixel by pixel, updating the network parameters through backpropagation, so that the predicted heatmap gradually approximates the distribution pattern of the actual heatmap. Because the heatmap has a continuous distribution of response values around the actual locations of keypoints, the model can fully utilize gradient information in the neighborhood of the keypoints during training. Compared to directly regressing coordinate values, the heatmap supervision signal provides richer spatial constraints, which helps improve the model's localization accuracy and generalization ability.
[0095] Prediction phase, the first The locations of key points are obtained through the location of the maximum response:
[0096]
[0097] Since the heatmap outputs response values at the discrete pixel level, this invention employs a Soft-Argmax operation for optimization to further improve positioning accuracy to the sub-pixel level.
[0098]
[0099]
[0100] in:
[0101]
[0102] This is a temperature parameter used to control the sharpness of the response peak. When When the value is large, Soft-Argmax approaches the operation of directly taking the position of the maximum value, resulting in high positioning accuracy but potentially sparse gradients; when... When the value is small, the response distribution is smoother and the gradient is more stable, but the positioning accuracy decreases slightly.
[0103] Soft-Argmax calculates key point coordinates by weighted averaging of heatmap response values, making the coordinate estimation results insensitive to noise in individual pixels and achieving sub-pixel level positioning accuracy, meeting the needs for millimeter-level deviation detection in sling-to-box operations.
[0104] As a common implementation method, the keypoint regression model can also use a fully connected layer to directly regress the keypoint coordinates. In this method, the image features are input into the fully connected layer after global average pooling or flattening, and the coordinate values are directly output.
[0105] The model's training data is collected from various operating environments involving spreader and container operations, including but not limited to: daytime, nighttime, backlighting, rain, and fog conditions; different container types such as 20-foot, 40-foot, and 45-foot; different container colors, dirt, rust, and marking obstructions; spreader heights, offsets, and rotation angles; different operational stages such as spreader approach to containers, lock pin alignment, and before and after locking; and single-container, double-container, or stacked container scenarios. This diverse data collection enables the trained model to adapt to various complex working conditions in port operations, eliminating the need for additional adaptations for specific operating conditions during actual deployment.
[0106] Manually annotate key points in the acquired images:
[0107]
[0108] in: The coordinates of key points are manually labeled; This indicates that key points are visible; This indicates that the key point is not visible or is obscured.
[0109] During model training, various data augmentation techniques were employed to enhance generalization ability, including random rotation, random scaling, random translation, color dithering, and Gaussian noise addition. These data augmentation techniques simulated various image changes that might occur in real-world scenarios, enabling the model to learn feature representations that are invariant to changes in viewpoint, scale, and illumination.
[0110] In this embodiment of the invention, combined with Figure 2 As shown, each keypoint output by the keypoint regression model carries a confidence score. This indicates the reliability of the keypoint prediction result. The system determines the reliability based on a preset confidence threshold. Preliminary screening of key points: If If so, then the key point is considered unreliable.
[0111] In this embodiment, the confidence threshold is dynamically adjusted based on lighting conditions, motion blur level, and historical detection stability. Specifically:
[0112]
[0113] in, It is an image blur index, which can be calculated based on the image edge gradient energy; The target motion speed index can be estimated based on the displacement of key points between adjacent frames; It serves as a historical stability indicator and can be calculated based on the variance of confidence levels over several past frames. , , This is the adjustment coefficient. This adaptive mechanism allows the system to use a higher threshold to ensure accuracy when the image quality is good, and to appropriately lower the threshold to ensure usability when the image quality deteriorates, thus achieving a dynamic balance between accuracy and robustness.
[0114] After confidence level screening, the system further uses geometric constraints to verify the spatial rationality of key points.
[0115] Side length constraint: The lengths of adjacent sides of the container should satisfy a geometric proportional relationship. Let:
[0116]
[0117]
[0118] For standard 20-foot or 40 / 45-foot containers, the length-to-width ratio must meet the following requirements:
[0119]
[0120] like If so, it is considered that the combination of key points is abnormal, in which For standard aspect ratio, To allow for error, the side length constraint utilizes the prior information that shipping containers, as standard industrial products, have fixed geometric dimensions. This effectively eliminates key point combinations with severely distorted aspect ratios caused by false corner detections, and prevents physically meaningless corner configurations from participating in subsequent attitude calculations.
[0121] Parallel constraint: Opposite edges of the container should be approximately parallel.
[0122]
[0123] like If the edges are not aligned, then the edges are considered to be in the same direction. The parallel constraint threshold is defined as follows: the closer the value is to 1, the stricter the parallelism requirement. Based on the geometric characteristics of the rectangular structure of a container, the parallel constraint can eliminate key point combinations that result in non-parallelism of opposite sides or structural distortion due to corner detection errors.
[0124] Temporal continuity constraint: The positional changes of key points between adjacent frames should not be abrupt. Let the... Frame key points are , No. Frame key points are ,like If the key point is considered to be an outlier, it needs to be corrected or discarded based on historical values. This is the threshold for temporal continuity constraints. Temporal constraints utilize prior knowledge that the movement of the spreader and the container is physically continuous, which can effectively suppress occasional detection jumps in a single frame.
[0125] The keypoints are verified step by step using the three constraints mentioned above. When a keypoint does not meet any of the constraints, its confidence level is used to determine whether to mark it as an anomaly and remove or correct it. For keypoints that are occluded or temporarily out of the field of view, if their position can be reasonably estimated through temporal prediction, they are not removed and the predicted value is used instead; if they cannot be recovered in multiple consecutive frames, a corresponding status signal is sent to the control system.
[0126] After obtaining the coordinates of the key points in the image coordinate system, the initial attitude angle calculation is first performed in the image coordinate system.
[0127] S3: Calculate container pose data based on container key points.
[0128] Container orientation data includes the container center point and container attitude angle.
[0129] (1) The center point of the container is obtained by averaging the four corner points:
[0130]
[0131]
[0132]
[0133] The method for calculating the center point utilizes the symmetry of the coordinates of the four corner points. When the detection errors of the four corner points are independent of each other and follow a zero-mean distribution, the averaging operation can effectively suppress random errors and improve the positioning accuracy of the center point.
[0134] (2) The long side direction vector of the container can be represented in the image coordinate system as:
[0135]
[0136]
[0137] The container's direction vector is then the average of the direction vectors of its two longer sides:
[0138]
[0139] Right now .
[0140] The direction vector is calculated by averaging the two longer side vectors, which can offset the effect of the non-parallelism between the left and right sides caused by perspective projection, making the estimation of the direction vector more robust.
[0141] The visual pose angle of the container in the image coordinate system is:
[0142]
[0143] S4: Calculate the position and posture data of the lifting device based on the key points of the lifting device.
[0144] The spreader position data includes the spreader center point and the spreader attitude angle.
[0145] (1) The center point of the lifting device is:
[0146]
[0147]
[0148]
[0149] (2) Similarly, the direction vector of the lifting device is:
[0150]
[0151]
[0152]
[0153] Right now .
[0154] The visual attitude angle of the lifting device in the image coordinate system is:
[0155]
[0156] The visual angle deviation between the container and the spreader in the image coordinate system is:
[0157]
[0158] To avoid angle jumps, it is normalized to :
[0159]
[0160] Depending on the situation, it can be converted into an angle-based output:
[0161]
[0162] S5: Transform the image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle.
[0163] The image deviation between the spreader center and the container center is:
[0164]
[0165] Right now:
[0166]
[0167]
[0168] However, because image coordinates are affected by perspective, directly using... and This can only serve as a rough estimate. To obtain the actual physical deviation, camera calibration and coordinate transformation are required.
[0169] In this embodiment of the invention, two coordinate transformation implementation methods are provided, which can be flexibly selected according to on-site hardware conditions and operational requirements.
[0170] Method 1, Planar coordinate transformation based on homography matrix: Treat the top of the container or the working plane as a plane, and convert the image coordinates into working plane coordinates through the homography matrix.
[0171] Let the image coordinates be...
[0172]
[0173] The working plane coordinates are
[0174]
[0175] Both conditions are met:
[0176]
[0177] or:
[0178]
[0179] in, Homography matrix, The scaling factor is obtained through pre-calibration. After matrixing, for any image point, it expands to:
[0180]
[0181] After normalization, the coordinates of the working plane are obtained:
[0182]
[0183]
[0184] For the center point of the container and the center point of the lifting gear The actual plane deviation is:
[0185]
[0186]
[0187] This method only requires a monocular camera and a calibrated homography matrix to complete the coordinate transformation. It has low hardware costs and low computational requirements, and is suitable for operation scenarios where the top of the container is approximated as a plane.
[0188] Method 2: 3D coordinate transformation based on camera intrinsic and extrinsic parameters: combining keypoint depth information obtained from ranging devices, depth cameras, or depth estimation networks. This converts image points into 3D points in the camera coordinate system.
[0189] The camera imaging model is as follows:
[0190]
[0191] in Image pixel coordinates, For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. , , As a world coordinate system, is the scale factor.
[0192] The camera intrinsic parameter matrix is:
[0193]
[0194] in, and The pixel measure of the camera's focal length in the horizontal and vertical directions of the image. and The pixel coordinates of the principal point (the intersection of the optical axis and the imaging plane).
[0195] If the depth is known The three-dimensional coordinates in the camera coordinate system are obtained by the following formula:
[0196]
[0197]
[0198] The three-dimensional coordinates corresponding to the center point of the spreader are:
[0199]
[0200] The three-dimensional coordinates corresponding to the center point of the container are:
[0201]
[0202] The three-dimensional deviation is:
[0203]
[0204] Right now:
[0205]
[0206]
[0207]
[0208] in This refers to the height deviation between the spreader and the container. This method can recover the complete 3D information of key points and is suitable for scenarios requiring height deviation and 3D pose estimation.
[0209] In this embodiment of the invention, in order to make the deviation results more in line with the requirements of the spreader control system, the planar deviation is further projected onto the container's own coordinate system.
[0210] Let the long side direction vector of the container, calculated based on physical coordinates, be:
[0211]
[0212]
[0213]
[0214] Define the unit direction vector of the long side of the container as:
[0215]
[0216] Define the unit direction vector of the short side of the container as the unit direction vector of the long side rotated counterclockwise by 90 degrees:
[0217]
[0218] The longitudinal deviation (along the length of the container) is:
[0219]
[0220] Lateral deviation (along the width of the container) is:
[0221]
[0222] Through the above projection calculations, the planar deviation vector originally in the global working coordinate system is... It is decomposed into two independent components along the length and width of the container itself, making its physical meaning clearer, and the control system can directly... and As a control target for longitudinal and lateral correction, no additional coordinate transformation is required.
[0223] At the same time, the physical attitude angle of the container is recalculated based on the physical coordinates. and the physical attitude angle of the lifting gear :
[0224]
[0225]
[0226] The rotation angle deviation is:
[0227]
[0228] To avoid sudden changes in angle, Normalization to :
[0229]
[0230] Depending on the specific needs, the output can be converted to an angle-based format.
[0231]
[0232] The relative deviation of the final output can be expressed as:
[0233]
[0234] In this embodiment of the invention, after obtaining the three-dimensional coordinates of the key points using method two, the three-dimensional attitude relationship between the spreader plane and the top surface of the container can be further calculated. For example, the included angle between the two planes can be calculated. This included angle reflects the pitch or roll attitude deviation between the spreader and the container. When the top surface of the container is not completely parallel to the spreader plane, the axial direction of the locking pin and the corner fitting hole is inconsistent, which may cause the locking pin to get stuck during insertion. By calculating the included angle between the two planes, the system can detect the attitude deviation in advance and send a compensation command to the control system to prevent the pin from failing due to tilting attitude.
[0235] Furthermore, the relative height between the two planes, such as the height deviation at each of the four corner points, can be calculated by fitting the container plane equation and the spreader plane equation, providing data support for the active tilt control of the spreader.
[0236] S6: Output the positional deviation of the spreader relative to the container to the control system.
[0237] To reduce the impact of detection jitter on the deviation results, combined with Figure 3 As shown, this invention uses a Kalman filter to filter the deviation results.
[0238] Define the state variables as:
[0239]
[0240] The state prediction equation is:
[0241]
[0242] The observation equation is:
[0243]
[0244] in, This is the state transition matrix; The observation matrix; This is process noise; To observe noise.
[0245] The Kalman filter predicts the current state based on the state change relationship between adjacent time steps, and updates the predicted state by combining the observation results of the current frame, so as to obtain a smooth and continuous deviation output result.
[0246] In practice, the system comprehensively evaluates the reliability of the current frame observation. Evaluation factors include keypoint detection confidence, geometric constraint verification results, and changes in deviation between adjacent frames. When the keypoint detection confidence is below a preset threshold, the geometric relationship does not meet the constraints, or the change in deviation between adjacent frames exceeds a preset abrupt change threshold, even if the corresponding keypoints have been removed or corrected in the preceding steps, the current frame observation is still considered a low-reliability observation. For low-reliability observations, their weight in the filtering update process can be reduced, or the observation update can be paused while only the state prediction result is retained to maintain the continuity and robustness of the output deviation. Once subsequent frame detection returns to normal, the observation update step is re-executed.
[0247] By adopting the above method, the influence of visual inspection noise, image jitter, short-term occlusion and local false detection on deviation measurement results can be effectively suppressed, and the stability, real-time performance and control applicability of deviation output can be improved during automatic or semi-automatic assisted box alignment of the lifting device.
[0248] The final output includes:
[0249]
[0250] The specific meanings of each output item are as follows:
[0251] : Longitudinal deviation of the spreader relative to the container;
[0252] Lateral deviation of the spreader relative to the container;
[0253] The height deviation of the spreader relative to the container;
[0254] : The deviation of the rotation angle of the spreader relative to the container;
[0255] The physical attitude angle of the container in the operating coordinate system;
[0256] The physical attitude angle of the lifting device in the working coordinate system;
[0257] Overall confidence level: ;
[0258] Detection status, including normal, low confidence, occlusion, abnormal, and requiring re-detection.
[0259] The output results are sent to the spreader's programmable logic controller (PLC) or automation control system via standard industrial communication protocols. The control system drives the spreader's lateral, longitudinal, lifting, and slewing mechanisms to perform closed-loop correction based on the received deviations until all deviations converge within the allowable error range. At this point, it indicates that the spreader's locking pins are precisely aligned with the container corner fitting holes, and the locking pin insertion and locking operations can be performed.
[0260] Below is an implementation case study based on a specific application scenario.
[0261] 1. Taking the quay crane operation scenario as an example, two sets of six high-definition industrial cameras are deployed at the sea-side and land-side trolley frames of the quay crane, using lenses with different focal lengths (8mm / 4mm / 2.8mm) to cover three distance levels: far, medium, and near.
[0262] Two long-range cameras, with a downward angle of approximately 35 degrees, cover the entire seaside working area (approximately 80m × 30m) for global scene monitoring and hatch / bay position identification. The long-range cameras use 8mm focal length lenses, providing sufficient angular resolution across the large field of view to distinguish container bay positions.
[0263] Two mid-range cameras are used, covering the area below the trolley's running track (approximately 40m × 20m), for container surface feature detection and keyhole recognition. The mid-range cameras employ 4mm focal length lenses, striking a balance between field of view and detail resolution, effectively capturing the visual features of the container's top corner fittings.
[0264] Two close-range cameras cover an area of approximately 5m x 5m below the spreader, used for precise monitoring of the spreader's attitude and close-range, accurate guidance during the spreader-container alignment phase. The close-range cameras employ 2.8mm focal length lenses, providing a wide field of view and high detail resolution at close range, clearly showing the relative positional relationship between the spreader's locking pins and the container's corner fitting holes.
[0265] The aforementioned multi-focal-length lens configuration enables the system to obtain the optimal image resolution for the current distance throughout the entire process of the spreader descending and approaching the container. The long-range camera provides global contextual information to assist in coarse positioning, the mid-range camera achieves target detection and tracking at intermediate distances, and the close-range camera provides high-precision local visual information in the final alignment stage. The three-level camera work together to achieve a seamless transition from long-range coarse positioning to close-range precise guidance.
[0266] 2. Calibrate the camera's internal and external parameters.
[0267] 3. Obtain keypoint depth using matching or deep networks.
[0268] 4. The key point regression model outputs image coordinates.
[0269] 5. Recover the 3D coordinates of key points based on depth and camera intrinsic parameters.
[0270] 6. Fit the top plane of the container using the four three-dimensional corner points of the container.
[0271] 7. Fit the plane of the lifting device using the three-dimensional points of the four locking pins of the lifting device.
[0272] 8. Calculate the height difference, the angle between the normal vectors, and the center deviation between the top plane of the container and the plane of the spreader.
[0273] 9. Output three-dimensional relative pose.
[0274] The three-dimensional key set is:
[0275]
[0276]
[0277] The container plane normal vector can be obtained from:
[0278]
[0279] The normal vector of the lifting device plane is:
[0280]
[0281] The angle between the two planes is:
[0282]
[0283] This invention also proposes a container and spreader pose estimation system based on keypoint regression, used to execute the aforementioned keypoint regression-based container and spreader pose estimation method, such as... Figure 4 As shown, it includes:
[0284] An image acquisition device is installed in the container and spreader operation area to acquire images of the containers and spreaders.
[0285] The keypoint regression module is used to input container images and spreader images into a pre-trained keypoint regression model to obtain container keypoints and spreader keypoints.
[0286] The pose estimation module is used to calculate container pose data based on container key points and spreader pose data based on spreader key points; wherein, the container pose data includes the container center point and container attitude angle; the spreader pose data includes the spreader center point and spreader attitude angle.
[0287] The deviation calculation module is used to convert image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle.
[0288] The output module is used to output the positional deviation of the spreader relative to the container to the control system.
[0289] In summary, the present invention has the following advantages compared with the prior art:
[0290] First, it offers high detection accuracy. This invention directly regresses key points with clear geometric meaning, such as spreader pins and container corner fittings. It can describe the structural position and orientation of the target more accurately than ordinary target detection boxes. The regression accuracy of key point coordinates directly affects the attitude estimation accuracy. Compared with rectangular boxes, which only provide a rough position and scale of the target, key points provide rich structural information for precise pose calculation.
[0291] Second, it can calculate fine-grained deviations. This invention can output deviations in four dimensions: lateral deviation, longitudinal deviation, height deviation, and rotation angle deviation. Moreover, the deviations have real physical scale rather than relative values at the image pixel level, which fully meets the requirements of the lifting device for the box control system. The control loop can directly use these deviations as closed-loop feedback without additional coordinate transformation or unit conversion.
[0292] Third, it adapts to complex environments. Based on a deep learning keypoint model, this invention uses diverse training data to cover complex conditions such as different lighting, box colors, dirt, partial occlusion, and background interference. This enables the model to learn feature representations that are robust to these changes, resulting in stronger environmental adaptability and generalization performance compared to traditional image recognition methods that rely on manually designed features.
[0293] Fourth, it facilitates control system integration. This invention outputs a standardized deviation format, which can be directly transmitted to PLCs, automation controllers, driver assistance systems, or remote control systems via industrial communication protocols. The interface is clear and unambiguous, requiring no additional data parsing or conversion, and is plug-and-play compatible with the control system architecture of various port loading and unloading equipment.
[0294] Fifth, it has strong scalability. This invention is compatible with different hardware configurations such as single-camera, dual-camera, and multi-camera fusion. Hardware can be flexibly selected according to actual operation scenarios and accuracy requirements, and it can be deployed on different devices without making significant adjustments to the core architecture of the algorithm.
[0295] Sixth, it exhibits good robustness. This invention employs a multi-level quality assurance mechanism, including confidence level judgment, side length constraints, parallel constraints, and temporal continuity constraints, to rigorously screen key points. Combined with Kalman filtering, it smooths the deviation output, ensuring reliable and effective deviation estimates even under challenging conditions such as local occlusion, motion blur, and sudden changes in illumination. This significantly reduces false detection rates and output jitter, guaranteeing the safety and continuity of automated hoisting operations.
[0296] It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.
Claims
1. A method for estimating the pose of containers and spreaders based on keypoint regression, characterized in that, include: S1, acquire images of the container and the spreader; S2, input the container image and the spreader image into the pre-trained keypoint regression model to obtain the container keypoints and the spreader keypoints; S3, calculate container pose data based on container key points; the container pose data includes the container center point and container attitude angle; S4, Calculate the spreader's position and posture data based on the key points of the spreader; the spreader's position and posture data includes the center point of the spreader and the spreader's posture angle; S5, transform the image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle; S6 outputs the positional deviation of the spreader relative to the container to the control system.
2. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, The method further includes a step of geometrically constraining the container pose data, including: Side length constraints: Calculate the length ratio of adjacent sides of the container based on key points, and eliminate key point combinations whose length ratios do not conform to the container's length-to-width ratio; and / or, Parallel constraint: Calculate the direction vector of the opposite side of the container based on the key points of the container, and eliminate key point combinations whose direction vectors do not conform to the parallel relationship.
3. The container and spreader pose estimation method based on keypoint regression according to claim 1 or 2, characterized in that, The method further includes a step of applying temporal continuity constraints to the container pose data, including: If the coordinate difference of the same key point in adjacent frames is greater than a preset value, the coordinates of the key point are corrected or removed using historical data.
4. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, The pre-trained keypoint regression model outputs keypoints through a heatmap: ;in, For key point locations, The standard deviation of the Gaussian kernel. Represents pixels Belongs to the Thermal response values at key points; The training loss uses mean squared error: ;in, Predict heatmaps for the model; Key points The position is obtained through the position of the maximum response: .
5. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, S5 specifically includes: by Representing image coordinates, in Represents the planar coordinates of container operations; If satisfied or ,get After normalization, we get , ; For the center point of the container and the center point of the lifting gear Calculate the plane deviation as , ; in, It is a homography matrix. is the scale factor.
6. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, S5 specifically includes: based on and Transform the image point coordinates to the camera coordinate system; where... The coordinates of the image points, Depth in the camera coordinate system The pixel measure of the camera's focal length on the image plane. These are the pixel coordinates of the intersection of the optical axis and the imaging plane. Calculate the three-dimensional deviation between the spreader center point and the container center point: ;in The three-dimensional coordinates of the centerline point of the spreader. The coordinates of the container's center point are given.
7. The container and spreader pose estimation method based on keypoint regression according to claim 5 or 6, characterized in that, S5 also includes: Define the unit direction vector of the long side of the container. and the unit direction vector of the short side of the container ;in, Let be the long-side direction vector of the container in the operating coordinate system; Calculate the longitudinal deviation of the plane Lateral deviation of the plane ;in, , and For planar deviation; Output relative deviation ;in, This refers to the height deviation between the spreader and the container. This represents the rotational angle deviation of the spreader relative to the container in the operating coordinate system.
8. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, The method described prior to S6 also includes filtering and outlier removal steps, including: The position deviation and the angle deviation are filtered using a Kalman filter; When the prediction bias exceeds a preset threshold relative to the filtering result of the previous frame, the bias observation value of the current frame is marked as a low confidence observation. During the Kalman filter update process, reduce the weight of the low-confidence observations, or pause the observation update and retain only the state prediction results.
9. The container and spreader pose estimation method based on keypoint regression according to claim 1, characterized in that, When the keypoint regression model outputs container keypoints and spreader keypoints, it also outputs the confidence level of each keypoint; the threshold of the confidence level is dynamically adjusted based on illumination, motion blur, and historical stability.
10. A container and spreader pose estimation system based on keypoint regression, characterized in that, include: An image acquisition device is installed in the container and spreader operation area to acquire images of the container and spreader. The keypoint regression module is used to input container images and spreader images into a pre-trained keypoint regression model to obtain container keypoints and spreader keypoints. The pose estimation module is used to calculate container pose data based on container key points and spreader pose data based on spreader key points; wherein, the container pose data includes the container center point and container attitude angle; the spreader pose data includes the spreader center point and spreader attitude angle. The deviation calculation module is used to transform the image coordinates to the working coordinate system, calculate the positional deviation between the container center point and the spreader center point, and calculate the angular deviation between the container attitude angle and the spreader attitude angle. The output module is used to output the positional deviation of the spreader relative to the container to the control system.
