Port container loading and unloading positioning system and method based on visual identification

Abstract

The invention discloses a port container loading and unloading positioning system and method based on visual identification, and relates to the technical field of port logistics automation. The method comprises the following steps: acquiring a container surface image through an image acquisition device on a sling, and detecting an initial angular point through a deep learning model; when the angular points are missing or the confidence coefficient is insufficient, edge features are extracted, and sub-pixel coordinates are predicted through a reconstruction network so as to complement the angular points; sorting the complete angular point set to form closed-loop vectors, and calculating cross products of adjacent vectors to verify the topological effectiveness; and when the topology is effective, through coordinate conversion and pose optimization, outputting three-dimensional coordinates of the angular points in a world coordinate system, and further determining the position of the container and feeding back the position to a lifting appliance control system to guide loading and unloading. The system correspondingly comprises an image acquisition and corner detection module, a corner completion module, a corner sorting and cross product calculation module, a topology verification and coordinate conversion module and a positioning feedback module. The problem of inaccurate positioning caused by lifting tool shaking, stacking deviation and the like is solved, and the precision and efficiency of container loading and unloading are improved.

Description

Technical Field

[0001] This invention relates to the field of technology, specifically to a port container loading and unloading positioning system and method based on visual recognition. Background Technology

[0002] Precise positioning during container loading and unloading at ports is crucial for improving operational efficiency. Traditional positioning methods have significant limitations: laser ranging is susceptible to interference from port dust, rain, and fog, leading to decreased accuracy; GPS suffers from severe signal blockage in densely packed yards, with positioning accuracy only at the meter level, failing to meet millimeter-level alignment requirements; manual operation is inefficient, unsafe, and unsuitable for continuous operations. While existing visual recognition technologies are being gradually applied, they still have shortcomings: firstly, changes in lighting and occlusion can easily lead to missing or false detections of container corner points; secondly, insufficient verification of corner point topology relationships, with disordered sorting directly causing coordinate transformation failures; and thirdly, limited coordinate transformation accuracy, with accumulated rotation and translation errors making it difficult to address issues such as spreader vibration and container stacking tilt. Therefore, a visual positioning solution is needed that can adapt to complex environments, possess corner point completion and topology verification capabilities, and achieve high-precision coordinate transformation to solve the positioning challenges of container loading and unloading. Summary of the Invention

[0003] The purpose of this invention is to provide a port container loading and unloading positioning system and method based on visual recognition, so as to solve the problems raised in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a port container loading and unloading positioning system and method based on visual recognition, the method comprising:

[0005] Step S1: Acquire images of the container surface using an image acquisition device deployed on a port spreader, input them into a pre-trained deep learning corner detection model, and output an initial set of corner points.

[0006] Step S2: When initial corner point missing or confidence level below preset threshold is detected, extract container edge features, predict sub-pixel coordinates of missing corner points through reconstruction network, and update corner point set;

[0007] Step S3: Sort the complete set of corner points clockwise to form a closed-loop vector sequence, and calculate the cross product of adjacent vectors;

[0008] Step S4: When all cross product values ​​have the same sign and the absolute value is greater than the threshold, the topology is determined to be valid. After coordinate transformation and pose optimization, the three-dimensional coordinates of the container corner points in the world coordinate system are output.

[0009] Step S5: Determine the position of the container in the world coordinate system based on the three-dimensional coordinates described in step S4, and feed back the position information to guide the spreader to complete the alignment and loading / unloading.

[0010] Furthermore, step S1 includes:

[0011] Step S1-1: Perform preprocessing operations on the acquired container surface image, sequentially enhance image contrast through contrast-limited adaptive histogram equalization, suppress image noise and preserve corner edge sharpness with a median filter of fixed radius, where edge sharpness refers to the steepness of gray-scale changes at the corners of the container in the image.

[0012] Step S1-2: Input the preprocessed image into the pre-trained deep learning corner detection model, extract multi-scale feature maps through the feature pyramid network, and output the probability distribution of each pixel as a corner point through corner heatmap regression;

[0013] Step S1-3: Perform non-maximum suppression processing on the corner heatmap, and select the four initial corner coordinates with the highest corner confidence to form an initial corner set. The corner confidence refers to the probability value that a certain pixel is a corner of a container.

[0014] Furthermore, step S2 includes:

[0015] Step S2-1: When the number of valid corner points in the initial corner point set is less than 4 or the confidence of a corner point is less than the threshold δ, it is determined that reconstruction is required;

[0016] Step S2-2: Calculate the Sobel gradient magnitude map of the input image. The origin is defined by the top left corner of the image, with the x-direction extending horizontally to the right and the y-direction extending vertically downwards. x I y These are the gradients in the x and y directions, respectively;

[0017] Step S2-3: Generate the edge mask M using adaptive threshold binarization; threshold T = μ G +k×σ G , where μ G Let σ be the gradient mean. G Let be the standard deviation, and k be the threshold adjustment coefficient, where k ∈ [1.5, 2.5].

[0018] Step S2-4: Input the edge mask M into the reconstruction network to predict missing corner points and generate a sub-pixel coordinate probability distribution map. The sub-pixel coordinate probability distribution map refers to probability distribution data presented in the form of a two-dimensional matrix.

[0019] Step S2-5: Analyze the probability distribution map and update the corner point set by taking the coordinates of the candidate point with the highest confidence.

[0020] Furthermore, step S3 includes:

[0021] Step S3-1: Calculate the geometric center point O based on the updated corner point set A = {q1,q2,q3,q4} from step S2-5, where q1, q2, q3, and q4 are the four corner points in the updated corner point set; Where q i,x q i,y These represent the coordinates of the i-th corner point in the x and y directions of the image coordinate system, respectively.

[0022] Step S3-2: Establish a polar coordinate system with O as the origin, and calculate the polar angle θ at each corner point. i =arctan2(q i,y -O y ,q i,x -O x ), where the polar angle θ i This represents the azimuth angle of the i-th corner point relative to the center point O, measured counterclockwise starting from the positive x-axis direction;

[0023] Step S3-3: Press θ i Sort the corner points from smallest to largest to generate a clockwise sequence Q = {q} (1) ,q (2) ,q (3) ,q (4)}, where q (1) This represents the first corner point in the sorted sequence, with its polar angle θ. (1) It is the smallest among the four corner points;

[0024] Step S3-4: Construct a closed-loop vector based on the clockwise corner point sequence, V1 = q (2) -q (1) V2 = q (3) -q (2) V3 = q (4) -q (3) V4 = q (1) -q (4) , where V1, V2, V3, and V4 represent the vectors formed by adjacent corner points;

[0025] S3-5: Calculate the cross product of adjacent vectors based on the closed-loop vector, C1 = V1 × V2, C2 = V2 × V3, C3 = V3 × V4, C4 = V4 × V1, where C1, C2, C3, and C4 represent the cross product values ​​of two adjacent vectors, respectively.

[0026] Furthermore, step S4 includes:

[0027] Step S4-1: Verify the sign of the cross product. When any i,j∈{1,2,3,4} and Sign(C i ) = Sign(C j When ), Sign Uniform=1, otherwise Sign Uniform =0; where Sign Uniform The cross product sign consistency index, Sign(C) i ) represents the sign of the i-th cross product value;

[0028] Step S4-2: Verify the absolute value threshold of the cross product when i∈{1,2,3,4} and |C i When |>a, Thresh Pass =1, otherwise Thresh Pass =0; where Thresh Pass is the cross product absolute value compliance indicator, and 'a' is a preset threshold with a value range of [10, 50] pixel area units;

[0029] Step S4-3: When Sign Uniform =0 or Thresh Pass When = 0, the topology is determined to be invalid, corner re-detection is triggered, and the process returns to step S1; when Sign .... Uniform =1 and Thresh Pass When = 1, the topology is determined to be valid, and step S4-4 is executed.

[0030] Step S4-4: Define the preset 3D world coordinate sequence corresponding to the sorted corner points as P = {p1, p2, p3, p4}, where p i =(l i ,w i ,h i The coordinates of the corner point are preset. The world coordinate system has a fixed point on the port ground as the origin, the length extension direction of the container is the l-axis, the width extension direction is the w-axis, and the vertical upward direction is the h-axis. i w i h i These correspond to the coordinates in the length, width, and height directions, respectively.

[0031] Step S4-5: Call the solvePnP algorithm to calculate the camera extrinsic parameters, using the sorted image corner points q = {q (1) ,q (2) ,q (3) ,q (4) As a 2D point set based on the image coordinate system, P = {p1, p2, p3, p4} is a 3D point set based on the world coordinate system. The input is the camera intrinsic parameter matrix K, and the output is the rotation vector r and the translation vector t. The camera intrinsic parameter matrix K is an intrinsic parameter matrix that describes the optical characteristics of the camera and the imaging geometry. The rotation vector r is a vector that describes the rotation state of the camera, and the translation vector t is a vector that describes the translation state of the camera.

[0032] Step S4-6: Optimize the rotation matrix using the Kabsch algorithm, and construct the covariance matrix based on the correspondence between image corner points and 3D feature points. Decomposition of H=UΣV by SVD T Calculate the optimal rotation matrix R = VU T ;in This represents the average coordinates of the image corner points. Let U be the average value of the coordinates of the three-dimensional feature points, and let U, ∑, and V be the left singular matrix, singular value matrix, and right singular matrix after the covariance matrix is ​​decomposed by SVD, respectively. The optimal rotation matrix R is the orthogonal matrix that minimizes the mean square error between the coordinates of the image corner points and the three-dimensional world coordinates.

[0033] Step S4-7: Fuse the optimized rotation matrix R and translation vector t to calculate the 3D coordinates of the image corner points in the world coordinate system: Where x ω y ω z ω These are the x, y, and z coordinates in the world coordinate system, respectively. p y p These are the x and y coordinates in the image coordinate system, respectively.

[0034] The system includes: an image acquisition and corner detection module, a corner completion module, a corner sorting and cross product calculation module, a topology verification and coordinate transformation module, and a positioning feedback module;

[0035] The image acquisition and corner detection module is used to input the container surface image into a pre-trained deep learning corner detection model and output an initial set of corner points; including:

[0036] The preprocessing unit is used to sequentially perform contrast-limited adaptive histogram equalization and fixed-radius median filtering on the container surface image to enhance contrast, suppress noise, and preserve corner edge sharpness.

[0037] The deep learning detection unit is used to extract multi-scale feature maps through a feature pyramid network and output the probability distribution of each pixel as a corner point through corner heatmap regression.

[0038] The corner point filtering unit is used to perform non-maximum suppression on the corner point heatmap and filter the four corner points with the highest confidence to form an initial corner point set, wherein the confidence is the probability value that the pixel is a corner point of the container;

[0039] The corner completion module is used to extract container edge features when initial corners are missing or the confidence level is below a preset threshold. It then predicts the sub-pixel coordinates of the missing corners using a reconstruction network and updates the corner set. This includes:

[0040] The reconstruction trigger unit is used to determine that corner reconstruction needs to be started when the number of valid corner points in the initial corner point set is less than 4 or the confidence of a corner point is less than the threshold δ.

[0041] The edge feature extraction unit is used to calculate the Sobel gradient magnitude map of the input image. And through the adaptive threshold T = μ G +l×σ G Generate an edge mask M;

[0042] The subpixel prediction unit is used to input the edge mask into the reconstruction network, generate a subpixel coordinate probability distribution map, and parse out the candidate point with the highest confidence, and update the corner point set.

[0043] The corner sorting and cross product calculation module is used to sort the complete set of corner points clockwise to form a closed-loop vector sequence and calculate the cross product of adjacent vectors; it includes:

[0044] The center point calculation unit is used to calculate the coordinates of the geometric center point O based on the updated corner point set A = {q1, q2, q3, q4}.

[0045] The polar angle sorting unit is used to establish a polar coordinate system with O as the origin and calculate the polar angle θ of each corner point. i =arctan2(q i,y -O y ,q i,x -O x ), and according to θ i Sort the corner points from smallest to largest to generate a clockwise sequence Q = {q} (1) ,q (2) ,q (3) ,q (4)};

[0046] The vector and cross product calculation unit is used to construct closed-loop vectors, V1 = q (2) -q (1) V2 = q (3) -q (2) V3 = q (4) -q (3) V4 = q (1) -q (4) And calculate the cross product of adjacent vectors: C1 = V1 × V2, C2 = V2 × V3, C3 = V3 × V4, C4 = V4 × V1.

[0047] The topology verification and coordinate transformation module determines the topology is valid when all cross product values ​​have the same sign and their absolute values ​​are greater than a threshold. After coordinate transformation and pose optimization, it outputs the 3D coordinates of the container corner points in the world coordinate system. This includes:

[0048] Topological verification unit, used to verify the same sign of the cross product to obtain Sign. Uniform Verify the threshold of the absolute value of the cross product to obtain the Thresh. Pass When Sign Uniform =0 or Thresh Pass When = 0, corner re-detection is triggered and the image acquisition and corner detection module is returned; when Sign = 0, corner re-detection is triggered. Uniform =1 and Thresh Pass When = 1, the topology is deemed valid, and subsequent coordinate transformations are performed;

[0049] The world coordinate definition unit is used to define the preset 3D world coordinate sequence corresponding to the sorted corner points as P = {p1, p2, p3, p4}, where p i =(l i ,w i ,h i The world coordinate system is a preset corner point world coordinate system with a fixed point on the port ground as the origin, the length extension direction of the container as the l axis, the width extension direction as the w axis, and the vertical upward direction from the ground as the h axis.

[0050] The extrinsic parameter calculation unit is used to call the solvePnP algorithm to calculate the camera extrinsic parameters, based on the sorted image corner points Q = {q (1) ,q (2) ,q (3) ,q (4) As a 2D point set based on the image coordinate system, P = {p1, p2, p3, p4} is a 3D point set based on the world coordinate system. The input is the camera intrinsic parameter matrix K, and the output is the rotation vector r and the translation vector t.

[0051] The rotation matrix optimization unit uses the Kabsch algorithm to optimize the rotation matrix, constructing the covariance matrix based on the correspondence between image corner points and 3D feature points. Decomposition of H=U∑V using SVD T Calculate the optimal rotation matrix R = VU T ;

[0052] The world coordinate calculation unit is used to fuse the optimized rotation matrix R and translation vector t to calculate the three-dimensional coordinates of the image corner points in the world coordinate system.

[0053] The positioning feedback module is used to determine the position of the container in the world coordinate system based on the three-dimensional coordinates, and to feed back this position information to guide the spreader to complete the alignment and loading / unloading; it includes:

[0054] The geometric center calculation unit is used to calculate the position of the container's geometric center in the world coordinate system based on the three-dimensional coordinates of its corner points;

[0055] The spreader control interface unit is used to feed back the geometric center position information to the spreader control system to guide the spreader to adjust its posture to complete the alignment and loading / unloading.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. Traditional positioning methods, such as manual operation, rely on experience and judgment, resulting in low efficiency. Laser and GPS positioning are subject to environmental interference and require repeated calibration, causing delays in the loading and unloading process. In contrast, this solution, with its rapid image acquisition device and efficient deep learning corner detection model, can accurately determine the container's position in a short time, significantly reducing positioning time and improving loading and unloading efficiency.

[0058] 2. Previous positioning technologies lacked accuracy and could not meet millimeter-level alignment requirements, affecting loading and unloading precision. This visual positioning solution incorporates corner completion, topology verification, and high-precision coordinate transformation technologies, effectively avoiding corner detection errors and reducing deviations in rotation and translation vector calculations, successfully advancing positioning accuracy to the millimeter level.

[0059] 3. The port environment is harsh, with frequent dust, rain, and fog. Traditional laser ranging is easily interfered with, and GPS signals are easily blocked, resulting in a significant reduction in positioning accuracy. This vision solution is equipped with image enhancement, defogging and deraining algorithms, and a robust deep learning model, enabling stable operation under complex conditions such as nighttime and foggy weather, ensuring efficient loading and unloading operations at the port around the clock. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the structure of a port container loading and unloading positioning system based on vision recognition according to the present invention;

[0061] Figure 2 This is a schematic diagram of the topology verification and coordinate transformation module of a port container loading and unloading positioning system based on vision recognition according to the present invention.

[0062] Figure 3 This is a flowchart illustrating a port container loading and unloading positioning method based on visual recognition according to the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example: Figure 1 Figure 2As shown, the present invention provides a technical solution: a port container loading and unloading positioning system and method based on visual recognition.

[0065] Scenario setting: 40-foot container loading and unloading operations are being carried out at berth No. 5 of a port. An industrial camera with a resolution of 1920×1080 is deployed on the spreader, and the intrinsic parameter matrix... The goal is to achieve a loading and unloading operation with a positioning error of less than or equal to ±5mm through the process steps described in claim 1.

[0066] The method includes: Step S1: Acquire images of the container surface through an image acquisition device deployed on a port spreader, input them into a pre-trained deep learning corner detection model, and output an initial set of corner points;

[0067] Step S2: When initial corner point missing or confidence level below preset threshold is detected, extract container edge features, predict sub-pixel coordinates of missing corner points through reconstruction network, and update corner point set;

[0068] Step S3: Sort the complete set of corner points clockwise to form a closed-loop vector sequence, and calculate the cross product of adjacent vectors;

[0069] Step S4: When all cross product values ​​have the same sign and the absolute value is greater than the threshold, the topology is determined to be valid. After coordinate transformation and pose optimization, the three-dimensional coordinates of the container corner points in the world coordinate system are output.

[0070] Step S5: Determine the position of the container in the world coordinate system based on the three-dimensional coordinates described in step S4, and feed back the position information to guide the spreader to complete the alignment and loading / unloading.

[0071] Furthermore, step S1 includes:

[0072] Step S1-1: Perform the following steps on the container surface image in sequence: “Contrast-limited adaptive histogram equalization” to enhance contrast and “fixed radius 3-pixel median filtering” to suppress noise. After processing, the sharpness of corner edges is improved by 40%, and the gray-level gradient abruptly increases from 20 to 28.

[0073] Step S1-2: Input the pre-trained deep learning corner detection model, extract 4-layer scale feature maps through the feature pyramid network, and output the probability distribution of pixels as corners through corner heatmap regression. The probability value of the pixel corresponding to the upper right corner reaches 0.92. Corner heatmap regression refers to the feature pyramid network and heatmap regression. The feature pyramid network extracts feature maps of different resolutions from the input image through multi-scale feature fusion. Heatmap regression is based on the feature map output by the feature pyramid network. The corner heatmap is predicted by the convolutional neural network. The value of each pixel in the heatmap represents the probability that the position is a corner of the container.

[0074] Step S1-3: Perform non-maximum suppression to select the top 4 corner points with the highest confidence: q1=(320,210), q2=(1580,205), q3=(1575,890), q4=(315,895), with a confidence level greater than or equal to 0.85 and a threshold δ=0.7.

[0075] Furthermore, step S2 includes:

[0076] Step S2-1: The confidence level of q4 is detected to have dropped to 0.62, which is less than the threshold δ = 0.7, thus meeting the reconstruction conditions. Initiate completion.

[0077] Steps S2-2 to S2-3: Calculate the Sobel gradient magnitude map Obtain the gradient mean μ G =35, standard deviation σ G =12; take k=2, threshold T=μ G +k×σ G =35 + 2 × 12 = 59, binarize to generate edge mask M;

[0078] Steps S2-4 to S2-5: Input the mask M into the reconstruction network and output the sub-pixel coordinate probability distribution map, where the probability value at coordinates (314.3, 894.7) is the largest at 0.91. Update q4 to this coordinate, and the integrity of the corner point set is restored to 100%.

[0079] Furthermore, step S3 includes:

[0080] Step S3-1: Calculate the geometric center O according to the formula in claim 4:

[0081]

[0082] Steps S3-2 to S3-3: Establish a polar coordinate system with O as the origin, and follow the formula θ i =arctan2(q i,y -O y ,q i,x -O x Calculate the polar angles: θ1 = 158°, θ2 = 22°, θ3 = 38°, θ4 = 105°, according to θ i Sorting from smallest to largest, the clockwise sequence is Q = {q2, q3, q4, q1};

[0083] Steps S3-4 to S3-5: Construct closed-loop vectors: V1 = q3 - q2 = (-5, 685), V2 = q4 - q3 = (-1260.7, 4.7), V3 = q1 - q4 = (5.7, -684.7), V4 = q1 - q2 = (1260, -5);

[0084] Calculate the cross product: C1 = V1 × V2 = 863575.5, C2 = V2 × V3 = 863420.2, C3 = V3 × V4 = 862722.5, C4 = V4 × V1 = 8630975.

[0085] Furthermore, step S4 includes:

[0086] Steps S4-1 to S4-3: All C i All are positive values, Sign Uniform =1; Verify absolute value threshold: all |C i |>50, where threshold a = 50, Thresh Pass =1, indicating the topology is valid;

[0087] Step S4-4: Preset the world coordinates of the corner point of the 40-foot container to P = [(0,0,0),(12192,0,0),(12192,2438,0),(0,2438,0)], in mm. The world coordinate system takes the fixed point on the port ground as the origin and the length direction as the l-axis.

[0088] Step S4-5: Input the sorted image corner points Q, world coordinates P, and camera intrinsic parameters K, and output the rotation vector r = (0.02, -0.01, 0.03) and translation vector t = (5000, 3000, 8000), in mm;

[0089] Step S4-6: Construct the covariance matrix using the Kabsch algorithm. SVD decomposition H=U∑V T The optimal rotation matrix is ​​obtained as R = VU T The rotational error decreased from 0.5° to 0.12°;

[0090] Step S4-7: According to the formula The calculated world coordinates of all corner points are q2=(5.2,3002.1,8001.3), q3=(12200.3,3001.8,8000.9), q4=(12198.5,5440.2,8000.1), and q1=(2.1,5439.7,8000.7); the error of all coordinates is less than or equal to ±3mm.

[0091] Furthermore, step S5 includes:

[0092] Step S5-1: Based on the three-dimensional coordinates of the corner points output in step S4-7, calculate the geometric center of the container as (6101.5, 4220.9, 8000.5), in mm.

[0093] Step S5-2: Feed back the center position to the spreader control system, adjust the spreader by 1.5mm along the x-axis of the world coordinate system, 0.9mm along the y-axis, and rotate it by 0.1° around the z-axis of the world coordinate system. The final loading and unloading error is 4mm, which meets the positioning requirements.

[0094] The system includes: an image acquisition and corner detection module, a corner completion module, a corner sorting and cross product calculation module, a topology verification and coordinate transformation module, and a positioning feedback module;

[0095] The image acquisition and corner detection module is used to input the container surface image into a pre-trained deep learning corner detection model and output an initial set of corner points; including:

[0096] The preprocessing unit is used to sequentially perform contrast-limited adaptive histogram equalization and fixed-radius median filtering on the container surface image to enhance contrast, suppress noise, and preserve corner edge sharpness.

[0097] The deep learning detection unit is used to extract multi-scale feature maps through a feature pyramid network and output the probability distribution of each pixel as a corner point through corner heatmap regression.

[0098] The corner point filtering unit is used to perform non-maximum suppression on the corner point heatmap and filter the four corner points with the highest confidence to form an initial corner point set, wherein the confidence is the probability value that the pixel is a corner point of the container;

[0099] The corner completion module is used to extract container edge features when initial corners are missing or the confidence level is below a preset threshold. It then predicts the sub-pixel coordinates of the missing corners using a reconstruction network and updates the corner set. This includes:

[0100] The reconstruction trigger unit is used to determine that corner reconstruction needs to be started when the number of valid corner points in the initial corner point set is less than 4 or the confidence of a corner point is less than the threshold δ.

[0101] The edge feature extraction unit is used to calculate the Sobel gradient magnitude map of the input image. And through the adaptive threshold T = μ G +k×σ G Generate an edge mask M;

[0102] The subpixel prediction unit is used to input the edge mask into the reconstruction network, generate a subpixel coordinate probability distribution map, and parse out the candidate point with the highest confidence, and update the corner point set.

[0103] The corner sorting and cross product calculation module is used to sort the complete set of corner points clockwise to form a closed-loop vector sequence and calculate the cross product of adjacent vectors; it includes:

[0104] The center point calculation unit is used to calculate the coordinates of the geometric center point O based on the updated corner point set A = {q1, q2, q3, q4}.

[0105] The polar angle sorting unit is used to establish a polar coordinate system with O as the origin and calculate the polar angle θ of each corner point. i =arctan2(q i,y -O y ,q i,x -O x ), and according to θ i Sort the corner points from smallest to largest to generate a clockwise sequence Q = {q} (1) ,q (2) ,q (3) ,q (4)};

[0106] The vector and cross product calculation unit is used to construct closed-loop vectors, V1 = q (2) -q (1) V2 = q (3) -q (2) V3 = q (4) -q (3) V4 = q (1) -q (4) And calculate the cross product of adjacent vectors: C1 = V1 × V2, C2 = V2 × V3, C3 = V3 × V4, C4 = V4 × V1.

[0107] The topology verification and coordinate transformation module determines the topology is valid when all cross product values ​​have the same sign and their absolute values ​​are greater than a threshold. After coordinate transformation and pose optimization, it outputs the 3D coordinates of the container corner points in the world coordinate system. This includes:

[0108] Topological verification unit, used to verify the same sign of the cross product to obtain Sign. Uniform Verify the threshold of the absolute value of the cross product to obtain the Thresh. Pass When Sign Uniform =0 or Thresh Pass When = 0, corner re-detection is triggered and the image acquisition and corner detection module is returned; when Sign = 0, corner re-detection is triggered. Uniform =1 and Thresh Pass When = 1, the topology is deemed valid, and subsequent coordinate transformations are performed;

[0109] The world coordinate definition unit is used to define the preset 3D world coordinate sequence corresponding to the sorted corner points as P = {p1, p2, p3, p4}, where p i =(l i ,w i ,h iThe world coordinate system is a preset corner point world coordinate system with a fixed point on the port ground as the origin, the length extension direction of the container as the l axis, the width extension direction as the w axis, and the vertical upward direction from the ground as the h axis.

[0110] The extrinsic parameter calculation unit is used to call the solvePnP algorithm to calculate the camera extrinsic parameters, based on the sorted image corner points Q = {q (1) ,q (2) ,q (3) ,q (4) As a 2D point set based on the image coordinate system, P = {p1, p2, p3, p4} is a 3D point set based on the world coordinate system. The input is the camera intrinsic parameter matrix K, and the output is the rotation vector r and the translation vector t.

[0111] The rotation matrix optimization unit uses the Kabsch algorithm to optimize the rotation matrix, constructing the covariance matrix based on the correspondence between image corner points and 3D feature points. Decomposition of H=U∑V using SVD T Calculate the optimal rotation matrix R = VU T ;

[0112] The world coordinate calculation unit is used to fuse the optimized rotation matrix R and translation vector t to calculate the three-dimensional coordinates of the image corner points in the world coordinate system.

[0113] The positioning feedback module is used to determine the position of the container in the world coordinate system based on the three-dimensional coordinates, and to feed back this position information to guide the spreader to complete the alignment and loading / unloading; it includes:

[0114] The geometric center calculation unit is used to calculate the position of the container's geometric center in the world coordinate system based on the three-dimensional coordinates of its corner points;

[0115] The spreader control interface unit is used to feed back the geometric center position information to the spreader control system to guide the spreader to adjust its posture to complete the alignment and loading / unloading.

Claims

1. A visual recognition based positioning method for port container handling, characterized in that: The method comprises the following steps: Step S1: obtaining a container surface image through an image acquisition device arranged on a port spreader, inputting a pre-trained deep learning corner point detection model, and outputting an initial corner point set; Step S2: when the initial corner point is detected to be missing or the confidence is lower than a preset threshold, extracting a container edge feature, predicting a sub-pixel coordinate of the missing corner point through a reconstruction network, and updating the corner point set; Step S3: sequentially arranging the complete corner point set in a clockwise direction to form a closed loop vector sequence, and calculating a cross product value of adjacent vectors; Step S4: when all the cross product values are of the same sign and the absolute value is greater than a threshold, determining that the topology is valid, performing coordinate conversion and pose optimization, and outputting three-dimensional coordinates of the container corner points in a world coordinate system; Step S5: determining the position of the container in the world coordinate system according to the three-dimensional coordinates in Step S4, and feeding back the position information to guide the spreader to complete the positioning loading and unloading. The step S1 comprises:

2. The port container loading and unloading positioning method based on visual recognition according to claim 1, characterized in that: Step S1-1: performing a pretreatment operation on the obtained container surface image, sequentially passing through a limited contrast adaptive histogram equalization to enhance the image contrast, and a fixed radius median filter to suppress image noise and retain the edge sharpness of the corner point, wherein the edge sharpness refers to the steepness of the gray scale change at the container corner point in the image; Step S1-2: inputting the pretreated image into the pre-trained deep learning corner point detection model, extracting a multi-scale feature map through a feature pyramid network, and outputting a probability distribution of each pixel point being a corner point through corner heat map regression; Step S1-3: performing a non-maximum suppression processing on the corner heat map, screening out four initial corner point coordinates with the highest corner point confidence to form an initial corner point set, and the corner point confidence refers to the probability value of a certain pixel point being a container corner point. The step S2 comprises:

3. The port container loading and unloading positioning method based on visual recognition according to claim 1, characterized in that: Step S2-1: when the number of valid corner points in the initial corner point set is less than 4 or there is a corner point confidence less than a threshold δ, determining that reconstruction is needed; Step S2-4: inputting the edge mask M into the reconstruction network, predicting the missing corner point and generating a sub-pixel coordinate probability distribution map, and the sub-pixel coordinate probability distribution map refers to the probability distribution data in the form of a two-dimensional matrix; Step S2-2: calculating the Sobel gradient amplitude map of the input image wherein the upper left corner of the image is the origin, the horizontal right is the x direction, and the vertical down is the y direction, I x , I y are the gradients in the x and y directions, respectively; Step S2-3: generating an edge mask M by adaptive threshold binarization; threshold T = μ G + k x σ G wherein μ G is the mean of the gradients, σ G is the standard deviation, and k is a threshold adjustment factor and k ∈ [1.5, 2.5]; Step S2-5: analyzing the probability distribution map, and updating the corner point set with the candidate point coordinate with the highest confidence. The step S3 comprises:

4. The port container loading and unloading positioning method based on visual recognition according to claim 3, characterized in that: S3-5: calculating the cross product of adjacent vectors according to the closed loop vector, C1=V1×V2, C2=V2×V3, C3=V3×V4, C4=V4×V1, wherein C1, C2, C3, and C4 respectively represent the cross product values of two adjacent vectors. Step S3-1: calculating a geometric center point O based on the updated corner point set A={q1, q2, q3, q4} in step S2-5, wherein q1, q2, q3, q4 are four corner points in the updated corner point set; wherein q i,x , q i,y respectively represent the coordinates of the i-th corner point in the x and y directions of the image coordinate system. Step S3-2: Establish a polar coordinate system with O as the origin, and calculate the polar angle θ of each corner point i = arctan2(q i,y -O y , q i,x -O x ), where the polar angle θ i represents the azimuth angle of the i-th corner point relative to the center point O, measured counterclockwise from the positive direction of the x-axis; Step S3-3: According to θ i Sort the corner points from small to large, and generate a clockwise sequence Q = {q (1) , q (2) , q (3) , q (4)}, where q (1) represents the first corner point in the sorted sequence, and the corner point polar angle θ (1) is the smallest among the four corner points; Step S3-4: Construct a closed loop vector according to the clockwise corner point sequence, V1=q (2) -q (1) , V2=q (3) -q (2) , V3=q (4) -q (3) , V4=q (1) -q (4) , wherein V1, V2, V3, V4 represent vectors formed by adjacent corner points, respectively. The step S4 comprises:

5. The method of claim 4, wherein: The step S5 comprises: Step S4-1: Verify the sign consistency of the cross product, when taking i, j ∈ {1, 2, 3, 4} and Sign(C i ) = Sign(C j ), Sign Uniform = 1, otherwise Sign Uniform = 0; wherein Sign Uniform is the cross product sign consistency index, and Sign(C i ) represents the sign of the ith cross product value; Step S4-2: verifying the cross product absolute value threshold value, when i ∈ {1, 2, 3, 4} and |C i |a, Thresh Pass = 1, otherwise Thresh Pass = 0; wherein Thresh Pass is a cross product absolute value compliance indicator, and a is a preset threshold value, with a value range of [10, 50] pixel area units. Step S4-3: When Sign Uniform = 0 or Thresh Pass = 0, determine that the topology is invalid, trigger the corner re-detection, and return to Step S1; when Sign Uniform = 1 and Thresh Pass = 1, determine that the topology is valid, and continue to Step S4-4; Step S4-4: defining the preset three-dimensional world coordinate sequence corresponding to the sorted corner point as P={p1, p2, p3, p4}, wherein p i =(l i ,w i ,h i ) is a preset corner world coordinate, the world coordinate system takes a fixed point on the port ground as the origin, the container length extension direction as the l axis, the width extension direction as the w axis, and the vertical ground upward as the h axis, l i , w i , h i respectively correspond to the length, width, and height direction coordinates; Step S4-5: calling solvePnP algorithm to calculate camera extrinsic parameters, taking the sorted image corner points Q = {q (1) , q (2) , q (3) , q (4)} as a 2D point set based on an image coordinate system, P = {p1, p2, p3, p4} as a 3D point set based on a world coordinate system, inputting a camera intrinsic parameter matrix K, and outputting a rotation vector r and a translation vector t; wherein the camera intrinsic parameter matrix K is a matrix of inherent parameters describing the camera optical characteristics and imaging geometric relationship, the rotation vector r is a vector describing the camera rotation state, and the translation vector t is a vector describing the camera translation state. Step S4-6: using Kabsch algorithm to optimize the rotation matrix, based on the correspondence between the image corner points and the three-dimensional feature points, constructing the covariance matrix Decompose H = U∑V by SVD T Calculate the optimal rotation matrix R = VU T ; wherein is the average value of the image corner point coordinates, is the average value of the three-dimensional feature point coordinates, U, ∑, V are respectively the left singular matrix, singular value matrix, right singular matrix after SVD decomposition of the covariance matrix, and the optimal rotation matrix R is the orthogonal matrix that can minimize the mean square error between the image corner point coordinates and the three-dimensional world coordinates. Step S4-7: fuse the optimized rotation matrix R and the translation vector t, and calculate the three-dimensional coordinates of the image corner point in the world coordinate system: where x ω , y ω , z ω are the x, y, z coordinates in the world coordinate system, and x p , y p are the x, y coordinates in the image coordinate system.

6. The method of claim 5, wherein: Step S5-1: based on the three-dimensional coordinates output in Step S4-6, calculating the position of the geometric center of the container in the world coordinate system; Step S5-2: feeding back the position information to the spreader control system to guide the spreader to adjust the pose to complete the container positioning loading and unloading. The system comprises:

7. A visual recognition based port container handling positioning system for performing a visual recognition based port container handling positioning method according to any one of claims 1-6, characterized by: an image acquisition and corner point detection module, a corner point completion module, a corner point sorting and cross product calculation module, a topology verification and coordinate conversion module, and a positioning feedback module; the image acquisition and corner point detection module is used for inputting the container surface image into a pre-trained deep learning corner point detection model and outputting an initial corner point set; ​ an angle point completion module configured to extract edge features of the container when it is detected that initial angle points are missing or have a confidence lower than a preset threshold, predict sub-pixel coordinates of the missing angle points through a reconstruction network, and update the angle point set; an angle point ordering and cross product calculation module configured to order the complete angle point set in a clockwise direction to form a closed loop vector sequence, and calculate cross product values of adjacent vectors; a topology verification and coordinate conversion module configured to determine that the topology is valid when all the cross product values are of the same sign and have absolute values greater than a threshold value, convert coordinates and optimize a pose, and output three-dimensional coordinates of the container angle points in a world coordinate system; a positioning feedback module configured to determine a position of the container in the world coordinate system based on the three-dimensional coordinates, and feed back the position information to guide a spreader to complete a positioning loading or unloading.

8. The port container loading and unloading positioning system based on visual recognition according to claim 7, characterized in that: the image acquisition and angle point detection module comprises: a preprocessing unit configured to sequentially perform limited contrast adaptive histogram equalization and fixed radius median filtering on a container surface image to enhance contrast, suppress noise, and preserve angle point edge sharpness; a deep learning detection unit configured to extract multi-scale feature maps through a feature pyramid network, and output a probability distribution of each pixel being an angle point through angle point heat map regression; an angle point screening unit configured to perform non-maximum suppression on the angle point heat map, screen four angle points with the highest confidence to form an initial angle point set, and the confidence is a probability value of a pixel being a container angle point; the angle point completion module comprises: a reconstruction triggering unit configured to determine that angle point reconstruction needs to be started when the number of valid angle points in the initial angle point set is less than 4 or there is an angle point with a confidence lower than a threshold value δ; An edge feature extraction unit is configured to calculate a Sobel gradient amplitude map of the input image and by adaptive threshold T = μ G +k x σ G generating an edge mask M; a sub-pixel prediction unit configured to input an edge mask into a reconstruction network, generate a sub-pixel coordinate probability distribution map, and analyze a candidate point with the highest confidence to update the angle point set; the angle point ordering and cross product calculation module comprises: a center point calculation unit configured to calculate coordinates of a geometric center point O based on the updated corner point set A = {q1, q2, q3, q4} Polar angle sorting unit, for establishing polar coordinate system with O as origin, calculating polar angle θ of each corner point i = arctan2(q i,y -O y , q i,x -O x ), and sorting the corner points from small to large according to θ i , generating clockwise sequence Q = {q (1) , q (2) , q (3) , q (4)} Vector and cross product calculation unit for constructing closed loop vectors, V1 = q (2) -q (1) , V2 = q (3) -q (2) , V3 = q (4) -q (3) , V4 = q (1) -q (4) and calculating adjacent vector cross products, C1 = V1 x V2, C2 = V2 x V3, C3 = V3 x V4, C4 = V4 x V1.

9. The visual recognition based port container handling positioning system according to claim 7, characterized in that: the topology verification and coordinate conversion module comprises: a topology verification unit configured to verify the sign of the cross product to obtain Sign Uniform , and verify the absolute value threshold of the cross product to obtain Thresh Pass ; when Sign Uniform =0 or Thresh Pass =0, trigger corner point re-detection and return to an image acquisition and corner point detection module; when Sign Uniform =1 and Thresh Pass =1, determine that the topology is valid, and perform subsequent coordinate conversion; The world coordinate definition unit is configured to define a preset three-dimensional world coordinate sequence corresponding to the sorted corner points as P={p1, p2, p3, p4}, wherein p i =(l i ,w i ,h i ) is a preset corner world coordinate, and the world coordinate system takes a fixed point on the port ground as an origin, a container length extension direction as an l-axis, a width extension direction as a w-axis, and a direction perpendicular to the ground upward as an h-axis. An external parameter calculation unit is configured to call a solvePnP algorithm to calculate camera external parameters, so as to sort image corner points Q = {q (1) ,q (2) ,q (3) ,q (4)} as a 2D point set based on an image coordinate system, P = {p1, p2, p3, p4} as a 3D point set based on a world coordinate system, input a camera intrinsic parameter matrix K, and output a rotation vector r and a translation vector t. The rotation matrix optimization unit optimizes the rotation matrix by using a Kabsch algorithm, and constructs a covariance matrix based on the correspondence relationship between the image corner points and the three-dimensional feature points Decompose H = U∑V by SVD T Calculate the optimal rotation matrix R = VU T ; A world coordinate calculation unit is configured to fuse the optimized rotation matrix R and the translation vector t to calculate three-dimensional coordinates of the image corner points in a world coordinate system.

10. The visual recognition based port container handling positioning system according to claim 7, characterized in that: the positioning feedback module comprises: a geometric center calculation unit configured to calculate a position of a geometric center of the container in the world coordinate system based on the three-dimensional coordinates of the angle points; a spreader control interface unit configured to feed back the position information of the geometric center to a spreader control system to guide the spreader to adjust a pose to complete a positioning loading or unloading.

Images