Vision-based automatic container placing method and system for container in storage yard

Abstract

The invention belongs to the technical field of visual inspection, and discloses a visual-based automatic container placing method and system for containers in a storage yard. The method comprises the following steps: respectively mounting image acquisition devices on four corners of the lifting appliance, and acquiring images including a container below the lifting appliance and a target container in real time by using the image acquisition devices; identifying and marking the bottom corner point of the container below the lifting appliance and the top corner point of the target container by using the key point detection model to obtain detection information; wherein the key point detection model comprises a boundary perception attention module; and finally, automatically placing the container in the storage yard according to the detection information and the lifting appliance information. According to the invention, the positioning precision of automatic container placement of the storage yard container can be improved, and the cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual inspection, and more specifically, relates to a method and system for automatically placing container in the yard based on vision. Background Art

[0002] In traditional terminal operations, the gantry crane driver needs to manually control the spreader and completely rely on the human eye to observe the alignment of the container hole and the truck lock pin. This method is not only time-consuming and inefficient, but also prone to safety accidents due to fatigue or misoperation. With the growth of global trade, the cargo throughput of ports is increasing continuously, and the traditional manual operation mode has been difficult to meet the efficient and rapid loading and unloading requirements.

[0003] In order to achieve automatic container placement, researchers in this field have proposed some solutions. However, the current automatic container placement of the yard crane is mainly realized by installing laser scanners on the trolley frame or the spreader. Such solutions have high hardware costs, poor positioning accuracy, and many problems. For example, when aligning based on the recognition and detection of the lock hole, when the container under the spreader approaches the target container, the lock hole is blocked and the position offset cannot be provided. Another example is that the computational complexity is high, the inference speed is slow, or due to the container placement scenario, the target is far and blurred, and the recognition and positioning effect of the existing algorithms is poor. Summary of the Invention

[0004] The present invention provides a method and system for automatically placing container in the yard based on vision, which solves the problems of poor positioning accuracy and high cost in the automatic placement of container in the yard in the prior art.

[0005] The present invention provides a method for automatically placing container in the yard based on vision, including the following steps:

[0006] Image acquisition devices are respectively installed at the four corners of the spreader, and images containing the container under the spreader and the target container are collected in real time by using the image acquisition devices;

[0007] The bottom corner points of the container under the spreader and the top corner points of the target container are identified and marked by using a key point detection model to obtain detection information; the key point detection model includes a boundary-aware attention module;

[0008] Automatic placement of container in the yard is performed according to the detection information and spreader information.

[0009] Preferably, the boundary-aware attention module is applied after each C2f module of the backbone, and the boundary-aware attention module is used to enhance the model's attention to the boundary region;

[0010] The boundary-aware attention module performs multi-directional edge perception and fuses the multi-directional edge detection information into the attention weights. The boundary-aware attention module combines spatial attention and channel attention, and incorporates an edge guidance mechanism. It uses the spatial attention to enhance the edge region response and the channel attention to filter the boundary-related feature channels.

[0011] Preferably, after the original feature map is input into the boundary-aware attention module, the original feature map first passes through multiple parallel convolutional layers. Different convolutional layers use edge detection kernels in different directions, and then these edge feature maps in different directions are added together and fused through a convolutional layer to generate an edge saliency map.

[0012] Apply spatial attention to the edge saliency map to generate a spatial weight matrix; apply channel attention to the edge saliency map to generate channel weights.

[0013] Multiply the original feature map by the spatial weight matrix and the channel weights to obtain an enhanced feature map.

[0014] Add the original feature map and the enhanced feature map or concatenate them along the channel dimension, and then pass through a 1x1 convolution to obtain the final output feature map of the boundary-aware attention module.

[0015] Preferably, the boundary-aware attention module adopts a lightweight design and uses depthwise separable convolution for channel compression.

[0016] Preferably, the spreader information includes some or all of the trolley position of the spreader, the gantry position of the spreader, the lifting height of the spreader, the lifting speed of the spreader, the height of the target container, the open / close state of the spreader, and the container size. When automatically placing the container in the yard, adjust the position of the trolley and / or the gantry, as well as the rotation angle and height of the spreader.

[0017] Preferably, before using the key point detection model for recognition and annotation, it further includes: judging whether the detection condition is triggered according to the spreader information, and if it holds, triggering the detection process.

[0018] Preferably, after the image is acquired in real time and before the detection information is obtained, it further includes: preprocessing the image; the preprocessing includes selecting the ROI region, as well as denoising and geometric correction of the image.

[0019] Preferably, after obtaining the detection information using the key point detection model, it further includes: smoothing the detection information using the extended Kalman filter.

[0020] Preferably, the offset of a single image acquisition device is calculated according to the detection information and the spreader information, and the calculation results of the offsets corresponding to multiple image acquisition devices are fused to obtain offset information, and the automatic placement of the container in the yard is performed in combination with the offset information.

[0021] On the other hand, the present invention provides a vision-based automatic container placement system in the yard, including:

[0022] An image acquisition device for collecting an image containing the container under the spreader and the target container in real time;

[0023] A detection unit for identifying and marking the bottom corner points of the container under the spreader and the top corner points of the target container to obtain detection information;

[0024] A control unit for automatically placing the container in the yard according to the detection information and the spreader information;

[0025] The vision-based automatic container placement system in the yard is used to execute the steps in the vision-based automatic container placement method as described above.

[0026] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0027] (1) The present invention installs image acquisition devices at the four corners of the spreader, and uses the image acquisition devices to collect images containing the container below the spreader and the target container in real time; uses a key point detection model to identify and label the bottom corner points of the container below the spreader and the top corner points of the target container, and obtains detection information; among them, the key point detection model includes a boundary-aware attention module; finally, automatic container placement in the yard is performed according to the detection information and spreader information. The present invention aims at the multi-layer stacked container scenario. Considering that the corner points of the container have strong features, a key point detection method is adopted, and a point annotation method (directly annotating the bottom corner points of the container below the spreader and the top corner points of the target container) is used. Compared with the existing identification and annotation schemes, it has the advantages of faster inference speed, no need for complex post-processing logic, simple annotation, and low annotation cost. The recognition target of the present invention is the corner inflection point of the container, so the edge or boundary information of the object is very important for accurately positioning the key points. Considering this point, the present invention proposes a boundary-aware attention module (Boundary-Aware Attention Module, BAAM), which simultaneously considers edge-guided spatial attention and channel attention, and can effectively improve the model's attention to boundary features and improve the accuracy of key point positioning. By identifying the corner points of the target container and the container below the spreader, compared with the existing methods, the present invention can still provide stable trolley offset, gantry offset and rotation angle data when the keyhole is blocked, has a higher container placement success rate, and the measured success rate can reach more than 98%. The alignment accuracy is high, and the measured alignment accuracy is less than 3 cm, which greatly improves the safety and reliability of the operation. In addition, the port operation environment is complex and is often affected by factors such as weather and light. The vision-based automatic container placement scheme proposed by the present invention can operate normally in harsh environments such as rainy days, foggy days, snowy days and nights, ensuring the efficient operation of the port throughout the year without interruption.

[0028] (2) The boundary-aware attention module in the present invention is designed for convolutional neural networks and can be further lightweight designed to suit scenarios with requirements for real-time performance, and is very suitable for the application scenario of automatic container placement in the yard.

[0029] (3) The present invention uses the extended Kalman filter to make the detected key points smoother, reduce the influence of the detection result fluctuations, and obtain more stable data. Combining image preprocessing and data fusion of multiple image acquisition devices, the overall scheme can further improve the monitoring accuracy, and then achieve more accurate automatic container placement. Description of the Drawings

[0030] Figure 1 It is the algorithm processing flow chart of a vision-based automatic container placement method provided by Embodiment 1 of the present invention;

[0031] Figure 2 Images captured in real time by four image acquisition devices in an automatic container placement method based on vision provided in Embodiment 1 of the present invention;

[0032] Figure 3 An annotation example of an automatic container placement method based on vision provided in Embodiment 1 of the present invention;

[0033] Figure 4 A schematic structural diagram of a boundary-aware attention module in an automatic container placement method based on vision provided in Embodiment 1 of the present invention;

[0034] Figure 5 A schematic diagram for calculating the rotation angle in an automatic container placement method based on vision provided in Embodiment 1 of the present invention. Detailed implementation manners

[0035] To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0036] Embodiment 1:

[0037] Embodiment 1 provides an automatic container placement method based on vision, which mainly includes the following steps:

[0038] Image acquisition devices are respectively installed at the four corners of the spreader, and images including the container below the spreader and the target container are captured in real time by the image acquisition devices;

[0039] The bottom corner points of the container below the spreader and the top corner points of the target container are identified and annotated by using a key point detection model to obtain detection information; the key point detection model includes a boundary-aware attention module;

[0040] Automatic placement of the container in the yard is performed according to the detection information and the spreader information.

[0041] Among them, the boundary-aware attention module is applied after each C2f module of the backbone, and the boundary-aware attention module is used to enhance the model's attention to the boundary region. The boundary-aware attention module performs multi-directional edge perception and fuses the multi-directional edge detection information into the attention weight; the boundary-aware attention module combines spatial attention and channel attention, and adds an edge guidance mechanism, uses the spatial attention to strengthen the edge region response, and uses the channel attention to filter the boundary-related feature channels.

[0042] Specifically, after the original feature map is input into the boundary-aware attention module, the original feature map first passes through multiple parallel convolutional layers. Different convolutional layers use edge detection kernels in different directions. Then, these edge feature maps in different directions are added together and fused through a convolutional layer to generate an edge saliency map. Spatial attention is applied to the edge saliency map to generate a spatial weight matrix. Channel attention is applied to the edge saliency map to generate channel weights. The original feature map is multiplied by the spatial weight matrix and the channel weights to obtain an enhanced feature map. The original feature map and the enhanced feature map are added together or concatenated along the channel dimension, and then passed through a 1x1 convolution to obtain the final output feature map of the boundary-aware attention module.

[0043] In addition, the boundary-aware attention module can be further optimized by adopting a lightweight design and using depthwise separable convolutions for channel compression.

[0044] Among them, the spreader information includes some or all of the trolley position of the spreader, the gantry position of the spreader, the lifting height of the spreader, the lifting speed of the spreader, the height of the target container, the spreader locking / unlocking state, and the container size. When automatically placing a container in the yard, the positions of the trolley and / or gantry are adjusted, as well as the rotation angle and height of the spreader.

[0045] Based on the detection information and the spreader information, the offset of a single image acquisition device is calculated. The offset calculation results corresponding to multiple image acquisition devices are fused to obtain offset information, and the automatic container placement in the yard is performed in combination with the offset information.

[0046] In a preferred solution, before using the key point detection model for recognition and annotation, it may further include: judging whether the detection trigger condition is established according to the spreader information. If it is established, the detection process is triggered. Before obtaining the detection information after real-time image acquisition, it may further include: preprocessing the image. The preprocessing includes selecting the ROI region, as well as denoising and geometric correction of the image. After obtaining the detection information using the key point detection model, it may further include: smoothing the detection information using the extended Kalman filter.

[0047] Taking the image acquisition device as a camera and using a Programmable Logic Controller (PLC) for control as an example, a solution with several preferred designs will be described as a whole.

[0048] See Figure 1, after the present invention uses a camera to collect in real time an image including the container under the spreader and the target container, the algorithm processing flow mainly includes the following steps: reading PLC data and video frame data (including several real-time images), determining whether the trigger detection condition is satisfied. If it is satisfied, the image preprocessing module is used to preprocess the image, then key point detection and extended Kalman filtering are performed. After that, the offset calculation module is used to calculate the offset, and the multi-camera structure is fused. Then it is determined whether the container placement confirmation condition is satisfied. If it is not satisfied, the spreader rotation, trolley position, and gantry position are fine-tuned, etc. If it is satisfied, the automatic container placement is completed.

[0049] Among them, the present invention installs image acquisition modules at the four corners of the spreader to capture in real time the images of the container under the spreader and the target container. The images collected in real time by the four image acquisition devices are as Figure 2 shown. At the same time, information such as the spreader trolley position, gantry position, spreader lifting height, spreader lifting speed, target container height, spreader locking and unlocking state, and container size is read from the control unit to perform the judgment of the trigger detection condition, the judgment of the container placement confirmation condition, and the conversion calculation between pixels and actual distances. When the height difference between the spreader and the target container is less than a given threshold (such as 5 m) and the spreader is in the locked state, the trigger detection process is triggered.

[0050] Specifically, the image preprocessing in the present invention mainly includes the following steps:

[0051] 1. Select ROI: According to the corner point positions of the target container and the container under the spreader, the ROI area is preset, and then the ROI area is intercepted from the original image for processing.

[0052] 2. Image denoising: Adaptive histogram equalization is used to improve the image contrast, and median filtering is used to denoise the image.

[0053] 3. Distortion correction: Based on the camera calibration parameters (intrinsic matrix, distortion coefficient), geometric correction of the image is performed.

[0054] The present invention uses key point detection. The labeling scheme adopted by the present invention is to label the top corner points of the target container and the bottom corner points of the container under the spreader, as Figure 3 shown. In order to improve the generalization ability of the key point detection model, the present invention can also perform data augmentation processing. Specifically, the diversity of training data is enriched through data augmentation strategies such as random brightness, random saturation, random hue, random flipping, random rotation, random scaling, and random Gaussian blur to improve the generalization ability of the model.

[0055] To meet the real-time requirements, the present invention can select the YOLOv8 keypoint detection model for training and make improvements on this basis. Of course, the improvements proposed in the present invention are not limited to the YOLOv8 keypoint detection model and can also be extended to other models.

[0056] Since the recognition target of the solution proposed in the present invention is the corner inflection point of the container, the edge or boundary information of the object is very important for accurately locating the keypoints. Considering this, the present invention proposes a boundary-aware attention module and applies this module after each C2f module in the backbone to enhance the model's attention to the boundary region.

[0057] Specifically, the boundary-aware attention module proposed by the present invention has the following characteristics.

[0058] 1. Multi-directional edge perception:

[0059] Apply convolutional kernels for edge detection on the feature map, such as horizontal, vertical, and diagonal edge detection, and then fuse this edge information into the attention weights.

[0060] To enable the model to adaptively learn good edge features, the parameters of the convolutional kernel are learnable and can be initialized using common edge detection operators such as Sobel.

[0061] To maintain the light weight of the module, depthwise separable convolutional kernels are used for channel compression to maintain computational efficiency.

[0062] 2. Dual-path attention mechanism.

[0063] Combine spatial attention and channel attention, but add an edge-guided mechanism.

[0064] Among them, spatial attention strengthens the response in the edge region, and channel attention filters the boundary-related feature channels.

[0065] 3. Lightweight design.

[0066] Use depthwise separable convolution (the computational cost is reduced to 1 / 8 - 1 / 9 of the standard convolution) and channel compression (reduction_ratio = 4) to reduce the parameters of the fully connected layer.

[0067] Specifically, for the structure of the boundary-aware attention module proposed by the present invention, please refer to Figure 4 , and its working principle includes:

[0068] 1. The input feature map passes through several parallel convolutional layers. Each convolutional layer uses edge detection kernels in different directions (such as horizontal, vertical, 45 degrees, 135 degrees). The parameters of these convolutional kernels are learnable to adapt to edge features. Add the edge feature maps in these different directions and fuse them through a convolutional layer to generate an edge saliency map. To keep the module lightweight, depthwise separable convolutions are used to reduce the number of parameters.

[0069] 2. Apply spatial attention to the edge saliency map to generate a spatial weight matrix and enhance the features in the edge regions. At the same time, apply channel attention (such as the SE module) to the edge saliency map to generate channel weights and perform screening in the channel dimension.

[0070] 3 Multiply the original feature map by the spatial weight matrix and the channel weights to obtain the enhanced features.

[0071] 4. Add the original feature map and the enhanced feature map or concatenate them along the channel dimension, and then pass through a 1x1 convolution to obtain the final output features of the BAAM module.

[0072] Based on the above description, it can be seen that the boundary-aware attention module proposed in the present invention simultaneously considers edge-guided spatial attention and channel attention, which can effectively enhance the model's attention to boundary features and improve the accuracy of key point localization.

[0073] The following describes the extended Kalman filter (EKF) and offset calculation used in the present invention.

[0074] Since the detection results of each frame are independent, and there may be noise fluctuations in the single-frame detection results of the model. To reduce the impact of result fluctuations, the present invention uses an extended Kalman filter to make the detected key points smoother.

[0075] 1. Definition of state variables.

[0076] State vector:

[0077]

[0078] Among them, (x, y) are the pixel coordinates of the key point, is the pixel position change speed.

[0079] 2. Nonlinear state transition model.

[0080] Kinematics model:

[0081]

[0082] Assume that the speed of the spreader is approximately constant in a short period of time. Among them, w is the process noise and Δt is the time step.

[0083] State transition function f(x):

[0084]

[0085] Take the partial derivative of the state transition function to obtain the Jacobian matrix F, which is expressed as follows:

[0086]

[0087] 3. Visual observation model.

[0088] Observation parameters:

[0089] The key point detection model detects the top corner points of the target container and the bottom corner points of the container under the spreader, and outputs the corner point coordinates (x obs , y obs ).

[0090] Observation vector:

[0091] z = [x obs , y obs T

[0092] Observation model:

[0093] h(x) = [x, y] Τ

[0094] Observation Jacobian matrix H:

[0095]

[0096] 4. EKF algorithm process.

[0097] 1) Initialization:

[0098] Initial state: x0 = [0, 0, 0, 0] T ;

[0099] Initial covariance matrix: P0 = diag([1, 1, 0.1, 0.1]);

[0100] Process noise covariance: Q = diag([0.1, 0.1, 0.5, 0.5]);

[0101] Observation noise covariance: R = diag([0.5, 0.5]).

[0102] 2) Prediction stage:

[0103] Calculate the state prediction: x k|k-1 = f(x k-1 );

[0104] Calculate the covariance prediction: P​k|k-1 = FP k-1 F T + Q。

[0105] 3) Update phase:

[0106] Obtain visual observation value: z k ;

[0107] Calculate observation residual: y = z k - h(x k|k-1 );

[0108] Calculate Kalman gain: K k = P k|k-1 H T (HP k|k-1 H T + R) -1 ;

[0109] Update state: x k = x k|k-1 + K k y;

[0110] Update covariance: P k = (I - K k H)P k|k-1 .

[0111] Specifically, the calculation of the single-camera offset in the present invention includes:

[0112] (1) Conversion relationship between pixel distance and actual distance.

[0113] 1. Place a checkerboard calibration board with known dimensions on the ground under the spreader, ensuring that it is within the camera's field of view.

[0114] 2. Gradually adjust the height of the spreader from the lowest to the highest, and take images of the calibration board at each height. After each height adjustment, record the actual distance h between the spreader and the calibration board.

[0115] 3. Use an image processing algorithm to detect the corner points or the center of the calibration board and obtain the pixel coordinates. Based on the detected points, calculate the pixel distance d between two points with known actual distances on the calibration board.

[0116] 4. Calculate the scale factor: According to the actual distance d' and the pixel distance d, calculate the scale factor s = d' / d.

[0117] 5. Fit the relationship model: Fit the scale factors at different heights with the spreader height H to obtain the conversion relationship model between pixel distance and actual distance, which is usually a linear relationship and is expressed as follows:

[0118] s x = ax H + b x

[0119] s y = a y H + b y

[0120] Among them, s x is the scale factor of the image in the x-axis direction, and s y is the scale factor of the image in the y-axis direction.

[0121] (2) Calculate the actual offset distances of the trolley direction and the gantry direction based on the pixel distance.

[0122] Subtract the corner point coordinates of the target container from the corner point coordinates of the container under the spreader to obtain the pixel offsets d x and d y in the gantry direction (corresponding to the x-axis direction of the image coordinate system) and the trolley direction (corresponding to the y-axis direction of the image coordinate system) (i.e., obtain two pixel offsets in perpendicular directions), and then substitute them into the conversion relationship model obtained above to respectively obtain the actual offset distances of the single-camera trolley direction and the gantry direction, which are expressed as follows:

[0123] d x ′ = s x ·d x

[0124] d y ′ = s y ·d y

[0125] (3) Calculate the rotation angle between the container under the spreader and the target container, with counterclockwise being positive.

[0126] Establish a coordinate system with the position of the target container as the reference position. As Figure 5 shown, where L is the length of the container and W is the width of the container. By calculating using the formula above, the offsets of the 4 corners of the container under the spreader relative to the 4 corners of the target container can be obtained, and further the coordinates of the container under the spreader in this coordinate system can be obtained. Then, take one side of the target container and one side of the container under the spreader to calculate the rotation angle. Now, use the line segments AB and A'B' to illustrate the calculation process. There may actually be a corner where the offset value cannot be provided due to occlusion, and in this case, other sides need to be used for calculation.

[0127] a) Vector definition.

[0128] Line segment A: Assume the starting point is (0, 0) and the ending point is (L, 0), and its vector is

[0129] Line segment B: The starting point is offset by (dx1, dy1) relative to the starting point of line segment A, and the ending point is offset by (dx2, dy2) relative to the ending point of line segment A. Its vector is:

[0130] b) Calculate the angle size through the dot product.

[0131] Calculate the cosine value through the dot product formula:

[0132]

[0133] c) Determine the direction through the cross product.

[0134]

[0135] Counterclockwise direction (positive angle);

[0136] Clockwise direction (negative angle).

[0137] d) Obtain the radian value through the arccosine function and convert it to an angle.

[0138] Specifically, the principle of multi-camera data fusion in the present invention is: according to the confidence level or measurement accuracy of each camera, different weights are assigned and weighted averaging is performed. Specifically, multi-camera data fusion mainly includes the following steps:

[0139] 1. Calculate the confidence level of the measurement value of each camera (such as the confidence level of the detection box).

[0140] 2. Assign weights according to the confidence level, and the camera with a higher confidence level has a larger weight.

[0141] 3. Perform weighted averaging on the measurement values of the four cameras to obtain the final distance.

[0142] The formula used is as follows:

[0143]

[0144] Among them, D final is the final distance, D i is the offset distance obtained by the i-th camera, and w i is the weight corresponding to the i-th camera.

[0145] The spreader control of the present invention includes: when the distance between the container below the spreader and the target container is less than a given threshold (such as 10 cm), and the trolley direction offset, the gantry direction offset, and the rotation angle are less than the given threshold, perform the box-lowering operation; otherwise, control the spreader rotation, trolley direction movement, and gantry direction movement to finally meet the box-placement confirmation condition.

[0146] In summary, the vision-based automatic container placement method provided in Embodiment 1 can achieve automated operations, improve the success rate, alignment accuracy, and stability of container placement, accelerate the inference speed, and reduce costs.

[0147] Embodiment 2:

[0148] Embodiment 2 provides a vision-based automatic container placement system in a yard, including:

[0149] An image acquisition device for acquiring in real time an image including the container under the spreader and the target container;

[0150] A detection unit for identifying and marking the bottom corner points of the container under the spreader and the top corner points of the target container to obtain detection information;

[0151] A control unit for performing automatic container placement in the yard according to the detection information and spreader information;

[0152] The vision-based automatic container placement system in the yard is used to execute the steps in the vision-based automatic container placement method described in Embodiment 1.

[0153] Among them, the image acquisition device can be a high-definition camera, a camera, etc.

[0154] The detection unit includes a key point detection model, and the key point detection model includes a boundary-aware attention module.

[0155] The control unit can adopt a programmable logic controller.

[0156] Since the functions of the devices or units in the system provided in Embodiment 2 correspond to the steps in the method provided in Embodiment 1, reference can be made to the description in Embodiment 1 for understanding and will not be elaborated here.

[0157] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the examples, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A vision-based automatic container placement method in the yard, characterized in that, It includes the following steps: Image acquisition devices are respectively installed at the four corners of the spreader, and images including the container below the spreader and the target container are acquired in real time by using the image acquisition devices; The bottom corner points of the container below the spreader and the top corner points of the target container are identified and labeled by using the key point detection model to obtain detection information; the key point detection model includes a boundary-aware attention module; Automatic placement of the container in the yard is performed according to the detection information and the spreader information.

2. The automatic container placement method for a yard based on vision according to claim 1, wherein The boundary-aware attention module is applied after each C2f module of the backbone, and the boundary-aware attention module is used to enhance the model's attention to the boundary area; The boundary-aware attention module performs multi-directional edge perception and fuses the multi-directional edge detection information into the attention weight; The boundary-aware attention module combines spatial attention and channel attention and adds an edge guidance mechanism. The spatial attention is used to strengthen the response of the edge area, and the channel attention is used to screen the boundary-related feature channels.

3. The automatic container placement method based on vision in the yard according to claim 2, wherein After the original feature map is input into the boundary-aware attention module, the original feature map first passes through multiple parallel convolutional layers. Different convolutional layers use edge detection kernels in different directions, and then these edge feature maps in different directions are added together and fused through a convolutional layer to generate an edge saliency map; Spatial attention is applied to the edge saliency map to generate a spatial weight matrix; channel attention is applied to the edge saliency map to generate channel weights; The original feature map is multiplied by the spatial weight matrix and the channel weights to obtain an enhanced feature map; The original feature map and the enhanced feature map are added or concatenated along the channel dimension, and then passed through a 1x1 convolution to obtain the final output feature map of the boundary-aware attention module.

4. The vision-based automatic container placing method in the yard according to claim 3, wherein The boundary-aware attention module adopts a lightweight design and uses depthwise separable convolution for channel compression.

5. The automatic container placing method based on vision in the yard according to claim 1, wherein The spreader information includes some or all of the trolley position of the spreader, the gantry position of the spreader, the lifting height of the spreader, the lifting speed of the spreader, the height of the target container, the open / lock state of the spreader, and the container size; when automatically placing the container in the yard, the positions of the trolley and / or the gantry, as well as the rotation angle and the height of the spreader are adjusted.

6. The method for automatically placing containers in a yard based on vision according to claim 1, characterized in that, Before using the key point detection model for identification and labeling, it further includes: judging whether the detection trigger condition is established according to the spreader information, and if it is established, triggering the detection process.

7. The method for automatically placing containers in a yard based on vision according to claim 1, wherein After the images are acquired in real time and before the detection information is obtained, it further includes: preprocessing the images; the preprocessing includes selecting the ROI area, and denoising and geometric correction of the images.

8. The automatic container placing method for a yard based on vision according to claim 1, characterized in that After using the key point detection model to obtain the detection information, it further includes: smoothing the detection information by using the extended Kalman filter.

9. The method for automatically placing containers in a yard based on vision according to claim 1, characterized in that According to the detection information and the spreader information, the offset of a single image acquisition device is calculated, the offset calculation results corresponding to multiple image acquisition devices are fused to obtain offset information, and the automatic placement of the container in the yard is performed in combination with the offset information.

10. An automatic container placing system for a yard based on vision, characterized in that, It includes: An image acquisition device for acquiring in real time an image including a container under a spreader and a target container; A detection unit for identifying and marking the bottom corner points of the container under the spreader and the top corner points of the target container to obtain detection information; A control unit for automatically placing the container in the yard according to the detection information and the spreader information; The vision-based automatic container placement system in the yard is used to execute the steps in the vision-based automatic container placement method according to any one of claims 1-9.

Images