Intelligent automatic box grabbing and placing method, device and medium for body of shore bridge or field bridge spreader

Abstract

The application discloses a body intelligence automatic box grabbing and releasing method and device of a quay crane or yard crane, and relates to the field of body intelligence. The method comprises the following steps: acquiring spatial calibration data of a plurality of cameras on a spreader and a mechanical center of the spreader, and establishing a three-dimensional mapping relationship from image pixels to a spreader coordinate system; acquiring images collected by the plurality of cameras; performing target identification on the images to obtain container corner point positions; performing pose solving based on the corner point positions and the three-dimensional mapping relationship to obtain a pose deviation of the container relative to the spreader; sending the pose deviation to a spreader motion control unit to adjust the pose of the spreader; acquiring review images and performing target identification to obtain review corner point positions and lower locking cone positions; judging whether the four corner holes and the lower locking cone positions are aligned; if yes, generating a box loading instruction; and if not, updating the pose deviation and returning to the adjusting step. The application realizes high-precision alignment of the spreader and the container and reduces hardware cost.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to an embodied intelligent automatic grabbing and placing method, equipment and medium for quay cranes or yard cranes. Background Technology

[0002] In automated container terminals, the spreaders of quay cranes and yard cranes need to achieve high-precision alignment with the corner fitting holes or locking cones on the top or bottom of the container when grabbing or placing containers. This process is often referred to as "last-meter" positioning. Due to the large mass and significant inertia of the spreaders, and the fact that containers often exhibit longitudinal or lateral tilting in the yard due to uneven stacking, how to stably obtain the relative position and attitude deviation between the spreader and the target container under dynamic operating conditions is a key challenge for achieving unmanned operation.

[0003] Currently, common technologies used for this positioning stage include laser scanning positioning systems and differential GPS positioning systems. Laser scanning systems measure the relative distance and angle between the spreader and the target by scanning the container's outline, corner pieces, or pre-installed reflectors. While offering high measurement accuracy, these systems are expensive, have complex installation and calibration processes, and are susceptible to signal attenuation or misjudgment due to factors like metal surface reflections, rain, fog, salt spray, and strong electromagnetic interference in port environments. This results in frequent and difficult maintenance. Furthermore, differential GPS positioning systems rely on differential correction between satellite signals and ground reference stations to obtain the spreader's position. However, the large number of metal containers in port yards obstructs satellite signals, leading to significant multipath effects and insufficient positioning stability. This makes it difficult to continuously provide high-precision position information, and similarly, it requires expensive receiving equipment.

[0004] In practical applications, the aforementioned positioning methods often require high procurement and maintenance costs to maintain usable positioning accuracy. Furthermore, their reliability decreases under adverse weather or strong reflective conditions, and manual verification or reduced operating speed may be necessary during operation, affecting the overall operational efficiency and level of automation at the port. Summary of the Invention

[0005] This invention provides a method, equipment, and medium for the intelligent automatic container grabbing and placing of a quay crane or yard crane spreader. The technical problem it aims to solve is: how to provide an effective solution that can stably achieve high-precision alignment between the spreader and the container in the complex environment of a port while reducing hardware costs and maintenance complexity.

[0006] In a first aspect, the present invention provides a method for the embodied intelligent automatic grabbing and placing of a container for a quay crane or yard crane spreader, comprising: Acquire spatial relationship calibration data between multiple cameras on the lifting device and the mechanical center of the lifting device, and establish a three-dimensional mapping relationship from image pixels to the lifting device coordinate system based on the spatial relationship calibration data; Acquire images captured by the multiple cameras; Target recognition is performed on the image to obtain the corner positions of the container; Based on the container corner position and the three-dimensional mapping relationship, the pose of the container corner position is calculated to obtain the pose deviation of the container relative to the spreader. The positional deviation is sent to the spreader motion control unit to adjust the spreader's positional orientation; The verification images re-captured by the multiple cameras are obtained, and target recognition is performed on the verification images to obtain the verification corner point position and the lower locking cone position; Determine whether the four corner holes of the container corresponding to the verification corner point are aligned with the position of the lower locking cone; If the four corner holes of the container corresponding to the verification corner point are aligned with the lower locking cone position, a container loading command is generated; If the four corner holes of the container corresponding to the verification corner point position are not aligned with the lower locking cone position, the positional deviation of the container relative to the spreader is updated based on the verification corner point position and the three-dimensional mapping relationship, and the process returns to the step of sending the positional deviation to the spreader motion control unit to adjust the spreader position.

[0007] Optionally, wide-angle cameras are installed at the four corners of the spreader, and a top-view camera is installed at the bottom of the spreader; the step of acquiring spatial relationship calibration data between the multiple cameras on the spreader and the mechanical center of the spreader, and establishing a three-dimensional mapping relationship from image pixels to the spreader coordinate system based on the spatial relationship calibration data, includes: Obtain the installation position parameters and attitude parameters of the wide-angle camera and the top-view camera relative to the mechanical center of the lifting device, respectively; Based on the installation position parameters and the attitude parameters, calibrate the intrinsic and extrinsic parameters of the wide-angle camera and the top-view camera; Based on the calibrated intrinsic and extrinsic parameters, an initial three-dimensional mapping relationship from image pixels to the lifting device coordinate system is established; During the lifting operation, a calibration reference image containing known geometric features is acquired; The initial three-dimensional mapping relationship is corrected online based on the calibration reference image to obtain the three-dimensional mapping relationship.

[0008] Optionally, the step of performing target recognition on the image to obtain the corner positions of the container includes: The image is preprocessed to obtain a preprocessed image; The preprocessed image is input into the feature extraction layer of the recognition model to obtain a multi-scale feature map. The multi-scale feature map is input into the corner branch of the recognition model to obtain the corner position of the container.

[0009] Optionally, the preprocessing of the image to obtain a preprocessed image includes: The light intensity and weather type of the environment in which the image was captured are obtained; Based on the light intensity and the weather type, determine the image preprocessing strategy; The image is preprocessed according to the image preprocessing strategy to obtain the preprocessed image.

[0010] Optionally, the step of calculating the pose of the container corner points based on the container corner positions and the three-dimensional mapping relationship to obtain the pose deviation of the container relative to the spreader includes: Select a subset of corner points that satisfy the geometric distribution conditions from the corner points of the containers; Based on the aforementioned corner point subset, a projection relationship between the container world coordinate system and the image coordinate system is constructed; The pose is calculated by performing pose calculation on the projection relationship to obtain the six-degree-of-freedom pose of the container relative to the camera; Based on the three-dimensional mapping relationship, the six-degree-of-freedom pose of the container relative to the camera is converted into the pose deviation of the container relative to the spreader. The pose deviation includes yaw, roll, pitch and translation.

[0011] Optionally, sending the pose deviation to the spreader motion control unit to adjust the spreader pose includes: The yaw, roll, and pitch values ​​in the attitude deviation are labeled as attitude deviation components, and the translation is labeled as position deviation components. First, the attitude deviation component is sent to the spreader motion control unit to drive the spreader to adjust its attitude; After the spreader completes its attitude adjustment, the position deviation component is sent to the spreader motion control unit to drive the spreader to adjust its position.

[0012] Optionally, determining whether the four corner holes of the container corresponding to the verification corner point are aligned with the lower locking cone position includes: Based on the verified corner point positions, determine the positions of the four corner holes of the container corresponding to the verified corner point positions; Based on the positions of the four corner holes of the container, determine the projection areas of the four corner holes in the spreader coordinate system; Based on the position of the lower locking cone, determine the projection point of each locking cone in the lifting coordinate system; Calculate the overlap of each projection point falling into the corresponding projection area; Determine whether each of the aforementioned overlap degrees is greater than a preset overlap threshold; If each of the aforementioned overlaps is greater than the preset overlap threshold, then it is determined that the positions of the four corner holes of the container are aligned with the positions of the lower locking cones. If any of the aforementioned overlaps is not greater than the preset overlap threshold, then it is determined that the positions of the four corner holes of the container are not aligned with the position of the lower locking cone.

[0013] Optionally, after generating the box-setting instruction, the method further includes: After the box is captured, the box capture confirmation images collected by the multiple cameras are obtained; Target recognition is performed on the box-grabbing confirmation image to obtain the twist lock status; Determine whether the torsion lock state is a fully locked state; If the torsion lock is in a fully locked state, a box grabbing completion confirmation signal is generated; If the twist lock is not fully locked, an alarm signal is generated and the current operation is stopped.

[0014] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0016] This invention provides a method, device, and medium for the embodied intelligent automatic container grabbing and placing of a quay crane or yard crane spreader. By acquiring spatial relationship calibration data between multiple cameras and the mechanical center of the spreader and establishing a three-dimensional mapping relationship from image pixels to the spreader coordinate system, the detected pixel positions in the image can be converted into spatial positions in the spreader coordinate system, providing a unified benchmark for subsequent pose calculation. Furthermore, by acquiring images and identifying the corner positions of the container, pose calculation is performed on the corner positions based on the above mapping relationship to obtain the pose deviation of the container relative to the spreader. This achieves high-precision, non-contact visual pose measurement without relying on external positioning devices such as laser scanning or differential GPS. This approach reduces hardware configuration costs and the complexity of precision calibration and maintenance. Furthermore, the pose deviation is sent to the spreader motion control unit to adjust the spreader's pose and eliminate the current relative pose error. Further, a new verification image is acquired and the positions of the verification corner points and the lower locking cone are identified to determine if the four corner holes of the container are aligned with the lower locking cone. If aligned, a container placement command is generated to complete the container gripping and placement. If misaligned, the pose deviation is updated based on the verification corner point positions and the 3D mapping relationship, and the spreader's pose is adjusted again. This verification and update adjustment step forms an iterative correction mechanism, which can correct residual deviations caused by dynamic disturbances in complex port environments, ensuring the stability and reliability of spreader-container alignment. Therefore, through multi-camera visual calibration, pose calculation, and iterative verification correction, high-precision and stable alignment of the spreader and container in complex port environments is achieved, while reducing hardware costs and maintenance complexity. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an intelligent automatic grabbing and releasing method for a quay crane or yard crane spreader provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] Please see Figure 1This invention provides an embodied intelligent automated container grabbing and placing method using a quay crane or yard crane spreader. A quay crane is a type of container crane used for loading and unloading containers from ships at the quayhead. A yard crane is a type of container crane used for storing and retrieving containers within a container yard. Both quay cranes and yard cranes utilize spreaders to grab, lift, move, and place containers. The spreader is a container grabbing device installed below the quay crane or yard crane trolley, with a locking cone at its bottom. The locking cone is a conical protrusion used to insert into the corner fitting hole of the container and lock it in place. Embodied intelligence refers to the system's ability to continuously perceive its own and its surrounding environment through sensors deployed in the physical environment, and autonomously generate control actions based on the perception results to achieve specific operational goals. In automated terminal operations, high-precision alignment between the spreader and the target container is the core of the container grabbing and placing operation; the alignment accuracy directly determines whether the locking cone can accurately insert into the corner fitting hole of the container. Existing solutions typically use laser scanning or differential GPS for positioning, which are costly in terms of hardware and complex to maintain. Therefore, this embodiment uses multiple cameras to acquire visual information about the area beneath the spreader, combining visual calibration, target recognition, pose calculation, and closed-loop control to enable the spreader to automatically achieve high-precision alignment with the container and perform safety checks before container placement. Specifically, the method includes the following steps: S1, acquire spatial relationship calibration data between multiple cameras on the lifting device and the mechanical center of the lifting device, and establish a three-dimensional mapping relationship from image pixels to the lifting device coordinate system based on the spatial relationship calibration data.

[0025] In specific implementation, spatial relationship calibration data between multiple cameras on the spreader and the mechanical center of the spreader is acquired. Based on this spatial relationship calibration data, a three-dimensional mapping relationship from image pixels to the spreader coordinate system is established. Multiple cameras are fixedly installed at preset positions on the spreader to acquire images of the container, locking cone, and surrounding area below the spreader from different perspectives. Further, the spatial relationship calibration data includes the installation position parameters and attitude parameters of each camera relative to the mechanical center of the spreader. The installation position parameters describe the three-dimensional coordinates of the camera's optical center in the spreader coordinate system, and the attitude parameters describe the orientation angle of the camera's optical axis in the spreader coordinate system. Further, the spreader coordinate system is a three-dimensional Cartesian coordinate system established with the mechanical center of the spreader as the origin and the direction of spreader movement as the coordinate axis, used to uniformly describe the spatial positional relationships between the spreader, cameras, container, and locking cone. Further, the three-dimensional mapping relationship is used to convert pixel coordinates in the camera images into physical coordinates in the spreader coordinate system; its establishment process includes camera intrinsic parameter calibration and extrinsic parameter calibration. Furthermore, the intrinsic parameters are the camera's own optical parameters, including focal length, principal point coordinates, and lens distortion coefficients. The principal point is the intersection of the camera's optical axis and the image plane. Furthermore, the extrinsic parameters are the camera's rotation matrix and translation vector relative to the rigging coordinate system. Furthermore, by capturing multiple images of a calibration board of known dimensions for each camera, and establishing constraint equations using the known 3D coordinates of feature points on the calibration board and the corresponding pixel coordinates of feature points in the images, the intrinsic and extrinsic parameters of each camera can be solved, thereby constructing a 3D mapping relationship from image pixels to the rigging coordinate system. This 3D mapping relationship provides the spatial coordinate conversion basis for subsequent pose calculation.

[0026] For example, in some preferred embodiments, wide-angle cameras are installed at the four corners of the spreader, and a top-view camera is installed at the bottom of the spreader. The step of acquiring spatial relationship calibration data between the multiple cameras on the spreader and the mechanical center of the spreader, and establishing a three-dimensional mapping relationship from image pixels to the spreader coordinate system based on the spatial relationship calibration data, includes: acquiring the installation position parameters and attitude parameters of the wide-angle camera and the top-view camera relative to the mechanical center of the spreader, respectively; calibrating the intrinsic and extrinsic parameters of the wide-angle camera and the top-view camera according to the installation position parameters and the attitude parameters; establishing an initial three-dimensional mapping relationship from image pixels to the spreader coordinate system based on the calibrated intrinsic and extrinsic parameters; acquiring a calibration reference image containing known geometric features during spreader operation; and correcting the initial three-dimensional mapping relationship online based on the calibration reference image to obtain the final three-dimensional mapping relationship.

[0027] In practice, port spreaders are affected by load changes, mechanical vibrations, and temperature variations during long-term operation. This can cause slight changes in the spatial relationship between the camera and the spreader's mechanical center, leading to drift in the pre-calibrated 3D mapping relationship and consequently reducing pose calculation accuracy. Therefore, this embodiment addresses this by rationally configuring the camera's installation position, calibrating the camera's intrinsic and extrinsic parameters, establishing an initial 3D mapping relationship, and performing online corrections using known geometric features during spreader operation to maintain the long-term accuracy of the 3D mapping relationship. The specific implementation method is as follows: Wide-angle cameras are installed at the four corners of the spreader, and a top-view camera is installed at the bottom of the spreader. Wide-angle cameras refer to cameras with a large field of view, typically greater than 90 degrees, which is the maximum angle the camera can capture. Positioning wide-angle cameras at the four corners of the spreader allows for observation of the four corner areas of the container below the spreader from four oblique perspectives, ensuring that the corner pieces and locking cones are all within the field of view. Furthermore, the top-view camera, with its optical axis perpendicular to the bottom of the spreader, is positioned at the center of the bottom of the spreader or a predetermined location, providing a vertical view of the area directly below the spreader for observing the relative positional relationship between the locking cones and the corner holes of the container. This combination of wide-angle and top-view cameras balances the need for wide field of view coverage with the requirement for close-up detail observation. The camera installation positions are determined based on the mechanical structure and operating range of the spreader, ensuring that key targets are effectively captured by at least one camera under various operating postures.

[0028] Furthermore, the installation position parameters and attitude parameters of the wide-angle camera and the top-view camera relative to the mechanical center of the lifting device are obtained. The installation position parameters are the three-dimensional coordinates of the optical center of each camera in the lifting device coordinate system, and the attitude parameters are the deflection angle and pitch angle of the optical axis of each camera in the lifting device coordinate system. These parameters can be obtained theoretically from mechanical design drawings or directly measured using measuring instruments after installation. Furthermore, since the lifting device is subjected to significant impacts during movement, the installation position parameters and attitude parameters may shift after long-term use; therefore, these parameters need to be acquired periodically or online for subsequent calibration and correction. Acquisition methods can include measurement using specialized tooling during lifting device maintenance or online estimation using visual information during operation.

[0029] Furthermore, based on the installation position parameters and the attitude parameters, the intrinsic and extrinsic parameters of the wide-angle camera and the top-view camera are calibrated. Intrinsic parameters refer to the inherent optical parameters of the camera itself, including focal length, principal point coordinates, radial distortion coefficient, and tangential distortion coefficient. The focal length is the distance from the optical center of the lens to the imaging plane, the principal point coordinates are the coordinates of the intersection of the optical axis and the imaging plane, and the distortion coefficient describes the geometric deformation produced by the lens imaging. Extrinsic parameters refer to the rotation matrix and translation vector of the camera in the hoist coordinate system. The calibration process involves acquiring an image containing a calibration board of known dimensions, establishing constraint equations using the known three-dimensional coordinates of feature points on the calibration board and the pixel coordinates of corresponding feature points in the image, and solving for the intrinsic and extrinsic parameters. Furthermore, the calibration board can adopt a checkerboard pattern, where the corner spacing is known. The corner positions in the image are extracted using a corner detection algorithm, and then the known spacing is used for calculation. Furthermore, during the calibration process, the camera needs to capture multiple images of the calibration board from different angles. Each image provides a set of constraint equations, and a sufficient number of constraint equations can stably solve the intrinsic and extrinsic parameters.

[0030] Furthermore, based on the calibrated intrinsic and extrinsic parameters, an initial 3D mapping relationship from image pixels to the rigging coordinate system is established. This initial 3D mapping relationship, obtained for the first time after calibration, is used to convert pixel coordinates in the camera image into physical coordinates in the rigging coordinate system. This mapping relationship is further represented in mathematical transformation form, including a projection matrix or an equivalent rotation matrix and translation vector. The projection matrix is ​​a 3x4 matrix that projects the homogeneous 3D coordinates in the rigging coordinate system onto the image pixel coordinates. After the initial 3D mapping relationship is established, the system uses this mapping relationship for pose calculation during its first operation.

[0031] Furthermore, during spreader operations, calibration reference images containing known geometric features are acquired. Calibration reference images refer to target images containing markers or known geometric features with known dimensions and shapes. In port scenarios, containers themselves have regular geometric shapes, and their corner joint spacing is of standard size. The corner joint spacing of 20-foot and 40-foot containers has defined values. Further, during operations, the spreader can photograph containers already placed in the yard; the distances between the corner joints of these containers represent known geometric features. Acquiring calibration reference images can be done automatically when the spreader is idle, or images of containers undergoing alignment can be used as a reference. By examining the imaging positions of known geometric features in the images, the accuracy of the current 3D mapping relationship can be verified.

[0032] Furthermore, the initial 3D mapping relationship is corrected online based on the calibration reference image to obtain the 3D mapping relationship. The online correction process includes: projecting the 3D coordinates of known geometric features onto the image plane using the current initial 3D mapping relationship to obtain predicted pixel coordinates; extracting the actual pixel coordinates of the corresponding geometric features in the calibration reference image; calculating the deviation between the predicted pixel coordinates and the actual pixel coordinates; and adjusting the parameters in the mapping relationship according to the deviation to minimize the projection error. Further, the corrected mapping relationship is the 3D mapping relationship, used for subsequent pose calculation. Online correction does not require stopping the machine to perform the complete calibration process; instead, it is automatically completed during operation using existing scenes, enabling timely restoration of mapping accuracy even after minor deformations in the lifting device structure.

[0033] This embodiment achieves omnidirectional visual coverage of the container corner fittings and locking cones by placing wide-angle cameras at the four corners of the spreader and a top-view camera at the bottom of the spreader. The wide field of view of the wide-angle cameras ensures that the corner fittings can still be captured even when there is a certain positional deviation between the spreader and the container. The top-view camera provides a clear close-up image of the locking cones and corner holes directly below, providing a direct basis for alignment judgment. Furthermore, calibrating the intrinsic and extrinsic parameters and establishing an initial three-dimensional mapping relationship provides an accurate spatial coordinate transformation basis for visual measurement. Furthermore, online correction is performed during spreader operation using known geometric features, solving the problem of decreased mapping accuracy caused by structural drift after long-term use and avoiding frequent downtime and manual recalibration. Furthermore, the online correction mechanism does not require additional dedicated calibration equipment; it can be completed using the standard geometric features of the container itself, offering advantages such as low implementation cost and convenient maintenance.

[0034] S2, acquire images captured by the multiple cameras.

[0035] In practice, images are acquired from the multiple cameras. These images are raw video frames captured in real-time by the cameras, containing information about the container top surface, corner fittings, locking cones, and surrounding yard or transport vehicles beneath the spreader. Furthermore, the corner fittings are cast steel components at the eight corners of the container, each with holes for inserting locking cones. The image acquisition process is continuous; multiple cameras can acquire images synchronously or asynchronously to cover different areas beneath the spreader. The acquired images serve as input data for subsequent target recognition steps.

[0036] S3, perform target recognition on the image to obtain the corner positions of the container.

[0037] In practice, target recognition is performed on the image to obtain the container corner positions. Target recognition is the process of locating a specific target in an image and extracting its feature information. Corner points are feature points on the four corners of the top or bottom surface of a container, and their spatial positions have a fixed correspondence with the container's geometry. By performing target recognition on the image, the pixel coordinates of the corner points in the image can be obtained, i.e., the container corner positions. The accuracy of the corner position directly affects the accuracy of subsequent pose calculation. Furthermore, target recognition can be implemented using a deep learning-based recognition model. This model is pre-trained on a large amount of labeled data and can stably detect container corners in the complex environment of a port. The container corner positions serve as input data for subsequent pose calculation.

[0038] In some preferred embodiments, the step of performing target recognition on the image to obtain the corner position of the container includes: preprocessing the image to obtain a preprocessed image; inputting the preprocessed image into the feature extraction layer of the recognition model to obtain a multi-scale feature map; and inputting the multi-scale feature map into the corner branch of the recognition model to obtain the corner position of the container.

[0039] In practice, port environments experience drastic changes in lighting, resulting in severe metallic reflections on container surfaces. Rain, fog, and low-light conditions at night further degrade image quality. Directly detecting corners in the original image is susceptible to noise interference, leading to inaccurate corner localization and consequently affecting pose calculation accuracy. Therefore, this embodiment enhances image quality through preprocessing, generates multi-scale feature maps using a feature extraction layer, and outputs corner positions via a corner branch, improving the robustness and accuracy of corner recognition. The specific implementation is as follows: The image is preprocessed to obtain a preprocessed image. Preprocessing refers to a series of image enhancement operations performed on the original image to improve image quality, highlight corner features, and suppress noise and interference. Preprocessing operations may include brightness adjustment, contrast enhancement, dehazing, and de-reflection. Under strong port lighting conditions, images may exhibit localized overexposure and underexposed shadows; dynamic range compression technology can simultaneously preserve details in both bright and dark areas. Metallic reflections manifest as bright spots in the image; reflection suppression algorithms reduce reflection interference by detecting and repairing bright areas. Furthermore, under rain and fog conditions, image contrast is reduced; dehazing algorithms can restore scene clarity. Under low-light conditions at night, gain enhancement and noise reduction processing can improve the image signal-to-noise ratio. The preprocessed image serves as input data for subsequent feature extraction layers.

[0040] Furthermore, the preprocessed image is input into the feature extraction layer of the recognition model to obtain multi-scale feature maps. The recognition model is a neural network model used to detect targets and extract features from images. The feature extraction layer is a neural network structure in the recognition model used to extract feature representations from the input image, typically composed of multiple stacked convolutional layers. Further, the convolutional layers slide convolutional kernels across the input image and perform multiplication-accumulation operations to extract local features such as edges, textures, and corner shapes. As the network depth increases, the semantic information of the feature maps gradually increases, while the spatial resolution gradually decreases. Furthermore, multi-scale feature maps refer to feature maps with different spatial resolutions output at different depths of the feature extraction layer. Higher spatial resolution feature maps retain more precise location information, which is beneficial for accurate corner localization, while lower spatial resolution feature maps contain more abstract semantic information, which is beneficial for distinguishing corners from the background. Furthermore, in port scenarios, the distance between the camera and the container changes with the movement of the spreader, and the size of the corners in the image also changes accordingly. Multi-scale feature maps can effectively handle this scale change.

[0041] Further, the multi-scale feature map is input into the corner branch of the recognition model to obtain the container corner location. The corner branch is an output branch in the recognition model specifically used to predict corner locations. It fuses and further convolves the input multi-scale feature map to generate a corner heatmap. Further, the corner heatmap is a two-dimensional probability map corresponding to the spatial size of the input image. The value at each location in the map represents the confidence that the location is a corner; the higher the value, the more likely the location is to be a container corner. Further, by performing peak detection on the corner heatmap and extracting the pixel coordinates corresponding to the local maximum value, the container corner location can be obtained. Further, peak detection can employ a non-maximum suppression algorithm to retain the point with the highest confidence in the local neighborhood and suppress other points. The corner branch can also output sub-pixel level offsets for corners to further improve positioning accuracy. The sub-pixel offsets are predicted by an additional regression branch and added to the peak coordinates of the heatmap to obtain the sub-pixel precision corner location. Furthermore, in port operations, millimeter-level alignment accuracy requires corner positioning errors to be as small as possible, and sub-pixel positioning capability plays an important role in meeting this requirement.

[0042] It should be noted that the specific structure of the recognition model can adopt a multi-task learning architecture. This recognition model includes a shared feature extraction backbone and multiple parallel task branches. The shared feature extraction backbone is used to extract multi-scale feature maps from the preprocessed image, and each task branch generates the corresponding object detection or segmentation result based on the multi-scale feature maps.

[0043] Furthermore, the shared feature extraction backbone can be composed of multiple stacked convolutional layers. These convolutional layers slide their kernels across the input feature map and perform multiplication-accumulation operations, extracting local features such as edges, textures, and corner shapes layer by layer. As the network depth increases, the semantic information of the feature maps gradually increases, while the spatial resolution gradually decreases. The shared feature extraction backbone outputs multiple feature maps with different spatial resolutions at different depths—i.e., multi-scale feature maps—to balance detailed location information with abstract semantic information.

[0044] Furthermore, the multiple parallel task branches include at least corner point branches, boundary branches, corner fitting branches, twist lock branches, locking cone branches, container type branches, and twist lock state branches. The corner point branch predicts the container corner positions; the boundary branch predicts the container boundaries to obtain the container's precise outline; the corner fitting branch detects the positions of the container corner fittings; the twist lock branch detects the positions of the twist locks; the locking cone branch detects the positions of the lower locking cones; the container type branch identifies the container type, including 20-foot containers, 40-foot containers, and high cube containers; and the twist lock state branch identifies whether the twist lock is in an unlocked or fully locked state. Each task branch takes the multi-scale feature map output from the shared feature extraction backbone as input, and undergoes its own independent convolutional or fully connected processing to generate the corresponding output result.

[0045] Furthermore, the recognition model can be trained using supervised learning. During training, a large number of labeled images of port scenes are acquired. Each labeled image includes the container boundary, corner position, corner fitting position, twist lock position, locking cone position, container type label, and twist lock status label. The labeled images are input into the recognition model, and the model outputs the prediction results for each task branch. The loss value of each branch is calculated based on the prediction results and corresponding label information, and then the loss values ​​of each branch are weighted and summed to obtain the total loss value. The parameters of the recognition model are updated by backpropagation based on the total loss value until the total loss value converges or reaches the preset training rounds. After training, the recognition model can simultaneously output the container corner position, container boundary, corner fitting position, twist lock position, locking cone position, container type, and twist lock status.

[0046] Furthermore, the container boundaries output by the recognition model can be used to help determine the precise range of the container, avoiding interference from background areas in pose calculation. Corner piece and locking cone positions can be used for alignment checks before container placement and for locking confirmation after container gripping. Container type information can be used to determine the standard dimensional parameters of the container, thereby improving the accuracy of pose calculation. Twist lock status information can be used to determine whether the twist lock is fully locked after container gripping, deciding whether to allow the spreader to lift and move the container. These output information are used separately at different stages of automatic container gripping and placement, enabling visual perception to cover the entire process of container finding, alignment, gripping, lifting, moving, positioning, landing, and unlocking.

[0047] Furthermore, wide-angle cameras positioned at the four corners of the spreader face the four corner areas below the spreader, with each camera's field of view covering the corresponding corner and its surrounding area. A top-view camera at the bottom of the spreader shoots vertically downwards, covering the central area below the spreader and the entire top surface of the container. When determining the container corner positions, target recognition can be performed simultaneously on the images captured by each camera, obtaining candidate corner positions in each image and the confidence level for each candidate position.

[0048] Furthermore, since the fields of view of adjacent wide-angle cameras partially overlap near the corner fittings of containers, the same container corner may appear simultaneously in images from two or more cameras. Using a pre-established 3D mapping relationship from image pixels to the spreader coordinate system for each camera, the candidate corner positions detected in each camera image can be transformed into the spreader coordinate system. For the same corner appearing simultaneously in multiple camera images, the transformation results in the spreader coordinate system should be consistent. These transformation results can be matched, and the matched multi-view observations can be fused, for example, by taking a weighted average to determine the 3D coordinates of the corner in the spreader coordinate system, to reduce the impact of single-camera image noise and viewing angle deviation on the corner positioning accuracy.

[0049] Furthermore, when a wide-angle camera fails to detect a corner point due to reflections on the container surface, rain or fog, or partial obstruction of its view, the candidate corner points detected by the other cameras can still be used for that corner point location, providing redundancy and fault tolerance. When the spreader and container are nearly aligned, the top-view camera can capture images of the corner areas on the top surface of the container from a top perspective. The corner point positions detected in this image can be fused or verified with those detected by the wide-angle camera, further improving the reliability of corner point location.

[0050] Furthermore, the final container corner position is the corner position obtained after multi-camera collaborative fusion, which can be represented as a three-dimensional coordinate in the spreader coordinate system or a two-dimensional coordinate in a certain reference image coordinate system, and is used for subsequent pose calculation steps.

[0051] This embodiment improves image quality through preprocessing, then generates multi-scale feature maps through a feature extraction layer, and outputs corner locations via a corner point branch, achieving high-precision detection of container corners. Furthermore, preprocessing suppresses the effects of adverse factors such as strong light, reflections, rain, fog, and low nighttime illumination, ensuring stable input for the feature extraction layer. Furthermore, the multi-scale feature maps take into account both detail and semantic information, enabling the recognition model to accurately locate container corners at different scales. Furthermore, the corner point branch focuses on the specific target of the corner, decoupling it from other task branches and avoiding interference between multiple tasks. This recognition process provides reliable feature point input for subsequent pose calculation. By employing a deep learning recognition model, the system can learn corner feature patterns from a large amount of actual port image data, exhibiting strong adaptability to complex backgrounds and occlusion conditions.

[0052] In some preferred embodiments, the preprocessing of the image to obtain a preprocessed image includes: acquiring the light intensity and weather type of the image acquisition environment; determining an image preprocessing strategy based on the light intensity and weather type; and preprocessing the image according to the image preprocessing strategy to obtain a preprocessed image.

[0053] In practice, the light intensity and weather type in port environments change frequently, making it difficult for fixed preprocessing methods to adapt to all scenarios. Images are prone to overexposure under bright sunlight, while images lack contrast under cloudy or rainy / foggy conditions. At night, it is necessary to increase brightness while suppressing noise. Therefore, this embodiment dynamically determines the preprocessing strategy based on the light intensity and weather type of the image acquisition environment, performing dynamic range compression and reflection suppression processing. This ensures that the preprocessed image has quality suitable for subsequent recognition model processing under different environmental conditions. The specific implementation method is as follows: The image acquisition process obtains the ambient light intensity and weather type of the image acquisition environment. Illumination intensity refers to the strength of ambient light during image acquisition, which can be measured by the camera's built-in photosensitive element or obtained through a separate ambient light sensor. Illumination intensity can be expressed in lux or indirectly reflected by the camera's automatic exposure parameters. Weather type refers to the weather conditions during image acquisition, including sunny, cloudy, rainy, foggy, and nighttime. Weather type can be determined by accessing real-time weather forecast data through a meteorological information interface or by analyzing the statistical characteristics of the image itself. Foggy images typically have high overall grayscale and low contrast, rainy images may contain water droplets or lens glare, and nighttime images generally have low brightness. The weather type can be determined by combining meteorological data and image analysis results. Illumination intensity and weather type serve as inputs for determining the image preprocessing strategy.

[0054] Further, an image preprocessing strategy is determined based on the light intensity and the weather type. The image preprocessing strategy specifies the algorithm type and parameters to be used in subsequent image processing. The preprocessing strategy can be determined through table lookup or rule matching. When the light intensity is greater than a preset strong light threshold and the weather type is sunny, the image preprocessing strategy is determined to be a strong light suppression mode. The preprocessing operations corresponding to this mode include reducing the brightness of highlight areas, increasing the brightness of shadow areas, dynamic range compression, and reflection suppression processing. Further, when the weather type is rain or fog, the image preprocessing strategy is determined to be a defogging enhancement mode. The preprocessing operations corresponding to this mode include atmospheric scattering model defogging and contrast stretching. Further, when the light intensity is lower than a preset low light threshold, the image preprocessing strategy is determined to be a nighttime enhancement mode. The preprocessing operations corresponding to this mode include gain enhancement and 3D noise reduction. Further, when the weather type is cloudy, the image preprocessing strategy is determined to be a brightness equalization mode. The preprocessing operations corresponding to this mode include histogram equalization. The preset strong light threshold and preset low light threshold can be determined based on historical data analysis. Image preprocessing strategies can also include combinations of the above preprocessing operations and corresponding parameter sets to adapt to image enhancement needs under different environmental conditions.

[0055] Further, the image is preprocessed according to the aforementioned image preprocessing strategy to obtain a preprocessed image. After determining the image preprocessing strategy, the image is processed accordingly based on the preprocessing operations specified by the strategy. The preprocessed image serves as the input data for subsequent feature extraction layers. Dynamic range compression refers to mapping areas with excessive brightness differences in a high dynamic range image to a displayable or processable range while preserving details in both bright and dark areas. Further, dynamic range compression can employ global tone mapping or local tone mapping methods. Global tone mapping uses the same mapping function for the entire image, while local tone mapping performs adaptive mapping based on pixel neighborhood information. Further, reflection suppression processing is used to eliminate locally bright areas caused by reflections from metal surfaces. This can be achieved by detecting bright saturated areas in the image and filling or repairing them using neighborhood information. Specifically, pixels in the image whose brightness exceeds a preset reflection threshold are marked as reflective areas, and interpolation repair is performed using surrounding normal pixels to restore corner or edge features that may exist below the reflective area. On the surface of port containers, reflections typically occur in the corner edges exposed to direct sunlight; therefore, reflection suppression is particularly important for corner detection. Furthermore, atmospheric scattering model dehazing is used to restore the clarity of images under rainy and foggy weather, and contrast stretching is used to enhance the distinction between foreground and background in the image. Furthermore, gain enhancement is used to increase the brightness of low-light images, and 3D noise reduction is used to suppress noise introduced after brightness enhancement. Furthermore, histogram equalization is used to improve the problem of overall darkness and insufficient contrast in images under cloudy conditions, making the image brightness distribution more uniform.

[0056] This embodiment dynamically determines the preprocessing strategy based on the light intensity and weather type of the image acquisition environment. Preprocessing the image according to this strategy solves the problem of fixed preprocessing methods performing poorly under different environmental conditions. Port environments exhibit diverse lighting and weather variations; dynamically adjusting the preprocessing strategy can specifically improve image quality. The preprocessing strategy is determined in real-time based on environmental parameters, allowing the system to adapt to weather changes without manual intervention, ensuring the stability of continuous operation. Compared to a fixed preprocessing workflow, the dynamic strategy reduces the recognition failure rate caused by environmental factors, improving the overall availability of the automated container handling system.

[0057] S4. Based on the container corner position and the three-dimensional mapping relationship, the pose of the container corner position is calculated to obtain the pose deviation of the container relative to the spreader.

[0058] In specific implementation, based on the container corner positions and the three-dimensional mapping relationship, the pose of the container corner positions is calculated to obtain the pose deviation of the container relative to the spreader. The pose deviation includes three translational and three rotational amounts. The three translational amounts are the displacement deviations of the container relative to the spreader in the forward / backward, left / right, and up / down directions, respectively. The three rotational amounts are yaw, roll, and pitch, respectively. Further, the pose calculation process involves using the three-dimensional mapping relationship to map the two-dimensional corner positions in the image to the three-dimensional geometric model of the container. By solving the spatial geometric relationship, the difference between the actual pose and the desired pose of the container in the spreader coordinate system is obtained. Further, the desired pose refers to the ideal position and attitude of the container relative to the spreader when the spreader and the container are perfectly aligned. The result of the pose calculation is the pose deviation, which quantifies the magnitude and direction of the spreader adjustment required, and is used as data subsequently sent to the spreader motion control unit.

[0059] In some preferred embodiments, the step of calculating the pose of the container corner points based on the container corner point positions and the three-dimensional mapping relationship to obtain the pose deviation of the container relative to the spreader includes: selecting a subset of corner points from the container corner point positions that meet the geometric distribution conditions; constructing a projection relationship between the container world coordinate system and the image coordinate system based on the corner point subset; calculating the pose of the projection relationship to obtain the six-degree-of-freedom pose of the container relative to the camera; and converting the six-degree-of-freedom pose of the container relative to the camera into the pose deviation of the container relative to the spreader according to the three-dimensional mapping relationship, wherein the pose deviation includes yaw, roll, pitch, and translation.

[0060] In practical implementation, the corner points obtained from target recognition may contain false detections or noise. Directly using all corner points for pose calculation can lead to unstable results, especially when some corner point positions deviate significantly from their true positions, potentially resulting in large errors in the calculated pose. Therefore, this embodiment improves the robustness and accuracy of pose calculation by selecting a subset of corner points that meet geometric distribution conditions, constructing projection relationships, and performing pose calculation. The specific implementation method is as follows: From the container corner locations, a subset of corner points that meet the geometric distribution conditions is selected. The geometric distribution conditions refer to the corner points being sufficiently dispersed in space to form stable geometric constraints. The four corner points of the container should be roughly rectangular in shape, and no three corner points should be collinear. Further, during the selection process, the Euclidean distance and angular relationships between each corner point can be calculated to exclude combinations of corner points that are too concentrated or approximately collinear. Euclidean distance is the straight-line distance between two points in space, which is the pixel distance between two pixels on the image plane. For the four corner points, their pairwise distances can be calculated. If the ratio of the maximum distance to the minimum distance exceeds a preset ratio, it indicates the presence of abnormal corner points. Alternatively, the interior angles of the quadrilateral formed by the four corner points can be calculated. If any interior angle deviates from 90 degrees by more than a preset angle threshold, it indicates that the corner point position may have errors. The selection process can be based on a random sampling consensus algorithm, randomly selecting the minimum number of corner point combinations for geometric verification, and retaining the subset of corner points that pass the verification. The random sample consensus algorithm iteratively selects the smallest sample set, calculates model parameters, and counts the number of interior points that satisfy the model, ultimately selecting the model with the most interior points. In this scenario, the model is the rectangular geometric model of a container, and the interior points are the corner points that conform to this model. By filtering based on geometric distribution conditions, falsely detected corner points can be effectively eliminated, improving the quality of corner points used for pose calculation.

[0061] Furthermore, a projection relationship between the container world coordinate system and the image coordinate system is constructed based on the aforementioned corner point subset. The container world coordinate system is a three-dimensional Cartesian coordinate system fixed on the container, with its origin defined at the geometric center of the container's bottom surface, and coordinate axes along the container's length, width, and height directions, respectively. The image coordinate system is a two-dimensional coordinate system on the camera's image plane, with its origin at the top left corner of the image, the horizontal axis representing the pixel column direction, and the vertical axis representing the pixel row direction. Further, the projection relationship describes how three-dimensional points in the container world coordinate system are projected into the image coordinate system. For container corner points, their world coordinates are determined by the container's standard dimensions. Further, the image coordinates are output from the corner point branch. The projection relationship is determined by the camera's intrinsic parameters and the container's extrinsic parameters relative to the camera. Constructing the projection relationship means establishing a series of correspondences between world coordinate points and image coordinate points.

[0062] Further, pose calculation is performed on the projection relationship to obtain the six-DOF pose of the container relative to the camera. The pose calculation uses the PnP algorithm. PnP stands for Perspective-n-Point, which takes n 3D points and their corresponding image projection points as input and outputs the 3D position and pose of the camera. For example, in this embodiment, each corner point in the corner point subset provides a correspondence between a 3D world coordinate point and a 2D image coordinate point. The PnP algorithm obtains the six-DOF pose of the container relative to the camera by minimizing the reprojection error. The reprojection error refers to the Euclidean distance between the predicted pixel coordinates and the actual pixel coordinates of a 3D point projected onto the image plane according to the current estimated pose. The goal of the PnP algorithm is to find the six-DOF pose that minimizes the sum of squared reprojection errors of all corner points. Commonly used PnP methods include direct linear transformation, EPnP, and UPnP. Furthermore, the direct linear transformation solves for the projection matrix by constructing a system of linear equations, and then decomposes it to obtain the pose; EPnP represents a 3D point as a linear combination of four control points, and uses these control points to solve for the pose, resulting in high computational efficiency and stable accuracy. In port operations, the number of corner points is usually four, which precisely meets the minimum requirement for PnP, and the rectangular geometry of the container provides strong constraints, ensuring that the pose calculation accuracy meets millimeter-level alignment requirements.

[0063] Furthermore, based on the aforementioned three-dimensional mapping relationship, the six-DOF pose of the container relative to the camera is converted into the pose deviation of the container relative to the spreader. This pose deviation includes yaw, roll, pitch, and translation. Furthermore, the three-dimensional mapping relationship includes coordinate transformation information of the camera relative to the spreader. By transforming the six-DOF pose of the container relative to the camera using the rotation matrix and translation vector in the three-dimensional mapping relationship, the six-DOF pose of the container relative to the spreader can be obtained. This is then compared with the desired pose of the spreader to obtain the pose deviation. Furthermore, yaw is the rotation angle deviation about the vertical axis, roll is the rotation angle deviation about the spreader's forward / backward axis, pitch is the rotation angle deviation about the left / right axis, and translation is the displacement deviation along the three coordinate axes. This pose deviation quantifies the direction and magnitude of adjustment required by the spreader.

[0064] This embodiment improves the stability of pose estimation by filtering a subset of corner points that meet geometric distribution conditions, eliminating the adverse effects of concentrated or collinear corner points. Based on the corner point subset, a projection relationship is constructed and solved using the PnP algorithm to obtain the six-DOF pose of the container relative to the camera. Furthermore, the pose in the camera coordinate system is transformed to the spreader coordinate system through a three-dimensional mapping relationship to obtain the pose deviation that the spreader needs to adjust. This process effectively utilizes the coordinate transformation relationship obtained from calibration to convert the visual measurement results into the pose deviation required for spreader control. Furthermore, the combined use of filtering and PnP calculation reduces the interference of individual erroneous corner points on the overall pose estimation, significantly improving the robustness and accuracy of pose estimation. This pose estimation method achieves high-precision spatial positioning using only visual information.

[0065] S5, send the position deviation to the spreader motion control unit to adjust the spreader position.

[0066] In practice, the posture deviation is sent to the spreader motion control unit to adjust the spreader's posture. The spreader motion control unit is responsible for driving the spreader to perform translational and rotational movements. After receiving the posture deviation, it generates corresponding control commands based on the translational and rotational amounts in the deviation, driving the spreader to move along the corresponding degrees of freedom, gradually bringing the spreader closer to and aligning it with the container. Furthermore, the sending process can be implemented via a high-speed fieldbus. The fieldbus is an industrial communication network connecting the control system and field devices. A high-speed fieldbus can provide millisecond-level communication cycles, meeting the real-time requirements of spreader motion control. After adjustment, the spreader's posture changes, and the process enters the verification stage.

[0067] In some preferred embodiments, sending the attitude deviation to the spreader motion control unit to adjust the spreader attitude includes: labeling the yaw, roll, and pitch of the attitude deviation as attitude deviation components, and labeling the translation as position deviation components; first sending the attitude deviation components to the spreader motion control unit to drive the spreader to perform attitude adjustment; and after the spreader completes attitude adjustment, then sending the position deviation components to the spreader motion control unit to drive the spreader to perform position adjustment.

[0068] In practice, if attitude and position adjustments are performed simultaneously during the lifting device adjustment process, motion coupling may cause oscillations or new deviations during the adjustment process. If the lifting device is directly translated horizontally when there is a horizontal rotational deviation, the locking cone may not align with the corner hole, requiring repeated adjustments. Therefore, this embodiment decomposes the pose deviation into attitude deviation and position deviation components, adjusting the attitude first and then the position, reducing interference from motion coupling and improving the stability and convergence speed of the alignment process. The specific implementation method is as follows: The yaw, roll, and pitch components of the pose deviation are labeled as attitude deviation components, and the translational component is labeled as position deviation components. The attitude deviation components are used to adjust the horizontal angle of the spreader, ensuring the bottom surface of the spreader is parallel to the top surface of the container. Further, the translational component includes displacement deviations in the forward / backward, left / right, and up / down directions. The position deviation components are used to adjust the spatial position of the spreader, aligning the center of the spreader with the center of the container. Decomposing the pose deviation into attitude and position deviation components forms the basis for subsequent step-by-step adjustments.

[0069] Further, the attitude deviation component is first sent to the spreader motion control unit to drive the spreader to adjust its attitude. After receiving the attitude deviation component, the spreader motion control unit controls the spreader's rotation mechanism to eliminate yaw, roll, and pitch, making the spreader's bottom surface parallel to the container's top surface. During attitude adjustment, the spreader's spatial position remains temporarily unchanged. Since attitude adjustment only requires driving the rotation actuator, the movement amplitude is relatively small, and the adjustment speed is fast. After attitude adjustment is completed, the spreader's locking cone direction is aligned with the container's corner hole direction, preparing for position adjustment.

[0070] Furthermore, after the spreader completes its attitude adjustment, the position deviation component is sent to the spreader motion control unit to drive the spreader to perform position adjustment. Upon receiving the position deviation component, the spreader motion control unit controls the spreader's translation mechanism to eliminate displacement deviations in the front-back, left-right, and up-down directions, ensuring the spreader center coincides with the container center. Position adjustment is performed after attitude adjustment; at this point, the spreader is parallel to the top surface of the container. During position adjustment, the directions of the locking cone and corner holes do not change due to attitude variations, thus avoiding interference from motion coupling. After position adjustment, the spreader's locking cone aligns with the four corner holes of the container, meeting the container placement alignment requirements.

[0071] This embodiment decomposes the pose deviation into attitude deviation components and position deviation components, adjusting the attitude first and then the position, avoiding motion coupling and error accumulation caused by simultaneous adjustments. The attitude adjustment first ensures that the spreader's gripping direction is consistent with the container's corner hole direction, while the position adjustment follows to ensure the locking cone accurately aligns with the corner hole. Furthermore, the step-by-step adjustment reduces mutual interference during spreader movement, improving the stability and convergence speed of the alignment process. Furthermore, this method matches the control characteristics of the spreader motion control unit, enabling more effective use of visual pose deviation information, allowing the spreader to achieve precise alignment with fewer adjustments. Furthermore, because attitude and position deviations are handled separately, the control logic is clearer, avoiding mechanical shocks and overshoot caused by simultaneous multi-degree-of-freedom movement. Step-by-step adjustment also facilitates fault diagnosis; if alignment cannot be achieved after attitude adjustment and position adjustment, an anomaly in the attitude adjustment process can be determined, allowing for rapid problem localization.

[0072] S6, acquire the verification images re-captured by the multiple cameras, perform target recognition on the verification images, and obtain the verification corner point position and the lower locking cone position.

[0073] In specific implementation, the verification images re-captured by the multiple cameras are acquired, and target recognition is performed on the verification images to obtain the verification corner point positions and the lower locking cone positions. After the spreader completes a pose adjustment, the multiple cameras re-capture images in the current state; these images are called verification images. Further, target recognition is performed on the verification images to obtain the adjusted container corner point positions, i.e., the verification corner point positions, and simultaneously, the position of the lower locking cone in the image is identified, i.e., the lower locking cone position. Further, the lower locking cone position is the pixel coordinates of the bottom locking cone in the verification image or the spreader coordinate system coordinates transformed through a 3D mapping relationship. The verification corner point positions and the lower locking cone positions are used together as input data for alignment judgment.

[0074] S7, determine whether the four corner holes of the container corresponding to the verification corner point are aligned with the position of the lower locking cone.

[0075] In practice, it is determined whether the four corner holes of the container corresponding to the verified corner point position are aligned with the position of the lower locking cone. The four corner holes of the container are through holes on the four corner fittings of the container for the insertion of the locking cone. There is a fixed geometric offset relationship between the center of the four corner holes on each corner fitting and the corresponding corner point. This offset relationship is determined by the container's design dimensions. Therefore, the position of the corresponding four corner holes of the container can be determined based on the verified corner point position. Alignment means that the position of the four corner holes of the container in the spreader coordinate system coincides with the position of the lower locking cone in the horizontal plane, and the allowable deviation is within a preset accuracy range. The alignment determination is used to confirm whether the current posture of the spreader meets the container placement conditions.

[0076] In some preferred embodiments, determining whether the four corner holes of the container corresponding to the verification corner point position are aligned with the position of the lower locking cone includes: determining the position of the four corner holes of the container corresponding to the verification corner point position; determining the projection area of ​​the four corner holes in the spreader coordinate system based on the position of the four corner holes of the container; determining the projection point of each locking cone in the spreader coordinate system based on the position of the lower locking cone; calculating the overlap of each projection point falling into the corresponding projection area; determining whether each overlap is greater than a preset overlap threshold; if each overlap is greater than the preset overlap threshold, then determining that the position of the four corner holes of the container is aligned with the position of the lower locking cone; if any overlap is not greater than the preset overlap threshold, then determining that the position of the four corner holes of the container is not aligned with the position of the lower locking cone.

[0077] In practice, before placing the container, it is necessary to ensure that the locking cone under the spreader is accurately aligned with the four corner holes on the top of the container. Judging the alignment solely by visual inspection or simple image overlay is prone to misjudgment due to viewing angle deviations, obstructions, or image noise. Therefore, this embodiment determines the positions of the four corner holes based on the verified corner point positions, then determines the projection areas of the corner holes and the projection points of the locking cone, calculates the projection overlap, and performs a threshold judgment. This transforms the alignment judgment into a quantifiable numerical comparison, improving the accuracy and reliability of the judgment. The specific implementation method is as follows: Based on the verified corner point positions, the corresponding container corner hole positions are determined. The container corner holes are located on four corner fittings, and there is a fixed geometric offset relationship between the center of each corner hole and the corresponding corner point. This geometric offset relationship is determined by the container's design dimensions and is a known constant for standard containers. Furthermore, based on the verified corner point positions and the relative offset vectors between the corner points and the centers of the corner holes, the positions of the four corner holes in the spreader coordinate system can be calculated. This offset vector can be pre-stored in the system and used directly after the corner point positions are determined.

[0078] Furthermore, based on the positions of the four corner holes of the container, the projection areas of the four corner holes in the spreader coordinate system are determined. The spreader coordinate system is a three-dimensional rectangular coordinate system, and its horizontal plane is defined as the plane where the bottom surface of the spreader lies. The projection area of ​​the four corner holes in the spreader coordinate system refers to the two-dimensional area formed after the three-dimensional positions of the four corner holes are vertically projected onto the horizontal plane of the bottom surface of the spreader. Since the corner holes have physical dimensions, the projection area is not a single point, but an area with a certain area. Furthermore, the size of the projection area is determined based on the actual diameter of the corner holes and the allowable positioning error. The corner holes of standard containers have a definite diameter, and the projection area can be taken as a circular area with the center projection of the corner hole as the center and the radius equal to the radius of the corner hole plus the preset positioning tolerance. After determining the projection area, the projection point of the locking cone can be compared with the projection area.

[0079] Further, based on the position of the lower locking cone, the projection point of each locking cone in the spreader coordinate system is determined. The position of the lower locking cone is the location of the bottom locking cone of the spreader in the image, obtained through target recognition. Further, according to the three-dimensional mapping relationship, the locking cone position is transformed into the spreader coordinate system to obtain the three-dimensional coordinates of each locking cone. The three-dimensional coordinates of the locking cone are then vertically projected onto the horizontal plane of the spreader bottom to obtain the projection point of each locking cone. The projection point reflects the position of the top of the locking cone on the plane of the spreader bottom.

[0080] Further, the overlap of each projection point falling into the corresponding projection area is calculated. The overlap represents the degree of coincidence between the projection point and the projection area. One approach is to define the overlap as a normalized value of the distance from the projection point to the center of the projection area. Let the projection area be a circle with radius R, and the distance from the projection point to the center of the circle be d. Then, the overlap can be defined as 1 minus the ratio of d to R. When d is less than R, the overlap is greater than 0; when d is 0, the overlap is at most 1. The method for calculating the overlap can be flexibly chosen based on the shape of the projection area. A higher overlap indicates that the locking cone projection point is closer to the center of the corner hole projection area, and the alignment is higher.

[0081] Further, it is determined whether each of the aforementioned overlap degrees is greater than a preset overlap threshold. The preset overlap threshold is a pre-set minimum overlap degree requirement for alignment, which can be set to 0.8, meaning that the projected point needs to be located near the center of the projection area to be considered aligned. Further, if each of the aforementioned overlap degrees is greater than the preset overlap threshold, then the four corner holes of the container are determined to be aligned with the lower locking cone position. Further, if any of the aforementioned overlap degrees is not greater than the preset overlap threshold, then the four corner holes of the container are determined to be misaligned with the lower locking cone position. This determination covers all locking cone projection points; alignment is only considered complete when all threshold conditions are met, ensuring the safety of container placement.

[0082] This embodiment determines the positions of the four corner holes based on the verified corner point locations, thereby defining the corner hole projection areas. It also determines the projection points based on the position of the lower locking cone, calculating the overlap for quantitative judgment, thus solving the problem of inaccurate alignment based on subjective visual judgment. Furthermore, the calculation of projection overlap transforms alignment judgment into a quantifiable numerical comparison, avoiding misjudgments caused by viewing angle, occlusion, or image noise. Moreover, the preset overlap threshold setting makes the judgment standard adjustable, allowing for flexible configuration according to actual operational accuracy requirements. This method improves the accuracy and reliability of safety confirmation before placing the box, reducing the risk of locking cones hitting the box or incorrect insertion into the corner holes due to alignment errors. Since the overlap calculation is based on projection in the lifting coordinate system, it is unaffected by the camera's installation position and viewing angle, ensuring objectivity and consistency in the judgment results.

[0083] S8. If the four corner holes of the container corresponding to the verification corner point are aligned with the lower locking cone position, a container loading command is generated.

[0084] In practice, if the four corner holes of the container corresponding to the verification corner point are aligned with the lower locking cone position, a container placement command is generated. The container placement command is a control signal that allows the spreader to descend and insert the locking cone into the four corner holes of the container. Generating the container placement command means that the vision system has confirmed that the alignment error between the spreader and the container is within a safe range, and the spreader can perform the container placement action.

[0085] In some preferred embodiments, after generating the box-grabbing instruction, the method further includes: after grabbing the box, acquiring box-grabbing confirmation images captured by the plurality of cameras; performing target recognition on the box-grabbing confirmation images to obtain the twist lock state; determining whether the twist lock state is a fully locked state; if the twist lock state is a fully locked state, generating a box-grabbing completion confirmation signal; if the twist lock state is not a fully locked state, generating a stop alarm signal and stopping the current operation.

[0086] In practice, during the automatic container placement process, after the spreader inserts the locking cone into the four corner holes of the container, it is necessary to confirm that the torsion lock is fully locked before the container can be safely lifted and moved. If the torsion lock is not fully locked, the container may fall off during the spreader's lifting. Therefore, this embodiment obtains the torsion lock status by acquiring a container grab confirmation image and performing target recognition, determining whether the torsion lock is fully locked. A confirmation signal is generated when the lock is fully locked, and a stop alarm signal is generated and the operation is stopped when the lock is not fully locked, thus achieving automatic safety confirmation after container grabbing. The specific implementation method is as follows: After container grabbing, confirmation images of the grabbing are acquired from the multiple cameras. Container grabbing refers to the action of lowering the spreader and inserting the locking cones into the four corner holes of the container. After grabbing, the spreader and the container are temporarily connected through the locking cones and corner holes. At this time, it is necessary to confirm that the twist locks are fully tightened to ensure a reliable connection. Multiple cameras acquire images of the bottom of the spreader and the locking cone area after container grabbing; these images are called the container grabbing confirmation images. The container grabbing confirmation images clearly show the state of the locking cones and the mating area of ​​the corner holes. Since the locking cones are located at the bottom of the spreader, a top-down camera usually provides a frontal image of the locking cone area, while a wide-angle camera can provide a side auxiliary view.

[0087] Further, target recognition is performed on the box-grabbing confirmation image to obtain the twist lock state. The twist lock is a locking mechanism installed inside the locking cone. When the locking cone is inserted into the corner hole, the twist lock rotates a certain angle, and its locking pin extends and engages with the edge of the corner hole, achieving locking. Further, the twist lock state includes a fully locked state and a partially locked state. A fully locked state refers to the state where the locking pin of the twist lock is fully extended and correctly engaged with the edge of the corner hole; a partially locked state refers to the state where the locking pin is not fully extended, not rotated to the correct position, or the engagement angle is insufficient. The target recognition model, after training, can identify the position of the locking pin, the extension length of the locking pin, or the locking indicator mark of the twist lock from the box-grabbing confirmation image, thereby determining the twist lock state. The recognition model can output the twist lock state classification result, or it can output intermediate information such as the extension length of the locking pin for judgment.

[0088] Further, it is determined whether the torsion lock state is fully locked. A fully locked state is a safety prerequisite for allowing the spreader to perform subsequent lifting and movement actions. The determination process involves comparing the identified torsion lock state with a preset fully locked state standard. If the torsion lock state is fully locked, a container grab completion confirmation signal is generated, allowing the spreader to lift and move the container. Further, the container grab completion confirmation signal is used to notify the spreader motion control unit or the operation scheduling system that the container is now safely locked. If the torsion lock state is not fully locked, a stop alarm signal is generated and the current operation is stopped. Stopping the operation includes stopping the spreader movement, maintaining the current position of the spreader and container, and issuing an audible and visual alarm. Operators can intervene to check the locking mechanism, or the system can automatically attempt to re-execute the locking action.

[0089] This embodiment achieves automatic safety confirmation after container grabbing by acquiring a confirmation image of the grabbing and performing target recognition to determine the twistlock status and whether it is fully locked. When the twistlock is fully locked, a confirmation signal is generated, allowing subsequent actions; when it is not fully locked, a stop alarm signal is generated and the operation is stopped to prevent the spreader from moving the container and causing an accident due to incomplete locking. Furthermore, this visual confirmation step extends the safety of automatic container grabbing and placement from before placement to after grabbing, forming a full-process safety interlocking mechanism. This process is entirely based on visual perception and is executed automatically without manual inspection. Furthermore, vision-based twistlock status recognition can achieve remote automatic confirmation without adding additional hardware, adapting to the efficient operational rhythm of automated terminal yards and quayside. This safety confirmation mechanism avoids the risk of container detachment due to incomplete twistlock locking, ensuring the safety of equipment and personnel.

[0090] S9, if the four corner holes of the container corresponding to the verification corner point position are not aligned with the lower locking cone position, then update the positional deviation of the container relative to the spreader based on the verification corner point position and the three-dimensional mapping relationship, and return to the step of sending the positional deviation to the spreader motion control unit to adjust the spreader position.

[0091] In specific implementation, if the four corner holes of the container corresponding to the verified corner point position are not aligned with the lower locking cone position, the pose deviation of the container relative to the spreader is updated based on the verified corner point position and the three-dimensional mapping relationship, and the process returns to the step of sending the pose deviation to the spreader motion control unit to adjust the spreader pose. Further, updating the pose deviation means recalculating the current pose deviation of the container relative to the spreader using the verified corner point position, and replacing the previously sent deviation with the recalculated deviation. Further, the updated pose deviation reflects the residual deviation that still exists after the previous adjustment of the spreader. Further, after returning to the sending step, the spreader motion control unit continues to adjust the spreader pose according to the updated deviation, then re-acquires the verification image and performs alignment judgment, forming a closed-loop iteration until the alignment condition is met.

[0092] This embodiment acquires images of the area beneath the spreader using multiple cameras, establishes a three-dimensional mapping relationship based on spatial calibration, and uses target recognition and pose calculation to obtain pose deviations, driving the spreader motion control unit to automatically adjust its alignment. Furthermore, after adjustment, a second image is acquired for alignment verification. If aligned, a container placement command is generated; otherwise, the pose deviation is updated and adjustment continues. This method replaces laser positioning with visual perception, reducing hardware costs and simplifying maintenance. Furthermore, the closed-loop control structure enables the spreader to continuously correct its pose based on real-time visual feedback, adapting to complex operating conditions such as uneven container stacking and spreader motion errors. Alignment confirmation before container placement extends visual perception to safety-critical stages, preventing locking cones from colliding with the container or incorrect insertion due to alignment deviations. The data flow forms a coherent closed loop from image acquisition, pose calculation, control adjustment to visual verification. The output of each step serves as the input for subsequent steps, eliminating manual intervention and achieving automated container handling and placement.

[0093] It should be noted that the "embodied intelligence" of the automated container grabbing and releasing method for quay cranes or yard cranes involved in this invention is not a decorative name for a general vision algorithm, but rather a direct product of a series of specific technical constraints formed by the deep coupling between the physical entity of the spreader and the port operating environment. Embodied intelligence systems differ fundamentally from pure vision software systems or cloud-based inference systems, and these differences directly determine the necessity and specificity of the various technical means in this embodiment. First, the spreader is subjected to alternating loads and mechanical vibrations during long-term operation, causing a slow drift in the spatial relationship between the camera fixedly installed on the spreader and the mechanical center of the spreader. If only factory calibration or periodic calibration is relied upon, the three-dimensional mapping relationship will become inaccurate over time, directly affecting the accuracy of pose calculation. This is the physical motivation for introducing an online calibration correction mechanism based on known geometric features in this embodiment; if removed from the embodied environment of long-term vibration of the spreader, only a one-time calibration is required without an online correction mechanism. Furthermore, the port environment presents harsh optical conditions such as strong reflections from metal surfaces, salt spray, rain and fog, and drastic changes in day and night illumination. Fixed preprocessing strategies cannot adapt to such diverse and dynamically changing environmental interferences, leading to a significant decrease in target recognition success rate. This is the practical reason why this embodiment sets up a dynamic adjustment strategy for image preprocessing based on light intensity and weather type. If the complex optical interferences of the port physical environment are removed, constant preprocessing parameters can meet the requirements, and there is no need to introduce environmental adaptive scheduling. Furthermore, the spreader has a large mass and significant motion inertia. If attitude deviation and position deviation are sent to the motion control unit for simultaneous adjustment, system oscillation will occur due to multi-degree-of-freedom motion coupling, resulting in slow convergence speed or even failure to achieve stable alignment. This is the physical reason why this embodiment decomposes the pose deviation into attitude and position components and forces attitude adjustment to be performed first, followed by position adjustment. If the physical motion characteristics of the spreader's large inertia are removed, any sequential adjustment will not affect stability, and a step-by-step strategy is not necessary. In summary, the core mechanisms in this embodiment, such as online calibration correction, dynamic preprocessing strategy, and attitude-position step-by-step adjustment, all originate from the inherent physical constraints and operational characteristics of the spreader being "embodied" in the port operation scenario, rather than a simple transfer of general visual positioning or image enhancement methods. Therefore, "embodied intelligence" has substantial technical connotation and creative contribution in this invention.

[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0095] Please see Figure 2 , Figure 2This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0096] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0097] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to perform an embodied intelligent automatic box-grabbing and releasing method for a quay crane or yard crane spreader.

[0098] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0099] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an embodied intelligent automatic grabbing and placing method for a quay crane or yard crane spreader.

[0100] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0101] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of the embodied intelligent automatic grabbing and releasing box method for a quay crane or yard crane spreader provided in any of the above method embodiments.

[0102] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0103] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0104] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of the embodied intelligent automatic grabbing and releasing box method for a quay crane or yard crane spreader provided in any of the above method embodiments.

[0105] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0107] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0108] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0110] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0111] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for the integrated intelligent automatic grabbing and placing of a container on a quay crane or yard crane spreader, characterized in that, include: Acquire spatial relationship calibration data between multiple cameras on the lifting device and the mechanical center of the lifting device, and establish a three-dimensional mapping relationship from image pixels to the lifting device coordinate system based on the spatial relationship calibration data; Acquire images captured by the multiple cameras; Target recognition is performed on the image to obtain the corner positions of the container; Based on the container corner position and the three-dimensional mapping relationship, the pose of the container corner position is calculated to obtain the pose deviation of the container relative to the spreader. The positional deviation is sent to the spreader motion control unit to adjust the spreader's positional orientation; The verification images re-captured by the multiple cameras are obtained, and target recognition is performed on the verification images to obtain the verification corner point position and the lower locking cone position; Determine whether the four corner holes of the container corresponding to the verification corner point are aligned with the position of the lower locking cone; If the four corner holes of the container corresponding to the verification corner point are aligned with the lower locking cone position, a container loading command is generated; If the four corner holes of the container corresponding to the verification corner point position are not aligned with the lower locking cone position, the positional deviation of the container relative to the spreader is updated based on the verification corner point position and the three-dimensional mapping relationship, and the process returns to the step of sending the positional deviation to the spreader motion control unit to adjust the spreader position.

2. The method according to claim 1, characterized in that, Wide-angle cameras are installed at the four corners of the lifting device, and a top-view camera is installed at the bottom of the lifting device; the process of acquiring spatial relationship calibration data between the multiple cameras on the lifting device and the mechanical center of the lifting device, and establishing a three-dimensional mapping relationship from image pixels to the lifting device coordinate system based on the spatial relationship calibration data, includes: Obtain the installation position parameters and attitude parameters of the wide-angle camera and the top-view camera relative to the mechanical center of the lifting device, respectively; Based on the installation position parameters and the attitude parameters, calibrate the intrinsic and extrinsic parameters of the wide-angle camera and the top-view camera; Based on the calibrated intrinsic and extrinsic parameters, an initial three-dimensional mapping relationship from image pixels to the lifting device coordinate system is established; During the lifting operation, a calibration reference image containing known geometric features is acquired; The initial three-dimensional mapping relationship is corrected online based on the calibration reference image to obtain the three-dimensional mapping relationship.

3. The method according to claim 1, characterized in that, The step of performing target recognition on the image to obtain the corner positions of the container includes: The image is preprocessed to obtain a preprocessed image; The preprocessed image is input into the feature extraction layer of the recognition model to obtain a multi-scale feature map. The multi-scale feature map is input into the corner branch of the recognition model to obtain the corner position of the container.

4. The method according to claim 3, characterized in that, The preprocessing of the image to obtain a preprocessed image includes: The light intensity and weather type of the environment in which the image was captured are obtained; Based on the light intensity and the weather type, determine the image preprocessing strategy; The image is preprocessed according to the image preprocessing strategy to obtain the preprocessed image.

5. The method according to claim 1, characterized in that, The method of calculating the pose of the container corner points based on the container corner positions and the three-dimensional mapping relationship to obtain the pose deviation of the container relative to the spreader includes: Select a subset of corner points that satisfy the geometric distribution conditions from the corner points of the containers; Based on the aforementioned corner point subset, a projection relationship between the container world coordinate system and the image coordinate system is constructed; The pose is calculated by performing pose calculation on the projection relationship to obtain the six-degree-of-freedom pose of the container relative to the camera; Based on the three-dimensional mapping relationship, the six-degree-of-freedom pose of the container relative to the camera is converted into the pose deviation of the container relative to the spreader. The pose deviation includes yaw, roll, pitch and translation.

6. The method according to claim 5, characterized in that, Sending the pose deviation to the spreader motion control unit to adjust the spreader pose includes: The yaw, roll, and pitch values ​​in the attitude deviation are labeled as attitude deviation components, and the translation is labeled as position deviation components. First, the attitude deviation component is sent to the spreader motion control unit to drive the spreader to adjust its attitude; After the spreader completes its attitude adjustment, the position deviation component is sent to the spreader motion control unit to drive the spreader to adjust its position.

7. The method according to claim 1, characterized in that, The step of determining whether the four corner holes of the container corresponding to the verification corner point position are aligned with the position of the lower locking cone includes: Based on the verified corner point positions, determine the positions of the four corner holes of the container corresponding to the verified corner point positions; Based on the positions of the four corner holes of the container, determine the projection areas of the four corner holes in the spreader coordinate system; Based on the position of the lower locking cone, determine the projection point of each locking cone in the lifting coordinate system; Calculate the overlap of each projection point falling into the corresponding projection area; Determine whether each of the aforementioned overlap degrees is greater than a preset overlap threshold; If each of the aforementioned overlaps is greater than the preset overlap threshold, then it is determined that the positions of the four corner holes of the container are aligned with the positions of the lower locking cones. If any of the aforementioned overlaps is not greater than the preset overlap threshold, then it is determined that the positions of the four corner holes of the container are not aligned with the position of the lower locking cone.

8. The method according to claim 1, characterized in that, After generating the box-setting instruction, the method further includes: After the box is captured, the box capture confirmation images collected by the multiple cameras are obtained; Target recognition is performed on the box-grabbing confirmation image to obtain the twist lock status; Determine whether the torsion lock state is a fully locked state; If the torsion lock is in a fully locked state, a box grabbing completion confirmation signal is generated; If the twist lock is not fully locked, an alarm signal is generated and the current operation is stopped.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images