Automatic box dropping synchronous control method and system based on multi-camera vision assistance

Abstract

This invention discloses an automatic box-dropping synchronization control method and system based on multi-camera vision assistance. The method includes: installing at least four cameras on a lifting device and establishing a unified coordinate system; assigning a unified clock timestamp to the images from each camera; detecting the spatial position of keyholes in the images and associating them with the timestamps; performing time window matching on the data from different cameras based on a synchronization time threshold to form a synchronized observation set; compensating for the time difference of keyhole positions within the set according to the lifting device's movement speed; verifying the geometric consistency of the compensated data using multi-hole positions and eliminating abnormal data; calculating translational and rotational deviations based on the verified keyhole positions, generating control commands to drive the lifting device in closed-loop adjustment until the box-dropping accuracy is met. This invention effectively eliminates detection errors caused by time asynchrony between multiple cameras and lifting device movement, significantly improving the accuracy and stability of automatic box-dropping.

Description

Technical Field

[0001] This invention relates to the field of port automated lifting equipment control technology, specifically to an automatic container lowering synchronous control method and system based on multi-camera vision assistance. Background Technology

[0002] In automated port operations, the precise alignment of the spreader's locking head with the container's locking hole is crucial for improving operational efficiency and safety when container cranes automatically lower containers. Traditional automated container lowering control methods often rely on single vision sensors or LiDAR, which struggle to simultaneously achieve high-precision detection and wide-area perception. In recent years, multi-camera vision-assisted solutions have gained increasing attention. By installing multiple cameras on the spreader, multi-view images of the container's locking hole can be acquired, allowing for the calculation of the relative pose between the spreader and the container.

[0003] However, in practical applications, multi-camera systems face significant technical challenges. First, it is difficult to achieve complete synchronization of image acquisition from different cameras, especially when the spreader is moving at high speed or swaying laterally. Even a small time difference can cause different cameras to detect keyhole positions corresponding to different spreader postures, introducing serious dynamic errors and reducing the accuracy of pose calculation. Second, existing methods lack an effective real-time synchronization and filtering mechanism for multi-channel detection data, which can easily mix conflicting keyhole data from different moments of motion into the same calculation process, leading to misinterpretation of control commands and even causing box lowering failures or equipment collisions.

[0004] Therefore, there is an urgent need for an automatic box-dropping control method that can perform high-precision time synchronization, keyhole pose dynamic compensation, and multi-hole geometric consistency verification of multi-camera images while the lifting device is in motion, so as to improve the accuracy, stability and robustness of automatic box-dropping. Summary of the Invention

[0005] In view of the above problems, the present invention provides a technical solution for automatic box lowering synchronization control based on multi-camera vision assistance, in order to solve the problems of low accuracy of keyhole pose detection, unstable automatic box lowering control and poor reliability caused by asynchronous acquisition of images from multiple cameras and vibration and swaying of the lifting device when the lifting device is in motion.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an automatic box-dropping synchronization control method based on multi-camera visual assistance, the method comprising: S1: Install at least four cameras on the spreader and establish a unified spreader coordinate system; calibrate each camera to establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters; S2: Create an independent image acquisition thread for each camera. When acquiring each frame of image, record the time when the frame of image is acquired as a timestamp based on a unified clock, thereby giving a unified time reference to the multiple image data streams. S3: Perform keyhole target detection on each frame image, obtain the spatial position of the keyhole in the frame image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; S4: Set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error; perform time window matching on keyhole detection data collected from different cameras; when the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronous observation set; otherwise, discard the keyhole detection data with the earliest timestamp and rematch. S5: When the lifting device is in motion, obtain the current speed of the lifting device, and use the earliest timestamp in the synchronous observation set as the reference time. Based on the current speed, perform time difference compensation on the spatial position of the keyholes corresponding to other timestamps in the lifting device coordinate system to eliminate dynamic errors caused by asynchronous frame image acquisition time. S6: Perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within the preset deviation range; at the same time, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; and reducing the weight of keyhole detection data with a confidence level lower than the preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and is removed. S7: Based on the actual position of the lock hole verified by the geometric consistency constraint of the multi-hole position and the ideal position of the lock hole, calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation around the vertical axis; generate control commands for driving the lifting device actuator to perform position adjustment according to the translational deviation and rotational deviation. S8: Send the control command to the actuator of the spreader to drive the spreader to move to the target position according to the control command; during the movement of the spreader, repeat steps S2 to S7 to form a closed loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, and determine that the automatic box dropping is completed.

[0007] Furthermore, in step S1, the at least four cameras are rigidly mounted at the four corners of the spreader, and each camera captures an image of a lock hole location area on the container.

[0008] Furthermore, in step S2, the unified clock is a monotonic clock built into the lifting device control system; the timestamp is a unique and monotonically increasing positive real number, satisfying: ; Where j is the camera number, This represents the image data captured by the j-th camera. This represents a single frame image from the j-th camera. Here is the acquisition timestamp corresponding to this frame of image, and i is the time index. It represents the positive real number field.

[0009] Furthermore, step S3 includes: The YOLO detection model based on deep learning or the geometric feature detection algorithm based on edge contour and corner point matching is used to detect each frame of the image. The detection confidence scores of the frame containing the bounding rectangle of the keyhole, the center pixel coordinates of the keyhole, and the corresponding interval [0,1] are obtained. Only the detection confidence scores greater than or equal to the preset confidence threshold c are retained. min Valid keyhole detection data; Transform the center pixel coordinates of the keyhole in the valid keyhole detection data to the lifting coordinate system to obtain two-dimensional spatial coordinates (x... j ,y j ), and compared with the timestamp t of the corresponding frame image. i Detection confidence level c j Binding, forming structured keyhole detection data P j =(x j ,y j ,c j ,t i ).

[0010] Furthermore, in step S4, the synchronization time threshold The settings are dynamically adjusted based on the lifting device's movement speed and the maximum permissible spatial error, satisfying the following relationship: ; in, The maximum permissible spatial error is preset. This refers to the speed of the lifting device.

[0011] Furthermore, in step S5, the calculation formula for the time difference compensation is as follows: ,in, This represents the spatial position of the keyhole in the lifting device coordinate system after time difference compensation. The spatial position of the lock hole in the lifting device coordinate system before time difference compensation. The earliest timestamp in the synchronized observation set. This refers to the speed of the lifting device.

[0012] Furthermore, in step S6, the verification of the geometric consistency constraints of the multi-pore site is performed step by step according to the following priority: First, perform confidence constraint verification, which includes: traversing all time-difference compensated keyhole detection data in the current synchronous observation set, and checking the detection confidence level c corresponding to each keyhole detection data. j Remove c j Below the preset confidence threshold c min Abnormal data should be retained, while valid data with acceptable confidence levels should be preserved. Then, distance constraint verification is performed, specifically including: for any two different cameras j and k retained after passing the confidence constraint verification, the spatial positions of the keyholes are... , Calculate the actual spatial distance between the two in the spreader coordinate system. and according to the ideal position Q of the keyhole j Q k Calculated ideal spatial distance A comparison is performed; if the deviation between the actual spatial distance and the ideal spatial distance exceeds the preset deviation range, the observation data is determined to not meet the distance constraint. Then, the parallelism and symmetry relationships are verified, specifically including: verifying the spatial positions of the four keyholes that have passed the distance constraint verification. , Verify whether the two pairs of opposite sides of the quadrilateral formed by the data satisfy the parallel relationship and whether the two diagonals satisfy the symmetry relationship. If either the parallel relationship or the symmetry relationship is not satisfied, the observation data set is determined to not satisfy the direction constraint. If only a single keyhole detection data set does not satisfy any of the above constraints, the single abnormal data set is removed, and a three-point geometric fitting is performed based on the spatial positions of the keyholes in the remaining three valid keyhole detection data sets to complete the spatial position of the fourth keyhole. If two or more keyhole detection data sets do not satisfy the constraints, the entire observation data set is directly determined to be abnormal and all of them are removed.

[0013] Furthermore, in step S7, the generated control commands are filtered before being output; the object of the filtering is the spreader pose adjustment command generated based on translational and rotational deviations, and the filtering uses moving average filtering or Kalman filtering.

[0014] Furthermore, step S8 also includes a safety fault tolerance and bin-based optimization mechanism, specifically including: When the calculated translational and rotational deviations are both less than the preset deceleration threshold, the speed of the spreader is automatically reduced to minimize the spatial deviation caused by time synchronization errors. When any camera experiences an image acquisition interruption, no valid keyhole detection result, or a detection confidence level that remains below the preset safety threshold, the system automatically switches to a three-point geometric fitting mode. It then uses the valid keyhole detection data from the other three cameras to fill in the missing keyhole spatial position and continue to calculate the pose of the lifting device. When the completed keyhole spatial location data has geometric conflicts, and the maximum time difference within the constructed synchronous observation set exceeds the synchronization time threshold, the automatic control process is paused and switched to manual control mode or semi-automatic control mode.

[0015] In a second aspect, the present invention provides an automatic box-dropping synchronization control system based on multi-camera visual assistance, the system being used to perform the method as described in the first aspect of the present invention, the system comprising: The calibration and initialization module is used to install at least four cameras on the spreader and establish a unified spreader coordinate system; to calibrate each camera, establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters; The image acquisition and time synchronization module is used to create an independent image acquisition thread for each camera. When acquiring each frame of image, it records the time when the frame of image was acquired as a timestamp based on a unified clock, thereby giving a unified time reference to multiple image data streams. The keyhole detection module is used to detect keyhole targets in each frame of image, obtain the spatial position of the keyhole in the frame of image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; The time window synchronization module is used to set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error. The module performs time window matching on keyhole detection data collected from different cameras. When the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronization observation set. Otherwise, the keyhole detection data with the earliest timestamp is discarded and the matching is performed again. The motion compensation module is used to obtain the current speed of the spreader when it is in motion, and to use the earliest timestamp in the synchronous observation set as a reference time to compensate for the time difference of the spatial position of the keyholes corresponding to other timestamps in the spreader coordinate system according to the current speed, so as to eliminate the dynamic error caused by the asynchronous acquisition time of frame images. The geometric consistency verification module is used to perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within a preset deviation range; simultaneously, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; furthermore, reducing the weight of keyhole detection data with a confidence level lower than a preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and discarded. The pose calculation module is used to calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation about the vertical axis based on the actual position of the key hole verified by the multi-hole geometric consistency constraint and the ideal position of the key hole; and to generate control commands for driving the lifting device actuator to perform position adjustment based on the translational deviation and rotational deviation. The closed-loop control module is used to send the control commands to the actuator of the spreader, driving the spreader to move towards the target position according to the control commands. During the movement of the spreader, the image acquisition and time synchronization module, keyhole detection module, time window synchronization module, motion compensation module, geometric consistency verification module, and pose calculation module are repeatedly called to form a closed-loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, at which point the automatic box dropping is determined to be completed.

[0016] Unlike existing technologies, the above-mentioned technical solution involves an automatic container lowering synchronization control method and system based on multi-camera vision assistance, belonging to the field of port automation control. The method includes: installing at least four cameras on the spreader and establishing a unified coordinate system; assigning a unified clock timestamp to the images from each camera; detecting the spatial position of the keyholes in the images and associating them with the timestamps; matching the data from different cameras using time windows based on a synchronization time threshold to form a synchronized observation set; compensating for the time difference in the keyhole positions within the set according to the spreader's movement speed; verifying the geometric consistency of the compensated data, including spatial distance comparison and quadrilateral parallel symmetry verification, and eliminating abnormal data; finally, calculating translational and rotational deviations based on the verified keyhole positions, generating control commands to drive the spreader to perform closed-loop adjustments until the container lowering accuracy is met. This invention effectively eliminates detection errors caused by time asynchrony between multiple cameras and spreader movement, significantly improving the accuracy and stability of automatic container lowering.

[0017] The above description of the invention is merely an overview of the technical solution of the present invention. In order to enable those skilled in the art to better understand the technical solution of the present invention and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of the present invention easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of the present invention. Attached Figure Description

[0018] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on the present invention.

[0019] In the accompanying drawings of the instruction manual: Figure 1 The flowchart illustrates an automatic box-dropping synchronization control method based on multi-camera visual assistance, as described in a first exemplary embodiment of the present invention. Figure 2 This is a schematic diagram of a module of an automatic box-dropping synchronization control system based on multi-camera visual assistance, which is an exemplary embodiment of the present invention. Figure 3 This is a schematic diagram of the overall structure of an automatic box-dropping synchronization control system based on multi-camera visual assistance, which is an exemplary embodiment of the present invention. Figure 4 The flowchart illustrates an automatic box-dropping synchronization control method based on multi-camera visual assistance, as described in a second exemplary embodiment of the present invention. Figure 5 This is a schematic diagram of a multi-camera time synchronization and time window constraint mechanism according to an exemplary embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the geometric structure relationship and pose deviation calculation principle of a container lock hole according to an exemplary embodiment of the present invention. Figure 7 This is a schematic diagram of the closed-loop control process of automatic box dropping according to an exemplary embodiment of the present invention; The reference numerals used in the above figures are explained as follows: 10. Automatic box-dropping synchronous control system based on multi-camera vision assistance; 101. Calibration and Initialization Module; 102. Image acquisition and time synchronization module; 103. Keyhole detection module; 104. Time window synchronization module; 105. Motion compensation module; 106. Geometric Consistency Verification Module; 107. Pose Calculation Module; 108. Closed-loop control module. Detailed Implementation

[0020] To explain in detail the possible application scenarios, technical principles, specific feasible solutions, and the objectives and effects that can be achieved by this invention, the following detailed description is provided in conjunction with the listed specific embodiments and accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this invention, and are therefore only examples, and should not be used to limit the scope of protection of this invention.

[0021] In the first aspect, such as Figure 1 As shown, the present invention provides an automatic box-dropping synchronization control method based on multi-camera visual assistance, the method comprising: S1: Install at least four cameras on the spreader and establish a unified spreader coordinate system; calibrate each camera to establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters; S2: Create an independent image acquisition thread for each camera. When acquiring each frame of image, record the time when the frame of image is acquired as a timestamp based on a unified clock, thereby giving a unified time reference to the multiple image data streams. S3: Perform keyhole target detection on each frame image, obtain the spatial position of the keyhole in the frame image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; S4: Set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error; perform time window matching on keyhole detection data collected from different cameras; when the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronous observation set; otherwise, discard the keyhole detection data with the earliest timestamp and rematch. S5: When the lifting device is in motion, obtain the current speed of the lifting device, and use the earliest timestamp in the synchronous observation set as the reference time. Based on the current speed, perform time difference compensation on the spatial position of the keyholes corresponding to other timestamps in the lifting device coordinate system to eliminate dynamic errors caused by asynchronous frame image acquisition time. S6: Perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within the preset deviation range; at the same time, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; and reducing the weight of keyhole detection data with a confidence level lower than the preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and is removed. S7: Based on the actual position of the lock hole verified by the geometric consistency constraint of the multi-hole position and the ideal position of the lock hole, calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation around the vertical axis; generate control commands for driving the lifting device actuator to perform position adjustment according to the translational deviation and rotational deviation. S8: Send the control command to the actuator of the spreader to drive the spreader to move to the target position according to the control command; during the movement of the spreader, repeat steps S2 to S7 to form a closed loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, and determine that the automatic box dropping is completed.

[0022] In this embodiment, the lifting device coordinate system refers to a two-dimensional rectangular coordinate system fixed to the lifting device body with the center of the lifting device as the origin, serving as a unified benchmark for spatial position calculation.

[0023] Camera calibration refers to solving the camera's intrinsic, extrinsic, and distortion parameters, establishing the mapping relationship between image pixel coordinates and the rigging coordinate system, and eliminating lens distortion and installation deviations.

[0024] The ideal position of the lock hole refers to the preset coordinates of the standard lock holes of the container in the spreader coordinate system. It is distributed in a standard rectangle and provides a comparison benchmark for the actual test results.

[0025] An image acquisition thread refers to an independent execution unit allocated to a single camera, ensuring that multiple images are acquired in parallel without blocking each other.

[0026] A timestamp is a unique and monotonically increasing value generated by a unified clock, used to mark the moment of image acquisition and to establish a unified time reference.

[0027] Keyhole detection data refers to structured data that includes the spatial location of the keyhole, detection confidence level, and corresponding timestamp, serving as the basic input for subsequent synchronization, compensation, and verification.

[0028] Synchronization time threshold refers to the maximum time difference dynamically determined based on the speed of the spreader movement and the allowable spatial error, used to filter time-consistent observation data.

[0029] Time window matching refers to filtering detection data that meet the synchronization threshold based on timestamps to form a set of synchronous observations at the same time.

[0030] Time difference compensation refers to converting the position of the keyhole at different acquisition times to the same reference time based on the speed of the lifting device, thereby eliminating asynchronous spatial errors caused by the movement.

[0031] Geometric consistency constraints for multi-hole locations refer to verifying the rationality of keyhole location data from three dimensions: confidence level, distance, and parallel symmetry relationship, eliminating anomalies, filling in missing data, and ensuring the reliability of observation data.

[0032] Translational / rotational deviation refers to the average horizontal offset of the actual position of the keyhole relative to its ideal position, and the angular deviation around the vertical axis, characterizing the positional error of the lifting device.

[0033] Closed-loop control refers to the continuous correction of the spreader's position and posture through a cyclical iterative process of "acquisition—detection—synchronization—compensation—verification—calculation—control" until the accuracy requirements are met.

[0034] This embodiment describes a method that uses software to achieve spatiotemporal synchronization of multiple cameras, dynamic error compensation, and geometric constraint verification, achieving high-precision closed-loop alignment. The method first deploys at least four cameras on the lifting device and completes calibration, establishing a unified spatial reference and pixel-space mapping. Simultaneously, it presets ideal keyhole positions to provide a benchmark for subsequent comparisons. To ensure temporal consistency across multiple images, each camera acquires data in parallel using an independent thread. All image frames share the same clock and generate a unique timestamp, ensuring a unified time reference for all data streams. After acquisition, keyhole detection is performed on the images, transforming pixel coordinates to the lifting device coordinate system to form structured data containing position, confidence level, and timestamp, ensuring complete and traceable information. To avoid asynchronous acquisition errors, a synchronization threshold is dynamically set based on the lifting device speed and allowable error. Time windows are used to match and filter observation data at the same moment, eliminating data with excessively large time differences to ensure temporal consistency of the observation set. During lifting device movement, real-time speed is used to compensate for the time difference in keyhole positions at different times, unifying asynchronous data to the same reference time and eliminating dynamic spatial errors caused by movement. The compensated data undergoes multi-dimensional geometric verification: first, low-confidence data is removed; then, the deviation between the keyhole spacing and the standard value is verified; finally, the parallel symmetry of the rectangle is verified. Abnormal data is removed in stages or completed using three-point fitting to ensure data reliability. Based on the verified actual and ideal positions, the horizontal translation and rotational deviations of the spreader are calculated, and position adjustment commands are generated. After the commands are issued, the spreader is driven to move, and the entire process is iterated in a loop, continuously correcting the pose until the deviation is less than the preset accuracy threshold, completing the automatic box lowering.

[0035] The above solution achieves spatiotemporal synchronization through software, significantly reducing equipment costs and deployment complexity. It is compatible with various port spreaders and has strong engineering practicality. Multiple cameras provide full-view coverage of the keyhole, avoiding single-view obstruction and lighting interference, while accurately calculating translational and rotational deviations, greatly improving alignment robustness and accuracy. Dynamic time difference compensation effectively eliminates asynchronous motion errors, adapting to static, low-speed, and high-speed operating conditions, with dynamic alignment accuracy stably controlled at the millimeter level. Multi-dimensional geometric constraints and anomaly classification processing eliminate noise and fill in missing points, avoiding data conflicts and control oscillations, significantly enhancing system stability. Closed-loop control provides continuous iterative correction, balancing operational efficiency and final accuracy to meet the needs of unmanned, high-precision automated container unloading operations in ports.

[0036] In some embodiments, in step S1, the at least four cameras are rigidly mounted at the four corners of the spreader, and each camera captures an image of a lock hole location area on the container.

[0037] In this embodiment, rigid installation means that the camera and the hoist body are firmly fixed by the bracket, with no relative displacement or vibration offset, ensuring the long-term stability of the calibration relationship.

[0038] The four corners of the spreader refer to the four sides of the rectangular frame of the spreader, which correspond one-to-one with the four locking holes of the container.

[0039] The installation method of the cameras directly determines the coverage area of ​​the keyholes, the detection accuracy, and the calibration stability. In this embodiment, at least four cameras are rigidly fixed to the four corners of the spreader, with each camera facing a specific keyhole area on the container, forming a dedicated "one camera, one keyhole" acquisition perspective. Rigid installation avoids camera displacement caused by vibration and shaking during spreader operation, ensuring a fixed relative position between the camera and the spreader's coordinate system. This makes the pixel-space mapping relationship established by S1 effective for a long time, eliminating the need for frequent recalibration. The corner distribution fits the rectangular layout of the container's keyholes, achieving comprehensive coverage without blind spots and providing a reliable hardware foundation for subsequent accurate detection and coordinate transformation.

[0040] The above solution focuses on a single keyhole with a dedicated one-to-one perspective, reducing image interference from multiple keyholes and improving keyhole detection accuracy and positioning precision. Rigid installation eliminates relative displacement, reducing image blurring and coordinate transformation errors caused by vibration and shaking, thus enhancing the system's long-term stability. Corner distribution adapts to standard container sizes, providing full-view coverage without blind spots, offering strong versatility and suitability for various container operating conditions.

[0041] In some embodiments, in step S2, the unified clock is a monotonic clock built into the lifting device control system; the timestamp is a unique and monotonically increasing positive real number that satisfies: ; Where j is the camera number, This represents the image data captured by the j-th camera. This represents a single frame image from the j-th camera. Here is the acquisition timestamp corresponding to this frame of image, and i is the time index. It represents the positive real number field.

[0042] In this embodiment, the monotonic clock refers to a high-precision clock built into the lifting device control system, which increments in one direction without jumps or rollbacks and is unaffected by the system's operating status.

[0043] The camera number j is an index that uniquely identifies different cameras (for example, j can be 1 to 4), distinguishing the image data from different sources.

[0044] The time index i is used to identify the order in which images were acquired, corresponding to different acquisition times.

[0045] Image data F j This refers to the combination of a single frame image and its corresponding timestamp, which binds the image content to the time of acquisition.

[0046] The prerequisite for multi-camera data synchronization is a unified, stable, and non-jumping time base. Ordinary clocks are prone to time rollback and jumps, leading to timing chaos and synchronization failure. Therefore, this embodiment uses the monotonic clock built into the hoist control system as the sole time source, ensuring unidirectional time increment and stable reliability. Each camera acquires a single frame image I. j At the same time, a unique, monotonically increasing positive real timestamp t is generated synchronously. i Combined to form image data All camera timestamps originate from the same monotonic clock, ensuring that all image data streams have a completely consistent time base, constructing a unified time domain, and ensuring that image data from different cameras can be accurately correlated and traced in time, thus avoiding asynchronous timing errors at the source.

[0047] The above solution ensures a stable time base without jumps or backtracking through a monotonic clock, avoiding synchronization failures caused by timestamp anomalies and improving time synchronization reliability. The unique and monotonically increasing timestamp accurately marks the acquisition time, enabling time-series correlation and traceability of multi-channel data, providing a reliable basis for time window matching. Pure software time synchronization requires no additional hardware, is low-cost, easy to deploy, and adaptable to various control systems.

[0048] In some embodiments, step S3 includes: The YOLO detection model based on deep learning or the geometric feature detection algorithm based on edge contour and corner point matching is used to detect each frame of the image. The detection confidence scores of the frame containing the bounding rectangle of the keyhole, the center pixel coordinates of the keyhole, and the corresponding interval [0,1] are obtained. Only the detection confidence scores greater than or equal to the preset confidence threshold c are retained. min Valid keyhole detection data; Transform the center pixel coordinates of the keyhole in the valid keyhole detection data to the lifting coordinate system to obtain two-dimensional spatial coordinates (x... j ,y j ), and compared with the timestamp t of the corresponding frame image. i Detection confidence level c j Binding, forming structured keyhole detection data P j =(x j ,y j ,c j ,t i ).

[0049] In some embodiments, the YOLO detection model refers to an efficient deep learning target detection model that balances real-time performance and accuracy, and is suitable for complex lighting and background interference scenarios in ports.

[0050] Geometric feature detection algorithms refer to detection methods based on inherent features such as keyhole edges and corners. They do not require a large amount of training data, have fast detection speed, and are adaptable to simple working conditions.

[0051] Detection confidence level c j This refers to the probability (0-1) that the algorithm determines the detection result to be a real keyhole; the higher the value, the stronger the reliability.

[0052] Structured keyhole inspection data P j It refers to standardized data that integrates spatial location, confidence level, and timestamps to facilitate interaction and processing between modules.

[0053] Keyhole detection is a core step in obtaining spatial location, requiring a balance of accuracy, real-time performance, and reliability, while adapting to the complex working conditions of ports. This embodiment provides two complementary detection algorithms: for complex scenes (strong lighting, shadows, and clutter interference), a YOLO deep learning model is used to accurately output the bounding rectangle of the keyhole, the coordinates of the center pixel, and a confidence score in the 0-1 range; for simpler conditions, a geometric feature algorithm is used to quickly locate the keyhole through edge contour and corner point matching, balancing detection speed. To eliminate low-quality data, only data with a confidence score ≥ c is retained. min Valid results are obtained to avoid invalid data interfering with subsequent processes. The pixel coordinates of valid data are accurately converted into two-dimensional spatial coordinates (x, y, z) in the lifting device coordinate system through the S1 calibration mapping relationship. j ,y j ), and then compared with the corresponding timestamp t i Confidence level cj Binding, forming standardized structured data P j =(x j ,y j ,c j ,t i The format is uniform and the information is complete, making it easy for subsequent synchronization, compensation, and verification modules to directly call it.

[0054] The above solution employs a dual-detection algorithm to adapt to different working conditions, balancing accuracy in complex environments with speed in simple conditions, demonstrating strong versatility and wide adaptability. Confidence-based filtering eliminates low-reliability data, reducing subsequent errors caused by detection noise and improving overall data quality. Precise pixel-to-space conversion and structured output ensure data consistency and interactivity, improving system processing efficiency.

[0055] In some embodiments, in step S4, the synchronization time threshold The settings are dynamically adjusted based on the lifting device's movement speed and the maximum permissible spatial error, satisfying the following relationship: ; in, The maximum permissible spatial error is preset. This refers to the speed of the lifting device.

[0056] In this embodiment, the maximum spatial error E is preset. max This refers to the maximum allowable deviation of the keyhole position (in millimeters), which is determined by the accuracy requirements for the box placement.

[0057] The speed v of the spreader refers to the real-time horizontal speed of the spreader (in m / s), which is collected in real time by an encoder or sensor.

[0058] The synchronization time threshold Δt refers to the maximum allowable time difference (s) calculated dynamically, used to filter synchronized observation data.

[0059] The synchronization time threshold directly determines the strictness of time window matching. A fixed threshold cannot adapt to the dynamic motion conditions of the lifting device: at high speeds, time differences easily translate into large spatial errors, requiring strict limitation; at low speeds, spatial errors have a smaller impact and can be appropriately relaxed to improve data utilization. This embodiment derives the synchronization threshold constraint relationship based on the core motion error formula Δs=v·Δt. E max The maximum acceptable spatial error is preset, where v is the real-time speed of the spreader. A faster speed results in a smaller threshold, strictly controlling the time difference to reduce spatial error; a slower speed results in a larger threshold, improving data matching efficiency. This dynamic setting logic is scientifically sound, ensuring that the spatial error of synchronized data is always controlled within the allowable range under different motion conditions, balancing alignment accuracy and data processing efficiency.

[0060] The above solution adapts to all motion conditions through dynamic thresholds, controlling errors at high speeds and improving efficiency at low speeds, demonstrating strong adaptability and wide coverage of working conditions. Based on the derivation of the motion error formula, the threshold setting is scientifically reasonable, ensuring controllable spatial errors in synchronization data and guaranteeing alignment accuracy. The system automatically calculates the thresholds in real time without manual intervention, exhibiting a high degree of automation and ease of operation.

[0061] In some embodiments, the calculation formula for the time difference compensation in step S5 is as follows: ,in, This represents the spatial position of the keyhole in the lifting device coordinate system after time difference compensation. The spatial position of the lock hole in the lifting device coordinate system before time difference compensation. The earliest timestamp in the synchronized observation set. This refers to the speed of the lifting device.

[0062] In this embodiment, the position before compensation This refers to the original spatial location of the keyhole during detection, corresponding to the acquisition time t. i Reference timestamp t ref This refers to the earliest acquisition time in the synchronous observation set, serving as a unified compensation benchmark. Compensated position. This refers to converting the original position to the spatial position after the reference time, eliminating time difference displacement error.

[0063] When the lifting device moves, the position of the keyhole collected at different times will have displacement deviation due to the time difference. Asynchronous data cannot be directly compared and needs to be unified to the same time. This embodiment uses the earliest timestamp t in the synchronous observation set. ref Based on this, the compensation amount is calculated using a uniform motion model: t i For the current data timestamp, (t) i -t ref ) represents the time difference between the current time and the reference time; v is the real-time horizontal velocity of the spreader, v·(t) i -t ref The displacement of the lifting device during the time difference is represented by the formula. From the original position P i Subtracting this displacement, we obtain the reference time t. ref The actual location below When the spreader is stationary, v=0, the compensation is zero, the original position is used directly without additional calculation, it is suitable for all working conditions, and the logic is simple and efficient.

[0064] The above solution precisely compensates for asynchronous displacement errors during motion, unifying multiple data streams to the same moment, significantly improving dynamic alignment accuracy and eliminating millimeter-level errors caused by motion. The compensation formula is simple, computationally efficient, and highly real-time, adaptable to high-speed motion conditions without affecting the closed-loop control response speed. Automatic compensation is unnecessary in static conditions, simplifying logic, reducing computational consumption, and ensuring high system stability.

[0065] In some embodiments, in step S6, the geometric consistency constraint verification of the multi-pore site is performed step by step according to the following priority: First, perform confidence constraint verification, which includes: traversing all time-difference compensated keyhole detection data in the current synchronous observation set, and checking the detection confidence level c corresponding to each keyhole detection data. j Remove c j Below the preset confidence threshold c min Abnormal data should be retained, while valid data with acceptable confidence levels should be preserved. Then, distance constraint verification is performed, specifically including: for any two different cameras j and k retained after passing the confidence constraint verification, the spatial positions of the keyholes are... , Calculate the actual spatial distance between the two in the spreader coordinate system. and according to the ideal position Q of the keyhole j Q k Calculated ideal spatial distance A comparison is performed; if the deviation between the actual spatial distance and the ideal spatial distance exceeds the preset deviation range, the observation data is determined to not meet the distance constraint. Then, the parallelism and symmetry relationships are verified, specifically including: verifying the spatial positions of the four keyholes that have passed the distance constraint verification. , Verify whether the two pairs of opposite sides of the quadrilateral formed by the data satisfy the parallel relationship and whether the two diagonals satisfy the symmetry relationship. If either the parallel relationship or the symmetry relationship is not satisfied, the observation data set is determined to not satisfy the direction constraint. If only a single keyhole detection data set does not satisfy any of the above constraints, the single abnormal data set is removed, and a three-point geometric fitting is performed based on the spatial positions of the keyholes in the remaining three valid keyhole detection data sets to complete the spatial position of the fourth keyhole. If two or more keyhole detection data sets do not satisfy the constraints, the entire observation data set is directly determined to be abnormal and all of them are removed.

[0066] Ideal keyhole position Q j This refers to the coordinates of the pre-set rectangle vertices of the standard lock hole of the container in the spreader coordinate system.

[0067] Actual spatial distance It refers to the straight-line distance between the two compensated keyhole positions, reflecting the actual keyhole spacing.

[0068] Three-point geometric fitting refers to a method that uses the geometric characteristics of a rectangle to deduce the fourth missing position from the three valid keyhole positions.

[0069] Even after compensation, the data may still contain anomalies caused by detection noise and environmental interference. These anomalies need to be removed and missing data filled through multi-dimensional, prioritized geometric constraints to ensure the reliability of the observation data. Validation is performed step-by-step according to the priority order of "confidence level → distance → parallel symmetry." If a prerequisite is not met, the validation of the current data is terminated to reduce invalid calculations: The first step is to perform a confidence level check, traversing the compensated data and removing c... j <c min The process involves three steps: First, initial screening of low-reliability data to identify valid data. Second, distance consistency is verified by calculating the deviation between the actual and ideal distances of any two keyholes, discarding data with excessive deviations to ensure that the keyhole spacing conforms to the standard rectangular dimensions. Third, directional consistency is verified by confirming that the two sets of opposite sides of the quadrilateral formed by the four keyholes are parallel and the diagonals are centrally symmetrical, discarding data with directional distortions to ensure that the keyhole distribution is compliant. Abnormal data is handled in a tiered manner: When a single keyhole is abnormal, the three-point shape-defining property of a rectangle is used to fit and complete the fourth position, avoiding data waste; when two or more keyholes are abnormal, the entire data is deemed invalid and discarded to prevent abnormal data from causing control errors, balancing fault tolerance and security.

[0070] The above scheme employs multi-dimensional priority verification and layer-by-layer filtering to accurately eliminate noise and outlier data, significantly improving the reliability of observation data and reducing alignment bias. Three-point fitting completes missing data at single points, improving data utilization, preventing system interruptions due to single-camera malfunctions, and enhancing fault tolerance. Prioritized execution reduces invalid calculations, resulting in high processing efficiency and adapting to the real-time requirements of closed-loop control.

[0071] In some embodiments, in step S7, the generated control command is filtered before being output; the object of the filtering is the spreader pose adjustment command generated based on translational and rotational deviations, and the filtering uses moving average filtering or Kalman filtering.

[0072] In this embodiment, the pose adjustment command refers to the control command that includes translational deviation and rotational deviation, which is used to drive the lifting device to adjust its position and angle.

[0073] Moving average filtering refers to averaging multiple consecutive frames of instructions to smooth out instantaneous fluctuations. It has a simple algorithm and strong real-time performance.

[0074] Kalman filtering refers to the use of optimal estimation theory for linear systems to suppress random noise, predict state changes, and adapt to dynamic operating conditions.

[0075] The raw control commands obtained from pose calculation are susceptible to detection noise and environmental disturbances, exhibiting instantaneous fluctuations and random jumps. Direct output of these commands can lead to frequent jittering and control oscillations, affecting alignment stability. This embodiment filters and smooths the control commands to suppress noise and eliminate fluctuations: the filtering target is the pose adjustment commands composed of translational and rotational deviations; under static and low-speed conditions, a moving average filter is used, weighted averaging of commands across multiple consecutive frames to quickly smooth instantaneous fluctuations; the algorithm is simple and computationally inexpensive; under dynamic and high-speed conditions, a Kalman filter is used, combining historical and observational data to optimally estimate the true control quantity, suppressing random noise, predicting pose change trends, and improving dynamic control accuracy. The filtered commands are stable and without abrupt changes, avoiding spreader jitter and improving closed-loop control stability.

[0076] The above solution significantly improves alignment stability by smoothing command fluctuations, suppressing random noise, and avoiding spreader jitter and control oscillations. The dual-filter algorithm adapts to different working conditions, balancing simplicity and accuracy, and is highly versatile and widely applicable. The filtering process involves low computational load and high real-time performance, without affecting the closed-loop control response speed, thus meeting real-time control requirements.

[0077] In some embodiments, step S8 further includes a security fault tolerance and bin-based optimization mechanism, specifically including: When the calculated translational and rotational deviations are both less than the preset deceleration threshold, the speed of the spreader is automatically reduced to minimize the spatial deviation caused by time synchronization errors. When any camera experiences an image acquisition interruption, no valid keyhole detection result, or a detection confidence level that remains below the preset safety threshold, the system automatically switches to a three-point geometric fitting mode. It then uses the valid keyhole detection data from the other three cameras to fill in the missing keyhole spatial position and continue to calculate the pose of the lifting device. When the completed keyhole spatial location data has geometric conflicts, and the maximum time difference within the constructed synchronous observation set exceeds the synchronization time threshold, the automatic control process is paused and switched to manual control mode or semi-automatic control mode.

[0078] In some embodiments, the deceleration threshold refers to a preset deviation critical value. When the deviation is less than this value, the near-position box dropping stage is entered, and the speed needs to be reduced to improve accuracy.

[0079] Camera failure refers to a state in which the camera cannot output data normally, such as interruption of data acquisition, lack of effective detection, or persistently low confidence level.

[0080] Geometric relationship conflict means that the position of the keyhole after completion does not satisfy the parallel symmetry property of the rectangle, and the data is deemed invalid.

[0081] The safe time threshold refers to the maximum time difference allowed for a set of synchronous observations. Exceeding the threshold indicates a synchronization failure and a safety risk.

[0082] Automated container lowering operations must balance accuracy, efficiency, and safety. The precision requirements are even higher during the near-positioning phase, and equipment malfunctions and data conflicts necessitate timely error correction or safe switching. This embodiment incorporates a safety error correction and container lowering optimization mechanism, covering three major scenarios: near-positioning accuracy, single-camera error correction, and safe switching in case of malfunctions. When both translational and rotational deviations are less than the deceleration threshold, the near-positioning phase is entered, automatically reducing the spreader's speed to minimize spatial deviations caused by time synchronization errors and improve final docking accuracy. If any camera experiences data acquisition interruption, lack of effective detection, or persistently low confidence, the automatic process is not interrupted. Instead, a three-point geometric fitting mode is automatically switched, utilizing the remaining three cameras' effective data to complete the keyhole position and continue calculating the pose, ensuring operational continuity. When geometric conflicts arise in the completed data, or the time difference between synchronized observation sets exceeds the safe time threshold, an automatic control risk is identified, and the automatic process is immediately paused, switching to manual / semi-automatic mode to prevent equipment or container damage due to misoperation and ensure operational safety.

[0083] The above solution reduces dynamic errors through near-position deceleration, significantly improving the final container placement accuracy and meeting high-precision docking requirements. Automatic fault tolerance in case of single camera failure eliminates the need for downtime maintenance, improving operational continuity and efficiency. Safe switching in abnormal operating conditions mitigates control risks, ensuring equipment and container safety and enhancing system reliability.

[0084] In the second aspect, such as Figure 2 As shown, the present invention provides an automatic box-dropping synchronization control system 10 based on multi-camera visual assistance. The system is used to execute the method described in the first aspect of the present invention, and the system includes: The calibration and initialization module 101 is used to install at least four cameras on the spreader and establish a unified spreader coordinate system; to calibrate each camera, establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters. The image acquisition and time synchronization module 102 is used to create an independent image acquisition thread for each camera. When acquiring each frame of image, it records the time when the frame of image is acquired as a timestamp based on a unified clock, thereby giving a unified time reference to the multi-channel image data stream. The keyhole detection module 103 is used to perform keyhole target detection on each frame of image, obtain the spatial position of the keyhole in the frame of image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; The time window synchronization module 104 is used to set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error. The time window matching is performed on the keyhole detection data collected from different cameras. When the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronization observation set. Otherwise, the keyhole detection data with the earliest timestamp is discarded and the matching is performed again. The motion compensation module 105 is used to obtain the current motion speed of the lifting device when the lifting device is in motion, and to use the earliest timestamp in the synchronous observation set as the reference time to perform time difference compensation on the spatial position of the keyhole corresponding to other timestamps in the lifting device coordinate system according to the current motion speed, so as to eliminate the dynamic error caused by the asynchronous acquisition time of frame images. The geometric consistency verification module 106 is used to perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within a preset deviation range; at the same time, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; and reducing the weight of keyhole detection data with a confidence level lower than a preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and is removed. The pose calculation module 107 is used to calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation about the vertical axis based on the actual position of the lock hole verified by the multi-hole geometric consistency constraint and the ideal position of the lock hole; and to generate control commands for driving the lifting device actuator to perform position adjustment based on the translational deviation and rotational deviation. The closed-loop control module 108 is used to send the control command to the actuator of the spreader, driving the spreader to move towards the target position according to the control command; during the movement of the spreader, the image acquisition and time synchronization module, keyhole detection module, time window synchronization module, motion compensation module, geometric consistency verification module and pose calculation module are repeatedly called to form a closed-loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, and the automatic box dropping is determined to be completed.

[0085] like Figure 3 The diagram shown is an overall structural schematic of an automatic box-dropping synchronous control system based on multi-camera visual assistance, which is an exemplary embodiment of the present invention.

[0086] The spreader body is rigidly equipped with cameras at its four corners: front left, front right, rear left, and rear right. The fields of view of these four cameras correspond to the four keyhole areas on the container, enabling comprehensive observation of the keyholes. Image data collected by each camera is transmitted to the processing unit via an image acquisition module. It is then processed sequentially through a time synchronization module, a keyhole detection module, a data fusion module, a pose calculation module, and a control execution module. Finally, a control signal is output to drive the spreader to adjust its pose, achieving automatic alignment with the container keyholes. This structure, through multi-view collaborative observation and modular data processing, provides the hardware foundation for subsequent software synchronization and high-precision control.

[0087] like Figure 4 As shown in Figure 4, this is a flowchart of an automatic box-dropping synchronization control method based on multi-camera visual assistance according to a second exemplary embodiment of the present invention. The process includes, in sequence: Step 1: Construct a multi-camera system and perform unified spatial calibration, establishing a mapping relationship between pixel coordinates and the rigging coordinate system.

[0088] Specifically, four cameras are rigidly mounted on the spreader structure, each covering one of the four lock hole areas of the container, and a unified spreader coordinate system W is established. Calibration is performed on each camera to establish a mapping relationship between pixels in the image coordinate system and spatial points in the spreader coordinate system. ,in, For pixels in the image coordinate system, A spatial point in the lifting device coordinate system. The mapping function consists of camera intrinsic parameters, extrinsic parameters, and distortion correction parameters; simultaneously, the ideal keyhole position set Q is defined. ,satisfy This provides a unified benchmark for subsequent spatial comparisons.

[0089] Step 2: Multi-threaded image acquisition and unified timestamp generation, assigning a unified time reference to multiple images.

[0090] Specifically, after the system is built and calibrated, a multi-threaded image acquisition and unified timestamp generation mechanism is implemented, with an independent acquisition thread built for each camera to acquire image frames in real time. At each frame acquisition time, a unified timestamp is generated based on the host monotonic clock. to form image data ,in All timestamps originate from the same time base and satisfy the monotonically increasing property, thereby constructing a unified time domain representation for multiple cameras to ensure that multiple image data have a consistent time series base.

[0091] Step 3: Detect and represent the keyhole target spatially, generating structured data containing location, confidence level, and timestamp.

[0092] Further processing of each frame image I j Perform keyhole target detection and localization to obtain the spatial position of the keyhole in the lifting device coordinate system, and generate detection data P. j =(x j ,y j ,c j ,t i ); where, (x j ,y j )∈W represents the position of the keyhole in the lifting coordinate system. To test the confidence level, t i For the corresponding timestamp, define the detection set. This completes the association between the keyhole spatial information and the temporal information.

[0093] Step 4: Based on time windows, multi-camera synchronous fusion is used to construct a synchronous observation set that satisfies time consistency.

[0094] Specifically, a multi-camera synchronous fusion mechanism based on a time window is implemented for the multi-keyhole detection results, and the maximum difference in timestamps of all detection data is calculated. And set a synchronization threshold. When satisfied At that time, the set of detection data was determined to be a synchronous observation set. If the conditions are not met, the detection data corresponding to the earliest timestamp is discarded and a new synchronization set is constructed to ensure the temporal consistency of the observation data participating in subsequent calculations.

[0095] Step 5: Based on the time difference compensation of the motion model, eliminate asynchronous spatial errors caused by the movement of the lifting device.

[0096] Specifically, when the spreader is in motion, a velocity model consisting of a horizontal component is introduced. Time difference compensation based on motion model is performed on the keyhole detection results at different timestamps. The compensation formula is as follows: ;in, This represents the spatial position of the keyhole in the lifting device coordinate system after time difference compensation. The spatial position of the lock hole in the lifting device coordinate system before time difference compensation. The earliest timestamp in the synchronous observation set is used to obtain the compensated detection set. This unifies asynchronous observation data to the same reference time.

[0097] Step 6: Geometric consistency constraints and anomaly removal at multiple pore sites to ensure the reliability of observation data.

[0098] Specifically, a multi-hole geometric consistency constraint and anomaly removal mechanism are applied to the compensated detection set P' to verify the data validity from multiple dimensions: one is the structural constraint, where the spatial distance between any two keyholes after compensation satisfies Secondly, there is a directional constraint, and the keyhole distribution boundary satisfies... Thirdly, there is the confidence constraint, where the detection confidence satisfies... If any constraint is not met, the corresponding detection data will be downweighted or removed. If necessary, the current control cycle will be terminated to ensure the spatial consistency and reliability of the observation data.

[0099] Step 7: Calculate the position and posture deviation of the spreader and generate control quantities. Calculate the translation and rotation deviations and generate control commands.

[0100] After completing the geometric consistency verification, the spreader position deviation calculation and control quantity generation are performed, and the lock hole deviation is defined. The overall positional error of the lifting device is then calculated based on this; the translational error is calculated using the following formulas: ; ; Rotational error is estimated using keyhole geometry or least squares. Thus, the control vector is constructed. ; Filtering is performed before the control vector output; the filtering formula is as follows: , thus obtaining the smoothed control quantity.

[0101] Step 8: Closed-loop control execution and safety constraints, drive the spreader to adjust and iterate cyclically until alignment is completed.

[0102] Specifically, the filtered control quantity U is sent to the spreader control system to perform automatic alignment and establish a closed-loop control link P→U→control execution→new observation; at the same time, multiple safety constraints are introduced: the time constraint requires the synchronization time difference to meet the following requirements. Otherwise, control is suspended; stability constraints require the keyhole deviation to meet the following requirements. The alignment is considered complete when the conditions are met. Anomaly handling mechanisms include switching between manual and semi-automatic modes in case of data conflicts, using multi-point fitting as a substitute when a single camera fails, and automatic speed reduction control during high-speed phases, thus ensuring the safety and reliability of the control process. This method achieves spatial consistency of multi-view keyhole detection data at a unified time by constructing a software synchronization mechanism with a unified timestamp, time window constraints, multi-camera observation fusion, and time compensation based on a motion model. Combined with geometric constraints and a pose calculation model, it achieves real-time high-precision alignment control of the spreader with the container keyholes, thereby realizing automatic container lowering operations without the need for hardware synchronization.

[0103] like Figure 5The diagram illustrates a multi-camera time synchronization and time window constraint mechanism according to an exemplary embodiment of the present invention. Image frames acquired by the four cameras correspond to different timestamps t1, t2, t3, and t4. The maximum difference among all timestamps is calculated. and synchronize with the preset threshold. Comparison. When When the data set satisfies time consistency, a synchronized observation set can be directly constructed; when If the frame timestamp is too high, the earliest frame is discarded and the matching is repeated until a set of synchronous observations that meets the requirements is constructed. This mechanism ensures that the data from multiple cameras participating in subsequent calculations correspond to the same moment, eliminating timing errors caused by asynchronous acquisition at the source.

[0104] like Figure 6 The diagram shown illustrates the geometric structure and pose deviation calculation principle of the container lock holes according to an exemplary embodiment of the present invention. The ideal positions Q1, Q2, Q3, and Q4 of the four lock holes form a standard rectangular structure, while the actual detected lock hole positions are P1, P2, P3, and P4. The translational deviation of the spreader is determined by the average deviation between the actual and ideal positions along the X and Y axes, respectively. and Rotational deviation Based on the relative positional relationship between the keyholes, it can be approximately expressed as the ratio of the height difference between the left and right keyholes to the width of the keyhole.

[0105] like Figure 7 The diagram shown is a schematic representation of the closed-loop control process for automatic container dropping according to an exemplary embodiment of the present invention, illustrating the closed-loop execution logic of "detection-calculation-control-feedback". Figure 7 As shown, the system first acquires keyhole position data through a vision detection module, and then the pose estimation module calculates the position and attitude deviations of the lifting device, generating a data set including translational deviations. , and rotational deviation The control signal is used to adjust the position and attitude of the spreader. Simultaneously, the adaptive speed adjustment module dynamically reduces the spreader's speed based on the error magnitude to improve alignment accuracy. The adjusted spreader posture is then fed back into the vision detection module via a feedback loop, forming a continuously iterative closed-loop control process. As the cycle continues, the posture error gradually converges. , and When all values ​​are less than the preset threshold, the alignment is considered complete, and the box dropping operation is performed.

[0106] The following specific implementation case will illustrate this method in detail: During the system calibration phase, the coordinate system of the lifting device is set to millimeters, and the ideal positions of the four keyholes are Q1=(-1200,800), Q2=(1200,800), Q3=(-1200,-800), and Q4=(1200,-800). The pixel coordinates detected by a certain camera are (640,360), which are converted to the coordinate system of the lifting device after calibration and mapping to obtain the spatial coordinates (-1180,790), thus completing the conversion from pixel to spatial coordinates.

[0107] During the timestamp generation stage, the acquisition times of the four cameras are t1=10.005s, t2=10.011s, t3=10.008s, and t4=10.013s, respectively. The time difference Δt is calculated as 10.013-10.005=0.008s=8ms. The synchronization threshold ε is set to 10ms, which satisfies the synchronization condition.

[0108] During the keyhole detection phase, the detection results from the four cameras were P1=(-1180,790,0.95), P2=(1215,810,0.93), P3=(-1195,-820,0.96), and P4=(1220,-790,0.92), respectively, and all detection confidence levels met the requirements.

[0109] During the time window fusion phase, the timestamp set is {10.005, 10.011, 10.008, 10.013}, with a time difference of 8ms < 10ms. A synchronous observation set P is then constructed. sync ={P1,P2,P3,P4}.

[0110] During the motion compensation phase, the speed of the spreader is set to v. x =0.3m / s, v y =0.1m / s, reference time t ref =10.005s, compensate for the detection data corresponding to t4=10.013s, and calculate x4′=1220-0.3×0.008=1217.6, y4′=-790-0.1×0.008=-790.8. Similarly, complete the compensation for the remaining data, and get P1′=(-1180,790), P2′=(1212.2,808.2), P3′=(-1192.6,-819.2), P4=(1217.6,-790.8).

[0111] During the consistency constraint verification phase, the actual distance between the left and right keyholes was calculated to be 2392.2 mm, which deviates from the ideal distance of 2400 mm by 7.8 mm; the actual distance between the upper and lower keyholes was 1609.2 mm, which deviates from the ideal distance of 1600 mm by 9.2 mm. Both deviations are less than the preset range of 20 mm, indicating that the geometric relationship is valid.

[0112] In the pose deviation calculation stage, the keyhole deviations are e1=(20,-10), e2=(12.2,8.2), e3=(7.4,-19.2), and e4=(17.6,9.2). The calculated translation errors are ΔX=14.3mm and ΔY=-2mm, and the rotation error is Δθ≈-0.24∘. The control vector U t =(14.3mm,-2mm,-0.24°); Set the filter coefficient α=0.7, the control quantity of the previous cycle is (10,0,-0.2°), and after filtering, we get U=(12.99mm,-1.4mm,-0.228°).

[0113] During the closed-loop control phase, after the control is executed, the deviation converges to ΔX=3mm, ΔY=1mm, and Δθ=0.05°, satisfying the box-dropping accuracy thresholds of |ΔX|<5mm, |ΔY|<5mm, and |Δθ|<0.1°, and the automatic box-dropping is determined to be completed.

[0114] Additional error verification shows that when the lifting speed v=0.5m / s and the synchronization time difference Δt=10ms, the spatial error Δs=v·Δt=5mm; thus, it can be seen that when the synchronization time error is controlled within 10ms, the present invention can stably control the spatial error at the millimeter level, meeting the requirements of high-precision automatic box dropping operation.

[0115] Finally, it should be noted that although the above embodiments have been described in the description and drawings of this invention, this should not limit the scope of patent protection of this invention. Any technical solutions that are based on the essential concept of this invention, utilize the content described in the description and drawings of this invention to make equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this invention.

Claims

1. A method for automatic box dropping synchronization control based on multi-camera vision assistance, characterized in that, The method includes: S1: Install at least four cameras on the spreader and establish a unified spreader coordinate system; calibrate each camera to establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters; S2: Create an independent image acquisition thread for each camera. When acquiring each frame of image, record the time when the frame of image is acquired as a timestamp based on a unified clock, thereby giving a unified time reference to the multiple image data streams. S3: Perform keyhole target detection on each frame image, obtain the spatial position of the keyhole in the frame image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; S4: Set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error; perform time window matching on keyhole detection data collected from different cameras; when the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronous observation set; otherwise, discard the keyhole detection data with the earliest timestamp and rematch. S5: When the lifting device is in motion, obtain the current speed of the lifting device, and use the earliest timestamp in the synchronous observation set as the reference time. Based on the current speed, perform time difference compensation on the spatial position of the keyholes corresponding to other timestamps in the lifting device coordinate system to eliminate dynamic errors caused by asynchronous frame image acquisition time. S6: Perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within the preset deviation range; at the same time, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; and reducing the weight of keyhole detection data with a confidence level lower than the preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and is removed. S7: Based on the actual position of the lock hole verified by the geometric consistency constraint of the multi-hole position and the ideal position of the lock hole, calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation around the vertical axis; generate control commands for driving the lifting device actuator to perform position adjustment according to the translational deviation and rotational deviation. S8: Send the control command to the actuator of the spreader to drive the spreader to move to the target position according to the control command; during the movement of the spreader, repeat steps S2 to S7 to form a closed loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, and determine that the automatic box dropping is completed.

2. The automatic box-dropping synchronization control method based on multi-camera visual assistance according to claim 1, characterized in that, In step S1, the at least four cameras are rigidly mounted at the four corners of the spreader, and each camera captures an image of a lock hole location area on the container.

3. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 1, characterized in that, In step S2, the unified clock is the monotonic clock built into the lifting device control system; the timestamp is a unique and monotonically increasing positive real number, satisfying: ； Where j is the camera number, This represents the image data captured by the j-th camera. This represents a single frame image from the j-th camera. Here is the acquisition timestamp corresponding to this frame of image, and i is the time index. It represents the positive real number field.

4. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 3, characterized in that, Step S3 includes: The YOLO detection model based on deep learning or the geometric feature detection algorithm based on edge contour and corner point matching is used to detect each frame of the image. The detection confidence scores of the frame containing the bounding rectangle of the keyhole, the center pixel coordinates of the keyhole, and the corresponding interval [0,1] are obtained. Only the detection confidence scores greater than or equal to the preset confidence threshold c are retained. min Valid keyhole detection data; Transform the center pixel coordinates of the keyhole in the valid keyhole detection data to the lifting coordinate system to obtain two-dimensional spatial coordinates (x... j ,y j ), and compared with the timestamp t of the corresponding frame image. i Detection confidence level c j Binding, forming structured keyhole detection data P j =(x j ,y j ,c j ,t i ).

5. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 1, characterized in that, In step S4, the synchronization time threshold The settings are dynamically adjusted based on the lifting device's movement speed and the maximum permissible spatial error, satisfying the following relationship: ； in, The maximum permissible spatial error is preset. This refers to the speed of the lifting device.

6. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 1, characterized in that, In step S5, the calculation formula for the time difference compensation is as follows: ,in, This represents the spatial position of the keyhole in the lifting device coordinate system after time difference compensation. The spatial position of the lock hole in the lifting device coordinate system before time difference compensation. The earliest timestamp in the synchronized observation set. This refers to the speed of the lifting device.

7. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 6, characterized in that, In step S6, the verification of the geometric consistency constraints of the multi-pore site is performed step by step according to the following priority: First, perform confidence constraint verification, which includes: traversing all time-difference compensated keyhole detection data in the current synchronous observation set, and checking the detection confidence level c corresponding to each keyhole detection data. j Remove c j Below the preset confidence threshold c min Abnormal data should be retained, while valid data with acceptable confidence levels should be preserved. Then, distance constraint verification is performed, specifically including: for any two different cameras j and k retained after passing the confidence constraint verification, the spatial positions of the keyholes are... , Calculate the actual spatial distance between the two in the spreader coordinate system. and according to the ideal position Q of the keyhole j Q k Calculated ideal spatial distance A comparison is performed; if the deviation between the actual spatial distance and the ideal spatial distance exceeds the preset deviation range, the observation data is determined to not meet the distance constraint. Then, the parallelism and symmetry relationships are verified, specifically including: verifying the spatial positions of the four keyholes that have passed the distance constraint verification. , Verify whether the two pairs of opposite sides of the quadrilateral formed by the data satisfy the parallel relationship and whether the two diagonals satisfy the symmetry relationship. If either the parallel relationship or the symmetry relationship is not satisfied, the observation data set is determined to not satisfy the direction constraint. If only a single keyhole detection data set does not satisfy any of the above constraints, the single abnormal data set is removed, and a three-point geometric fitting is performed based on the spatial positions of the keyholes in the remaining three valid keyhole detection data sets to complete the spatial position of the fourth keyhole. If two or more keyhole detection data sets do not satisfy the constraints, the entire observation data set is directly determined to be abnormal and all of them are removed.

8. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 7, characterized in that, In step S7, the generated control commands are filtered before being output; the object of the filtering is the spreader posture adjustment command generated based on translational and rotational deviations, and the filtering process uses moving average filtering or Kalman filtering.

9. The automatic box-dropping synchronization control method based on multi-camera vision assistance according to claim 7, characterized in that, Step S8 also includes a safety fault tolerance and bin-based optimization mechanism, specifically including: When the calculated translational and rotational deviations are both less than the preset deceleration threshold, the speed of the spreader is automatically reduced to minimize the spatial deviation caused by time synchronization errors. When any camera experiences an image acquisition interruption, no valid keyhole detection result, or a detection confidence level that remains below the preset safety threshold, the system automatically switches to a three-point geometric fitting mode. It then uses the valid keyhole detection data from the other three cameras to fill in the missing keyhole spatial position and continue to calculate the pose of the lifting device. When the completed keyhole spatial location data has geometric conflicts, and the maximum time difference within the constructed synchronous observation set exceeds the synchronization time threshold, the automatic control process is paused and switched to manual control mode or semi-automatic control mode.

10. An automatic box-dropping synchronous control system based on multi-camera vision assistance, characterized in that, The system is configured to perform the method as described in any one of claims 1-9, the system comprising: The calibration and initialization module is used to install at least four cameras on the spreader and establish a unified spreader coordinate system; to calibrate each camera, establish a spatial mapping relationship from image pixel coordinates to the spreader coordinate system, and preset the ideal positions of at least four lock holes on the container in the spreader coordinate system as system initialization parameters; The image acquisition and time synchronization module is used to create an independent image acquisition thread for each camera. When acquiring each frame of image, it records the time when the frame of image was acquired as a timestamp based on a unified clock, thereby giving a unified time reference to multiple image data streams. The keyhole detection module is used to detect keyhole targets in each frame of image, obtain the spatial position of the keyhole in the frame of image under the coordinate system of the lifting device and its detection confidence, and associate the spatial position of the keyhole with the timestamp of the corresponding frame image to form keyhole detection data containing keyhole position, detection confidence and timestamp; The time window synchronization module is used to set a synchronization time threshold, which is dynamically determined based on the speed of the lifting device and the maximum allowable spatial error. The module performs time window matching on keyhole detection data collected from different cameras. When the absolute value of the difference between the timestamps of any two keyhole detection data is less than or equal to the synchronization time threshold, these keyhole detection data are included in the same synchronization observation set. Otherwise, the keyhole detection data with the earliest timestamp is discarded and the matching is performed again. The motion compensation module is used to obtain the current speed of the spreader when it is in motion, and to use the earliest timestamp in the synchronous observation set as a reference time to compensate for the time difference of the spatial position of the keyholes corresponding to other timestamps in the spreader coordinate system according to the current speed, so as to eliminate the dynamic error caused by the asynchronous acquisition time of frame images. The geometric consistency verification module is used to perform multi-hole geometric consistency constraint verification on the compensated synchronous observation set. The multi-hole geometric consistency constraint verification includes: calculating the actual spatial distance between any two keyholes in the synchronous observation set in the lifting coordinate system, and calculating the deviation between the actual spatial distance and the ideal spatial distance calculated based on the ideal position of the keyhole, and determining whether the deviation is within a preset deviation range; simultaneously, verifying whether the quadrilateral formed by the actual positions of at least four keyholes in the lifting coordinate system satisfies the preset parallelism and symmetry relationship; furthermore, reducing the weight of keyhole detection data with a confidence level lower than a preset confidence threshold; if any constraint is not satisfied, the group of observation data is determined to be abnormal and discarded. The pose calculation module is used to calculate the translational deviation of the lifting device in the horizontal direction and the rotational deviation about the vertical axis based on the actual position of the key hole verified by the multi-hole geometric consistency constraint and the ideal position of the key hole; and to generate control commands for driving the lifting device actuator to perform position adjustment based on the translational deviation and rotational deviation. The closed-loop control module is used to send the control commands to the actuator of the spreader, driving the spreader to move towards the target position according to the control commands. During the movement of the spreader, the image acquisition and time synchronization module, keyhole detection module, time window synchronization module, motion compensation module, geometric consistency verification module, and pose calculation module are repeatedly called to form a closed-loop control until the detected translational deviation and rotational deviation are both less than the preset box dropping accuracy threshold, at which point the automatic box dropping is determined to be completed.

Images