Port container loading and unloading positioning method, system, equipment and medium

Abstract

The invention provides a port container loading and unloading positioning method, system and equipment and a medium, and belongs to the technical field of port automation and machine vision positioning. The method comprises the following steps: acquiring a container surface image and synchronous sling motion data; the jitter amplitude is evaluated, the image is combined to analyze the illumination condition, and the multispectral image is adaptively selected or fused to generate an anti-interference image; angular point coordinates are extracted, the angular point coordinates and motion data are input into a vision-inertia tight coupling optimization model together for state estimation, and the real-time pose is solved; driving a lifting appliance-load dynamic model by using the motion data, and predicting pose deviation generated by inertia and swinging in the future; and the feedback control quantity based on the real-time pose and the feed-forward compensation quantity based on the prediction deviation are fused, and an accurate control instruction for driving the lifting appliance is generated. The problems of image quality degradation and feature loss caused by strong light, weak light and sling shaking are solved, and high-precision and high-robustness container positioning and alignment in a complex dynamic environment are achieved.

Description

Technical Field

[0001] This invention relates to the field of port automation and machine vision positioning technology, and in particular to a port container loading and unloading positioning method, system, equipment and medium. Background Technology

[0002] Automation and intelligentization of port container handling operations are core directions for improving logistics efficiency, reducing operating costs, and ensuring operational safety. Among these, achieving rapid and high-precision alignment between spreader and container is a key technological bottleneck for automated handling systems. Traditional positioning methods, such as manual visual operation, laser ranging, or GPS, all have significant limitations under the complex realities of port conditions: manual operation is inefficient and unsafe; laser ranging is susceptible to interference from common port phenomena like dust and rain; and GPS lacks sufficient positioning accuracy in heavily obstructed areas such as container yards.

[0003] In recent years, machine vision-based positioning technology has become a research and application hotspot due to its advantages such as rich information, non-contact operation, and relatively low cost. Existing visual positioning solutions mainly focus on the identification and pose calculation of container top corner fittings, but they generally fail to effectively address the core challenges of real-world port operations, including dynamic interference and lighting changes. Existing technical solutions mainly have the following shortcomings: First, one type of approach relies on a single visible light camera and static image processing algorithms. These approaches detect corner points using models and perform completion and verification when corner points are missing. However, their algorithmic flow is based on the assumption that the images are relatively clear and stable. When the lifting equipment vibrates, the acquired images suffer from severe motion blur, leading to feature extraction failure. Furthermore, under conditions such as strong light overexposure or low-light nighttime conditions, image information based solely on the visible spectrum will be severely degraded or even lost, reducing the robustness of the entire recognition chain. This type of approach cannot overcome physical environmental interference at the source of perception.

[0004] Secondly, another approach attempts to improve robustness by incorporating data fusion from multiple sensors, such as combining vision and inertial measurement units (IMUs). However, these approaches often employ loosely coupled or post-fusion architectures, where the visual positioning results are output first, followed by filtering and fusion with inertial data. This fusion method fails to fundamentally establish a unified spatiotemporal reference and cannot model and compensate for jitter at the moment of image acquisition. Furthermore, for illumination issues, a single image enhancement algorithm is typically used, lacking a mechanism for adaptively selecting the sensing modality based on environmental conditions, thus limiting the improvement effect under extreme lighting conditions.

[0005] Another approach focuses on achieving global positioning by deploying a large number of visual base stations or laying out high-precision external infrastructure such as QR code arrays. While these solutions can provide absolute coordinates, they are expensive, require extensive retrofitting, are difficult to maintain, and are susceptible to interference from the complex electromagnetic environment of ports, making large-scale deployment at existing terminals difficult.

[0006] In summary, existing technologies either lack robustness to environmental dynamics and changes in lighting, or are complex and costly. Port container handling operations urgently require an innovative method to achieve real-time, high-precision, and highly reliable visual positioning in complex dynamic environments with strong vibrations and changing lighting. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, equipment and medium for positioning containers in port loading and unloading, so as to solve the problems of image quality degradation, difficulty in feature extraction, low positioning accuracy and lag caused by drastic changes in lighting and high-frequency vibration of spreaders in the complex and dynamic environment of ports.

[0008] To achieve the above objectives, the present invention provides a port container loading and unloading positioning method, comprising the following steps: Step S1: Acquire an image of the container surface, and acquire the angular velocity sequence and linear acceleration sequence measured by the inertial measurement unit within the exposure time window of the container surface image acquisition, as the spreader motion data; Step S2: Calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and evaluate the ambient light level based on the grayscale features of the container surface image; based on the evaluation results of the equivalent motion amplitude and ambient light level, perform adaptive enhancement processing on the container surface image and output an anti-interference image. Step S3: Extract the image coordinates of the container corner points from the anti-interference image; input the image coordinates of the corner points and the corresponding spreader motion data into the vision-inertial tightly coupled optimization model for joint state estimation, and solve for the real-time pose of the spreader relative to the container; Step S4: Input the spreader motion data into the spreader-load coupled dynamics model to predict the end pose disturbance deviation of the spreader due to inertial motion and load swing in the next control cycle; Step S5: Generate feedback control quantity based on the difference between real-time pose and target pose, and generate feedforward compensation quantity based on end pose disturbance deviation; fuse feedback control quantity and feedforward compensation quantity to generate final control command to drive spreader motion.

[0009] Preferably, in step S2, the method for calculating the equivalent motion amplitude is as follows: for the exposure time window angular velocity sequence within Integrating, we obtain the equivalent angular displacement. : ; Linear acceleration sequence By performing double integration, the equivalent linear displacement is obtained. : ; in, The starting moment of the exposure. The moment when the exposure ends; The equivalent motion amplitude is determined by the equivalent angular displacement. model and equivalent linear displacement model Characterization.

[0010] Preferably, in step S2, based on the evaluation results of equivalent motion amplitude and ambient light level, adaptive enhancement processing is performed on the container surface image to output an anti-interference image, specifically including: Step S21, if If , then the current image is determined to have significant blurring due to angular motion; where The preset angular displacement ambiguity threshold; Step S22: Calculate the grayscale values ​​in the container surface image that are greater than a high threshold. pixel ratio and grayscale values ​​less than the low threshold pixel ratio ; like If so, it is determined to be a strong light overexposure environment, among which This is the preset threshold for the proportion of overexposed pixels; like If so, it is determined to be a low-light, underexposed environment. This is the preset threshold for the proportion of underexposed pixels; Otherwise, it is considered a normal lighting environment; Step S23: Image processing strategy execution: A) If significant ambiguity is determined, the infrared thermal image acquired synchronously with the container surface image shall be selected as the main image for processing. B) If the environment is determined to be overexposed by strong light, the container surface image is compressed with high dynamic range, its edge information is extracted, and it is fused with the contour information of the infrared thermal image to generate an anti-interference image. C) If the environment is determined to be weak light and underexposed, the near-infrared supplementary light source is turned on and multi-frame temporal noise reduction is performed on the container surface image to generate an anti-interference image. D) If the lighting is normal and there is no significant blur, the image of the container surface is enhanced with contrast and used as an anti-interference image.

[0011] Preferably, in step S3, the vision-inertial tightly coupled optimization model is a nonlinear optimization model based on a sliding window, and its solution process includes: defining the sliding window containing... The state vector at each time step is ,in For the first The status of the lifting gear at any given moment. For position vectors, For velocity vectors, For attitude quaternions, and These are the bias vectors for the accelerometer and gyroscope, respectively. For the first Inverse depth parameters of each spatial corner point; Construct a global cost function that includes all visual observations and inertial measurement constraints. : ; in, The set of all corner observations. Corner point Reprojection error, For the inertial measurement unit at adjacent time intervals arrive The set of raw measurement data collected between them Based on Calculated inertial pre-integration error, Huber robust loss function and These are the covariance matrices corresponding to the errors; Minimize using an iterative optimization algorithm To obtain the optimal state estimate The latest status within the window Position vector in With attitude quaternions It constitutes the real-time pose.

[0012] Preferably, in step S4, the spreader-load coupled dynamics model simplifies the spreader and container load into a three-dimensional spatial pendulum system, and its prediction process includes: Step S41: Let the displacement of the lifting device along the direction of the trolley in the horizontal plane be... The displacement along the direction of the trolley is The swing angle of the load relative to the spreader is ,in The swing angle along the direction of the spreader trolley. Let be the swing angle along the direction of the vehicle; the system's state vector is . ,in For transpose; Step S42: Calculate the current linear acceleration from the spreader motion data. The angular velocity is used as an external disturbance input in the hoist-load coupled dynamics model, where... and These are the instantaneous linear acceleration components along the direction of the main vehicle and the direction of the trolley, extracted from the data of the inertial measurement unit at the current moment. Step S43: Use the estimated system state at the current moment as the initial value. The state equations of the coupled dynamic model of the spreader-load are numerically integrated using the fourth-order Runge-Kutta method to advance one control cycle. To obtain the predicted state ; Let the swing angle in the predicted state be . and ,Right now , ; Step S44: Based on the predicted swing angle and and the geometric connection length between the lifting device and the load Calculate the expected positional deviation of the end effector of the spreader in the horizontal plane: ; in, and These are the expected positional deviations along the direction of the main vehicle and the direction of the auxiliary vehicle, respectively. This refers to the end-effector pose disturbance deviation.

[0013] Preferably, in step S5, the final control command for driving the spreader's movement is generated, specifically as follows: Step S51: Set the real-time pose to be determined by the position. and attitude quaternions The target pose is composed of position and attitude quaternions The composition, then the positional error Attitude error Quaternion operations Convert to rotation vector ; Step S52: Calculate the feedback control quantity: ; in, , , , This is the gain matrix for the corresponding dimension. The differential of position error; Step S53: Calculate the feedforward compensation amount: ; in, This is the feedforward gain matrix; Step S54: Generate final instructions: ; in, This is the final control command.

[0014] Preferably, in step S23, when the environment is determined to be an overexposed strong light environment, the fusion process is specifically as follows: Let the visible light image after high dynamic range compression be... The edge map extracted by the Canny operator is Let the synchronized infrared thermal image be... The gradient magnitude map calculated by the Sobel operator is as follows: ; Generate fusion weight graph The position of each pixel weight ,in The preset saturation clipping value; Anti-interference image Generated by the following formula: ; Normalize the results.

[0015] The present invention also provides a port container loading and unloading positioning method system, comprising: Image acquisition unit, used to acquire images of the container surface; An inertial measurement unit is used to measure the angular velocity sequence and linear acceleration sequence within the exposure time window of acquiring images of the container surface, as data for the spreader motion. The adaptive enhancement module is connected to the image acquisition unit and the inertial measurement unit. It is used to calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and to evaluate the ambient light level based on the grayscale characteristics of the container surface image. Then, based on the evaluation results of the equivalent motion amplitude and the ambient light level, it performs adaptive enhancement processing on the container surface image and outputs an anti-interference image. The tightly coupled pose calculation module communicates with the adaptive enhancement module and the inertial measurement unit. It is used to extract the image coordinates of the container corner points from the anti-interference image, and input the image coordinates of the corner points and the corresponding spreader motion data into the vision-inertial tightly coupled optimization model for joint state estimation to obtain the real-time pose of the spreader relative to the container. The disturbance prediction module, which communicates with the inertial measurement unit, is used to input the spreader motion data into the spreader-load coupled dynamics model and predict the end pose disturbance deviation of the spreader due to inertial motion and load swing in a future control cycle. The composite control module, which is connected to the tightly coupled pose calculation module and the disturbance prediction module, is used to generate feedback control quantity based on the difference between the real-time pose and the target pose, and to generate feedforward compensation quantity based on the end pose disturbance deviation. The feedback control quantity and the feedforward compensation quantity are fused to generate the final control command to drive the spreader to move. The spreader drive actuator unit communicates with the composite control module to receive and execute the final control commands, driving the spreader to complete container loading, unloading and positioning operations.

[0016] The present invention also provides a computer device, including: a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described port container loading and unloading positioning method.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described port container loading and unloading positioning method.

[0018] Therefore, the present invention employs the above-mentioned port container loading and unloading positioning method, system, equipment, and medium, and the beneficial technical effects are as follows: (1) This invention combines jitter assessment based on inertial data with illumination assessment based on multispectral images, and executes an adaptive image enhancement strategy accordingly. This allows the system to effectively suppress the interference of strong light, weak light, and lifting device motion blur on visual perception from the source. Furthermore, by performing state-level tight coupling optimization between the enhanced visual features and high-frequency inertial data, direct compensation for dynamic jitter and high-precision pose calculation are achieved. This collaborative processing from front-end perception to back-end calculation enables the system to maintain stable and reliable positioning capabilities even under conditions of drastic changes in illumination and continuous mechanical vibration.

[0019] (2) While providing high-precision real-time pose, this invention uses the same inertial data source to drive the spreader-load dynamics model and predict the pose offset in the future instant. Subsequently, this predicted offset is used as a feedforward signal and fused with the feedback control quantity based on the real-time pose error to form a composite control command with anticipatory compensation capability. This deep synergy between perception, prediction and control enables the spreader to actively and smoothly counteract the swaying trend, significantly shortening the alignment adjustment time and improving the smoothness and final accuracy of loading and unloading operations.

[0020] (3) This invention relies on integrable sensors such as multispectral vision and inertial measurement units, without requiring large-scale modifications to port infrastructure. Through innovations at the algorithm level, such as multimodal data adaptive fusion, tightly coupled state estimation, and model predictive control, it fully explores and synergistically utilizes the data potential of existing sensors, effectively reducing reliance on expensive dedicated hardware or external auxiliary facilities by minimizing the complexity of software algorithms. This makes the high-precision positioning solution easy to deploy and upgrade on existing port loading and unloading equipment, and has significant engineering application value. Attached Figure Description

[0021] Figure 1 This is a flowchart of a port container loading and unloading positioning method according to the present invention; Figure 2 The curve showing the change of positioning error over time under strong light overexposure conditions; Figure 3 The curve of positioning error changing over time under low light and underexposure conditions; Figure 4 The curve shows the change in positioning error over time under normal lighting and vibration conditions. Figure 5 The steady-state positioning errors of the three methods under three working conditions are compared. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0024] Example 1 In this embodiment, the hardware system used includes: Image acquisition unit: Employs a multispectral vision system, including a visible light camera (resolution...). (30fps) and an infrared thermal imager (resolution) (Frame rate 25fps), both are rigidly installed at the lower end of the spreader, with the optical axis parallel and downward aligned with the top surface of the container.

[0025] Inertial Measurement Unit (IMU): A six-axis MEMS inertial measurement unit (gyroscope range ±2000° / s, accelerometer range ±16g, output frequency 200Hz) is used, which is installed near the camera and has a time synchronization accuracy with the camera better than 1ms.

[0026] Near-infrared supplementary light source: wavelength 850nm, invisible light supplementary light, installed on both sides of the camera.

[0027] Control unit: Industrial-grade industrial computer (Intel i7, 16GB RAM, GPU NVIDIA Jetson AGXXavier), running the algorithm proposed in this invention.

[0028] The lifting device drive system consists of a servo motor-driven trolley, crane, and lifting mechanism that receives control commands and executes positioning actions.

[0029] like Figure 1 As shown, a method for positioning and unloading containers in a port includes the following steps: Step S1: Information collection execution.

[0030] After the quay crane operation system is started, the image acquisition unit and IMU start synchronously. The visible light camera and infrared thermal imager continuously acquire images of the container surface at the set frame rate, and the IMU synchronously acquires angular velocity sequence and linear acceleration sequence.

[0031] Outlier removal from raw IMU data (using...) The guidelines convert the image data format (from RAW format to 8-bit grayscale) and achieve a one-to-one correspondence between the image and the IMU data through timestamp alignment.

[0032] Data Acquisition Results: Under the condition of a spreader moving at a speed of 0.8 m / s, the IMU acquired 30 sets of angular velocity data (range: -0.5~0.3 rad / s) and 30 sets of linear acceleration data (range: -0.8~1.2 m / s) within a single frame exposure time. 2 The visible light image clearly shows the four corner areas of the container's top surface.

[0033] Step S2: Adaptive enhancement processing.

[0034] (1) Calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and evaluate the ambient light level based on the grayscale characteristics of the container surface image.

[0035] Exposure time window angular velocity sequence within Integrating, we obtain the equivalent angular displacement. : ; Linear acceleration sequence By performing double integration, the equivalent linear displacement is obtained. : ; in, The starting moment of the exposure. The moment when the exposure ends; The equivalent motion amplitude is determined by the equivalent angular displacement. model and equivalent linear displacement model Characterization.

[0036] (2) Based on the evaluation results of equivalent motion amplitude and ambient light level, adaptive enhancement processing is performed on the container surface image to output an anti-interference image, specifically including: Step S21, Motion blur determination: If If , then the current image is determined to have significant blurring due to angular motion; where This is the preset angular displacement ambiguity threshold.

[0037] Step S22, Illumination Level Determination: Calculate the grayscale values ​​in the container surface image that are greater than the high threshold. pixel ratio and grayscale values ​​less than the low threshold pixel ratio ; like If so, it is determined to be a strong light overexposure environment, among which This is the preset threshold for the proportion of overexposed pixels; like If so, it is determined to be a low-light, underexposed environment. This is the preset threshold for the proportion of underexposed pixels; Otherwise, it is considered a normal lighting environment.

[0038] Step S23: Image processing strategy execution: A) If significant ambiguity is determined, the infrared thermal image acquired synchronously with the container surface image shall be selected as the primary image for processing.

[0039] B) If the environment is determined to be overexposed by strong light, the container surface image is compressed with high dynamic range, its edge information is extracted, and it is fused with the contour information of the infrared thermal image to generate an anti-interference image. The fusion process is as follows: Let the visible light image after high dynamic range compression be... The edge map extracted by the Canny operator is Let the synchronized infrared thermal image be... The gradient magnitude map calculated by the Sobel operator is as follows: ; Generate fusion weight graph The position of each pixel weight ,in The preset saturation clipping value; Anti-interference image Generated by the following formula: ; Normalize the results.

[0040] C) If the environment is determined to be weak light and underexposed, the near-infrared supplementary light source is turned on and multi-frame temporal noise reduction is performed on the container surface image to generate an anti-interference image.

[0041] D) If the lighting is normal and there is no significant blur, the image of the container surface is enhanced with contrast and used as an anti-interference image.

[0042] Step S3: Tightly coupled pose calculation.

[0043] The image coordinates of the container corner points are extracted from the anti-interference image; the image coordinates of the corner points and the corresponding spreader motion data are input into the vision-inertial tightly coupled optimization model for joint state estimation, and the real-time pose of the spreader relative to the container is obtained by solving the problem.

[0044] The vision-inertial tightly coupled optimization model is a nonlinear optimization model based on a sliding window. Its solution process includes: defining the sliding window containing... The state vector at each time step is ,in For the first The status of the lifting gear at any given moment. For position vectors, For velocity vectors, For attitude quaternions, and These are the bias vectors for the accelerometer and gyroscope, respectively. For the first Inverse depth parameters for each spatial corner point.

[0045] Construct a global cost function that includes all visual observations and inertial measurement constraints. : ; in, The set of all corner observations. Corner point Reprojection error, For the inertial measurement unit at adjacent time intervals arrive The set of raw measurement data collected between them Based on Calculated inertial pre-integration error, Huber robust loss function and These are the covariance matrices for the corresponding errors.

[0046] Minimize using the Levenberg-Marquardt iterative algorithm The number of iterations is set to 20, and the convergence threshold is set to 10. -6 To obtain the optimal state estimate The latest status within the window Position vector in With attitude quaternions It constitutes the real-time pose.

[0047] Step S4: Disturbance prediction.

[0048] Input the spreader motion data into the spreader-load coupled dynamics model to predict the end position and orientation disturbance deviation of the spreader due to inertial motion and load swing in the next control cycle.

[0049] The spreader-load coupled dynamics model simplifies the spreader and container load into a three-dimensional spatial pendulum system. Its prediction process includes: Step S41, Model Definition: Let the displacement of the lifting device along the trolley direction in the horizontal plane be... The displacement along the direction of the trolley is The swing angle of the load relative to the spreader is ,in The swing angle along the direction of the spreader trolley. Let be the swing angle along the direction of the vehicle; the system's state vector is . ,in For transpose; Step S42, Disturbance Input: Input the current linear acceleration from the spreader motion data. The angular velocity is used as an external disturbance input in the hoist-load coupled dynamics model, where... and These are the instantaneous linear acceleration components along the direction of the main vehicle and the direction of the trolley, extracted from the data of the inertial measurement unit at the current moment. Step S43, Numerical Prediction: Use the estimated system state at the current moment as the initial value. The state equations of the coupled dynamic model of the spreader-load are numerically integrated using the fourth-order Runge-Kutta method to advance one control cycle. To obtain the predicted state ; Let the swing angle in the predicted state be . and ,Right now , ; Step S44, Deviation Calculation: Based on the predicted swing angle and and the geometric connection length between the lifting device and the load Calculate the expected positional deviation of the end effector of the spreader in the horizontal plane: ; in, and These are the expected positional deviations along the direction of the main vehicle and the direction of the auxiliary vehicle, respectively. This refers to the end-effector pose disturbance deviation.

[0050] Step S5: Composite control.

[0051] The feedback control quantity is generated based on the difference between the real-time pose and the target pose, and the feedforward compensation quantity is generated based on the end-effector pose disturbance deviation. The feedback control quantity and the feedforward compensation quantity are fused to generate the final control command to drive the spreader motion.

[0052] The final control command is generated as follows: Step S51, Calculate pose error: Assume the real-time pose is determined by the position... and attitude quaternions The target pose is composed of position and attitude quaternions The composition, then the positional error Attitude error Quaternion operations Convert to rotation vector .

[0053] Step S52: Calculate the feedback control quantity: ; in, , , , This is the gain matrix for the corresponding dimension. This is the differential of the position error.

[0054] Step S53: Calculate the feedforward compensation amount: ; in, This is the feedforward gain matrix.

[0055] Step S54: Generate final instructions: ; in, This is the final control command.

[0056] The invention will be further illustrated below with specific examples.

[0057] A simulation platform was built in the Matlab / Simulink environment, and comparative experiments were conducted with traditional single vision positioning methods and vision-inertial loosely coupled methods.

[0058] Simulation environment parameters.

[0059] The lifting device motion model adopts a three-degree-of-freedom (X-axis for the main trolley, Y-axis for the auxiliary trolley, and Z-axis for the lifting) dynamic model, which includes the characteristics of the servo motor and the transmission clearance.

[0060] Load model: The container is simplified to a mass point and connected to the spreader by a steel wire rope with a length of 5m.

[0061] Environmental interference.

[0062] Lighting variations: Simulates three scenarios: strong light (overexposure rate >30%), weak light (underexposure rate >40%), and normal lighting.

[0063] Motion jitter: Gaussian white noise and periodic vibration (amplitude 0.1m, frequency 2Hz) are superimposed on the speed of the lifting device.

[0064] Camera and IMU parameters: sampling frequencies are 30Hz and 200Hz respectively.

[0065] Comparison method.

[0066] Method A (traditional pure vision): uses only visible light images, employing corner detection and PnP pose calculation with a fixed threshold.

[0067] Method B (Vision-Inertial Loose Coupling): The visual pose output is fused with IMU data using Kalman filtering.

[0068] Method C (the present invention).

[0069] like Figures 2-5 Under different operating conditions, Method C consistently exhibits the fastest convergence speed (reaching a stable state within approximately 2 seconds) and the smallest steady-state positioning error (overexposure in strong light: 8.3 mm, underexposure in weak light: 9.1 mm, normal lighting + jitter: 6.7 mm). Its performance indicators comprehensively outperform both the pure vision method (Method A) and the vision-inertial loosely coupled method (Method B). The results further demonstrate that this invention, through vision-inertial tight-coupled state estimation and feedforward compensation based on a dynamic model, effectively suppresses interference caused by drastic changes in lighting and mechanical jitter. This not only significantly improves positioning accuracy but also substantially shortens alignment adjustment time, verifying its effectiveness and advanced nature in achieving high-precision, highly robust real-time positioning in the complex dynamic environment of ports.

[0070] Example 2 A port container loading and unloading positioning method system includes: Image acquisition unit, used to acquire images of the container surface; An inertial measurement unit is used to measure the angular velocity sequence and linear acceleration sequence within the exposure time window of acquiring images of the container surface, as data for the spreader motion. The adaptive enhancement module is connected to the image acquisition unit and the inertial measurement unit. It is used to calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and to evaluate the ambient light level based on the grayscale characteristics of the container surface image. Then, based on the evaluation results of the equivalent motion amplitude and the ambient light level, it performs adaptive enhancement processing on the container surface image and outputs an anti-interference image. The tightly coupled pose calculation module communicates with the adaptive enhancement module and the inertial measurement unit. It is used to extract the image coordinates of the container corner points from the anti-interference image, and input the image coordinates of the corner points and the corresponding spreader motion data into the vision-inertial tightly coupled optimization model for joint state estimation to obtain the real-time pose of the spreader relative to the container. The disturbance prediction module, which communicates with the inertial measurement unit, is used to input the spreader motion data into the spreader-load coupled dynamics model and predict the end pose disturbance deviation of the spreader due to inertial motion and load swing in a future control cycle. The composite control module, which is connected to the tightly coupled pose calculation module and the disturbance prediction module, is used to generate feedback control quantity based on the difference between the real-time pose and the target pose, and to generate feedforward compensation quantity based on the end pose disturbance deviation. The feedback control quantity and the feedforward compensation quantity are fused to generate the final control command to drive the spreader to move. The spreader drive actuator unit communicates with the composite control module to receive and execute the final control commands, driving the spreader to complete container loading, unloading and positioning operations.

[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0073] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0074] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0075] Therefore, the present invention adopts the above-mentioned port container loading and unloading positioning method, system, equipment and medium, and forms a technical closed loop from robust perception to precise control through multi-sensor adaptive fusion, visual-inertial tight coupling state estimation and feedforward predictive control based on dynamic model. In this way, it can achieve high-precision, high-reliability, real-time positioning of the spreader relative to the container under actual working conditions of strong light, weak light and continuous vibration, and effectively improve the efficiency and success rate of automated loading and unloading operations.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for positioning and unloading containers at a port, characterized in that, Includes the following steps: Step S1: Acquire an image of the container surface, and acquire the angular velocity sequence and linear acceleration sequence measured by the inertial measurement unit within the exposure time window of the container surface image acquisition, as the spreader motion data; Step S2: Calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and evaluate the ambient light level based on the grayscale features of the container surface image; based on the evaluation results of the equivalent motion amplitude and ambient light level, perform adaptive enhancement processing on the container surface image and output an anti-interference image. Step S3: Extract the image coordinates of the container corner points from the anti-interference image; input the image coordinates of the corner points and the corresponding spreader motion data into the vision-inertial tightly coupled optimization model for joint state estimation, and solve for the real-time pose of the spreader relative to the container; Step S4: Input the spreader motion data into the spreader-load coupled dynamics model to predict the end pose disturbance deviation of the spreader due to inertial motion and load swing in the next control cycle; Step S5: Generate feedback control quantity based on the difference between real-time pose and target pose, and generate feedforward compensation quantity based on end pose disturbance deviation. By integrating feedback control quantities and feedforward compensation quantities, the final control command for driving the spreader's movement is generated.

2. The port container loading and unloading positioning method according to claim 1, characterized in that, In step S2, the equivalent motion amplitude is calculated as follows: for the exposure time window angular velocity sequence within Integrating, we obtain the equivalent angular displacement. : ； Linear acceleration sequence By performing double integration, the equivalent linear displacement is obtained. : ； in, The starting moment of the exposure. The moment when the exposure ends; The equivalent motion amplitude is determined by the equivalent angular displacement. model and equivalent linear displacement model Characterization.

3. The port container loading and unloading positioning method according to claim 2, characterized in that, In step S2, based on the evaluation results of equivalent motion amplitude and ambient light level, adaptive enhancement processing is performed on the container surface image to output an anti-interference image, specifically including: Step S21, if If , then the current image is determined to have significant blurring due to angular motion; where The preset angular displacement ambiguity threshold; Step S22: Calculate the grayscale values ​​in the container surface image that are greater than a high threshold. pixel ratio and grayscale values ​​less than the low threshold pixel ratio ; like If so, it is determined to be a strong light overexposure environment, among which This is the preset threshold for the proportion of overexposed pixels; like If so, it is determined to be a low-light, underexposed environment. This is the preset threshold for the proportion of underexposed pixels; Otherwise, it is considered a normal lighting environment; Step S23: Image processing strategy execution: A) If significant ambiguity is detected, the infrared thermal image acquired synchronously with the container surface image shall be selected as the primary image for processing. B) If the environment is determined to be overexposed by strong light, the container surface image is compressed with high dynamic range, its edge information is extracted, and it is fused with the contour information of the infrared thermal image to generate an anti-interference image. C) If the environment is determined to be weak light and underexposed, the near-infrared supplementary light source is turned on and multi-frame temporal noise reduction is performed on the container surface image to generate an anti-interference image. D) If the lighting is normal and there is no significant blur, the image of the container surface is enhanced with contrast and used as an anti-interference image.

4. A port container loading and unloading positioning method according to claim 3, characterized in that, In step S3, the vision-inertial tightly coupled optimization model is a nonlinear optimization model based on a sliding window. Its solution process includes: defining the sliding window containing... The state vector at each time step is ,in For the first The status of the lifting gear at any given moment. For position vectors, For velocity vector, For attitude quaternions, and These are the bias vectors for the accelerometer and gyroscope, respectively. For the first Inverse depth parameters of each spatial corner point; Construct a global cost function that includes all visual observations and inertial measurement constraints. : ； in, The set of all corner observations. Corner point Reprojection error, For the inertial measurement unit at adjacent time intervals arrive The set of raw measurement data collected between them For based on Calculated inertial pre-integration error, Huber robust loss function and These are the covariance matrices corresponding to the errors; Minimize using an iterative optimization algorithm To obtain the optimal state estimate The latest status within the window Position vector in With attitude quaternions It constitutes the real-time pose.

5. A port container loading and unloading positioning method according to claim 4, characterized in that, In step S4, the spreader-load coupled dynamics model simplifies the spreader and container load into a three-dimensional spatial pendulum system. Its prediction process includes: Step S41: Let the displacement of the lifting device along the direction of the trolley in the horizontal plane be... The displacement along the direction of the trolley is The swing angle of the load relative to the spreader is ,in The swing angle along the direction of the spreader trolley. Let be the swing angle along the direction of the vehicle; the system's state vector is . ,in For transpose; Step S42: Calculate the current linear acceleration from the spreader motion data. The angular velocity is used as an external disturbance input in the hoist-load coupled dynamics model, where... and These are the instantaneous linear acceleration components along the direction of the main vehicle and the direction of the trolley, extracted from the data of the inertial measurement unit at the current moment. Step S43: Use the estimated system state at the current moment as the initial value. The state equations of the coupled dynamic model of the spreader-load are numerically integrated using the fourth-order Runge-Kutta method to advance one control cycle. To obtain the predicted state ; Let the swing angle in the predicted state be . and ,Right now , ; Step S44: Based on the predicted swing angle and and the geometric connection length between the lifting device and the load Calculate the expected positional deviation of the end effector of the spreader in the horizontal plane: ； in, and These are the expected positional deviations along the direction of the main vehicle and the direction of the auxiliary vehicle, respectively. This refers to the end-effector pose disturbance deviation.

6. A port container loading and unloading positioning method according to claim 5, characterized in that, In step S5, the final control command for driving the spreader's movement is generated, specifically as follows: Step S51: Assume the real-time pose is determined by the position and attitude quaternions The target pose is composed of position and attitude quaternions The composition, then the positional error Attitude error Quaternion operations Convert to rotation vector ; Step S52: Calculate the feedback control quantity: ； in, , , , This is the gain matrix for the corresponding dimension. The differential of position error; Step S53: Calculate the feedforward compensation amount: ； in, This is the feedforward gain matrix; Step S54: Generate final instructions: ； in, This is the final control command.

7. A port container loading and unloading positioning method according to claim 3, characterized in that, In step S23, when the environment is determined to be overexposed due to strong light, the fusion process is as follows: Let the visible light image after high dynamic range compression be... The edge map extracted by the Canny operator is Let the synchronized infrared thermal image be... The gradient magnitude map calculated by the Sobel operator is as follows: ; Generate fusion weight graph The position of each pixel weight ,in The preset saturation clipping value; Anti-interference image Generated by the following formula: ； Normalize the results.

8. A port container loading and unloading positioning method system, characterized in that, include: Image acquisition unit, used to acquire images of the container surface; An inertial measurement unit is used to measure the angular velocity sequence and linear acceleration sequence within the exposure time window of acquiring images of the container surface, as data for the spreader motion. The adaptive enhancement module is connected to the image acquisition unit and the inertial measurement unit. It is used to calculate the equivalent motion amplitude of the spreader within the exposure time window based on the spreader motion data, and to evaluate the ambient light level based on the grayscale characteristics of the container surface image. Then, based on the evaluation results of the equivalent motion amplitude and the ambient light level, it performs adaptive enhancement processing on the container surface image and outputs an anti-interference image. The tightly coupled pose calculation module communicates with the adaptive enhancement module and the inertial measurement unit. It is used to extract the image coordinates of the container corner points from the anti-interference image, and input the image coordinates of the corner points and the corresponding spreader motion data into the vision-inertial tightly coupled optimization model for joint state estimation to obtain the real-time pose of the spreader relative to the container. The disturbance prediction module, which communicates with the inertial measurement unit, is used to input the spreader motion data into the spreader-load coupled dynamics model and predict the end pose disturbance deviation of the spreader due to inertial motion and load swing in a future control cycle. The composite control module, which is connected to the tightly coupled pose calculation module and the disturbance prediction module, is used to generate feedback control quantity based on the difference between the real-time pose and the target pose, and to generate feedforward compensation quantity based on the end pose disturbance deviation. The feedback control quantity and the feedforward compensation quantity are fused to generate the final control command to drive the spreader to move. The spreader drive actuator unit communicates with the composite control module to receive and execute the final control commands, driving the spreader to complete container loading, unloading and positioning operations.

9. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, it implements the steps of the port container loading and unloading positioning method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the port container loading and unloading positioning method according to any one of claims 1-7.

Images