Lifting appliance posture control method, device, equipment, storage medium and yard crane

Abstract

Embodiments of the present application provide a kind of spreader posture control method, device, equipment, storage medium and yard crane, involve port equipment automatic control technical field.The method comprises: receiving the image data that image sensor shoots and the radar data that laser radar sensor obtains, and radar data is projected into image data corresponding coordinate system, corresponding radar projection data is obtained, radar projection data and image data are input into bird's eye view fusion model, and the posture data of spreader and corresponding target container is output;Posture data is input into multilayer perception, and the posture deviation of spreader relative to target container is output;Based on posture deviation, the posture of the posture of spreader is controlled.The method of the present application effectively solves the problem that the stability and reliability of spreader posture control method based on image recognition algorithm are insufficient.

Description

Technical Field

[0001] This invention relates to the field of automatic control technology for port equipment, and in particular to a method, device, equipment, storage medium, and yard crane for controlling the attitude of a spreader. Background Technology

[0002] In container terminals or yards, gantry cranes are a primary piece of equipment for handling and stacking containers. The gantry crane moves horizontally across the yard via a gantry crane, uses spreaders to grab and unload containers, and a trolley drives the spreaders to move laterally and vertically in a direction perpendicular to the gantry crane's path. During container grabbing and unloading, the control system in the trolley also controls the spreader's attitude to ensure accurate container gripping and prevent collisions.

[0003] In related technologies, the control of spreader attitude is mainly based on image recognition algorithms. However, due to the large travel distance of the spreader, the variable size of the container, and the variable position of the container relative to the trolley, image recognition algorithms are prone to problems such as strong susceptibility to environmental interference and insufficient robustness. Summary of the Invention

[0004] This invention provides a method, apparatus, device, storage medium, and field bridge for controlling the attitude of a spreader, in order to solve the problem of insufficient stability and reliability of spreader attitude control methods based on image recognition algorithms in related technologies.

[0005] In a first aspect, the present invention provides a method for controlling the attitude of a lifting device, comprising:

[0006] The system receives image data captured by an image sensor and radar data acquired by a lidar sensor, and projects the radar data onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data. The image sensor and lidar sensor are mounted on the trolley corresponding to the lifting device, and there are at least two image sensors and at least two lidar sensors.

[0007] The radar projection data and image data are input into the bird's-eye view fusion model, and the attitude data of the spreader and the corresponding target container are output.

[0008] The attitude data is input into the multilayer sensor, and the attitude deviation of the spreader relative to the target container is output.

[0009] The attitude of the spreader is controlled based on the attitude deviation.

[0010] In a second aspect, the present invention provides a lifting device for controlling the attitude of a lifting device, comprising:

[0011] The receiving module is used to receive image data captured by the image sensor and radar data acquired by the lidar sensor, and project the radar data onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data. The image sensor and lidar sensor are set on the trolley corresponding to the lifting device, and there are at least two image sensors and at least two lidar sensors.

[0012] The processing module is used to input radar projection data and image data into the bird's-eye view fusion model and output the attitude data of the spreader and the corresponding target container.

[0013] The analysis module is used to input attitude data into the multilayer sensor and output the attitude deviation of the spreader relative to the target container.

[0014] The output module is used to control the attitude of the spreader based on the attitude deviation.

[0015] Thirdly, embodiments of the present invention provide an electronic device, including: a memory and a processor;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0018] Fourthly, embodiments of the present invention provide a field bridge, the field bridge including a spreader attitude control device as described in the second aspect above and / or various possible implementations of the second aspect.

[0019] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0020] In a sixth aspect, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0021] The spreader attitude control method, apparatus, device, storage medium, and yard crane provided in this disclosure receive image data captured by an image sensor and radar data acquired by a lidar sensor. The radar data is projected onto the coordinate system corresponding to the image data to obtain corresponding radar projection data. This radar projection data and image data are input into a bird's-eye view fusion model, outputting attitude data of the spreader and the corresponding target container. The attitude data is then input into a multilayer perceptron, which outputs the attitude deviation of the spreader relative to the target container. Based on this attitude deviation, the spreader's attitude is controlled. Therefore, by combining data from image sensors and lidar sensors, the stability and reliability of spreader attitude control are significantly improved. By projecting radar data into an image coordinate system and inputting it into a bird's-eye view fusion model, the attitude data of the spreader and the target container can be accurately obtained. Subsequently, the attitude deviation is calculated using a multilayer perceptron to achieve precise control of the spreader's attitude. This multi-sensor fusion method effectively overcomes the environmental interference and insufficient robustness problems associated with relying solely on image recognition algorithms, improving the operational accuracy and safety of the spreader in complex scenarios and ensuring accurate container gripping and placement. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 This is an application scenario diagram of the spreader attitude control method provided in the embodiments of this disclosure;

[0024] Figure 2 A flowchart of a spreader attitude control method provided in one embodiment of this disclosure;

[0025] Figure 3 A flowchart of a spreader attitude control method provided in yet another embodiment of this disclosure;

[0026] Figure 4 A schematic diagram of the structure of a lifting device for another embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of a field bridge provided in one embodiment of the present disclosure.

[0029] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. They do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] Furthermore, the present invention relates to a technical solution that performs big data analysis on user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology to make automated decisions, and makes decisions that have a significant impact on personal rights based on the results of automated decisions, and provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decisions; if the user chooses to reject, the process enters the expert decision-making process.

[0033] It should be noted that the method, apparatus, equipment, storage medium, and field bridge for spreader attitude control provided by the present invention can be used in the field of financial technology, or in any field other than financial technology. The application fields of the method, apparatus, equipment, storage medium, and field bridge for spreader attitude control in the present invention are not limited.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario of the lifting device attitude control method provided by the present invention, such as... Figure 1 As shown, during the spreader attitude control process, when the gantry crane 100 is grabbing and placing a container, it will collect data on the spreader 130 and the target container 140 through the image sensor 110 and the lidar sensor 120 to control the movement of the gantry crane 100's yard crane 150, trolley 160 and spreader 130, and complete the interaction between the spreader 130 and the target container 140.

[0035] It should be noted that, Figure 1The scenario shown includes a gantry crane, image sensor, lidar sensor, spreader, and target container. Only one or a specific number of these are used as examples for illustration, but this disclosure is not limited to this. That is to say, the number of gantry cranes, image sensors, lidar sensors, spreaders, and target containers can be arbitrary.

[0036] In port logistics, gantry cranes, as core equipment for container handling and stacking, are crucial to the overall operational efficiency of the terminal. Gantry cranes move horizontally via yard cranes and use spreaders to grab and unload containers. The spreaders can also move vertically and laterally to adapt to different operational needs. Precise control of the spreader's attitude is key to ensuring the safe and accurate handling of containers. However, traditional spreader attitude control relies primarily on image recognition algorithms, which face numerous challenges in practical applications. First, the large spreader movement distance and the variable size and position of containers make image recognition algorithms susceptible to interference in dynamic environments. Furthermore, the complex and variable port environment, including lighting conditions, weather changes, and interference from other mechanical equipment, all affect the accuracy of image recognition. These factors lead to deficiencies in the robustness and stability of image recognition algorithms, consequently impacting the operational precision and safety of the spreaders.

[0037] However, in practical research, image recognition often performs well in static and controlled environments, but its performance is often unsatisfactory in complex operational environments such as ports. Therefore, it is usually difficult to identify potential problems in the spreader control process during the research and development phase. Furthermore, when improving this solution, the special characteristics of the port environment must be considered, including equipment durability, system usability, and compatibility with existing operational processes. Consequently, there is a lack of solutions in related technologies that can meet the stability and accuracy requirements of spreader control in port logistics.

[0038] The spreader attitude control method provided by this invention utilizes the complementarity of data from image sensors and lidar sensors, projecting radar data onto an image coordinate system to form a comprehensive environmental perception view. This data is processed through a bird's-eye view fusion model to obtain precise attitude information of the spreader and container. Subsequently, a multilayer perceptron is used to calculate attitude deviations, achieving precise control of the spreader attitude and effectively improving the robustness and accuracy of spreader control in port logistics environments.

[0039] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0040] Figure 2Flowchart of the lifting device attitude control method provided by the present invention Figure 1 ,like Figure 2 As shown, the method includes:

[0041] S201. Receive image data captured by the image sensor and radar data acquired by the lidar sensor, and project the radar data onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data.

[0042] The image sensor and the lidar sensor are mounted on the trolley corresponding to the lifting device, with at least two image sensors and at least two lidar sensors.

[0043] Specifically, this embodiment is used to provide an overall description of the main steps of the spreader attitude control method.

[0044] In this embodiment, the executing entity is a controller that controls the operation of the gantry crane and can acquire data collected by the image sensor and lidar sensor installed on the trolley of the gantry crane. Specifically, it can be a control system installed in the gantry crane, or a cloud server that communicates with the control system of the gantry crane. For ease of explanation, it is referred to as the control system here.

[0045] During the movement of the gantry crane trolley, the control system first receives data from multiple image sensors and lidar sensors mounted on the spreader trolley corresponding to the spreader. The image sensors are typically high-resolution cameras that capture visual information about the environment around the spreader and the target container below the trolley, while the lidar sensors provide precise distance and depth information.

[0046] Because image sensors and LiDAR sensors operate in different coordinate systems, effective data fusion requires projecting the radar data into the coordinate system of the image data. This process involves coordinate transformation and data alignment, typically using sensor calibration parameters, including intrinsic and extrinsic parameters, to achieve accurate projection. Intrinsic parameters involve the sensor's focal length, principal point coordinates, and distortion coefficients, while extrinsic parameters describe the sensor's position and orientation in three-dimensional space.

[0047] These parameters allow the 3D point cloud in radar data to be projected onto the image plane, forming radar projection data. This projection enables the unified processing of radar data and image data.

[0048] In some embodiments, the image sensor and the lidar sensor are positioned in the same location. For example, if there are two image sensors and two lidar sensors, the image sensors can be symmetrically positioned at two locations off-center from the bottom of the vehicle, while the lidar sensors are positioned in the same locations as the image sensors.

[0049] S202. Input radar projection data and image data into the bird's-eye view fusion model, and output the attitude data of the spreader and the corresponding target container.

[0050] Specifically, the radar projection data and image data, after coordinate transformation, are input into the bird's-eye view fusion model. The bird's-eye view fusion model (also known as the BEV fusion network model, where BEV stands for Bird's-Eye View, hereinafter referred to as the BEV model) is an advanced data processing model designed to combine data from different sensors to provide a comprehensive environmental perception view. By fusing visual information from images and depth information from radar, this model can more accurately identify and locate the attitude of spreader and target container.

[0051] Before practical application, the BEV model can be trained with a large amount of training data based on deep learning technology to ensure high-precision attitude recognition in the complex environment of port logistics.

[0052] The model's inputs include preprocessed image data and radar projection data, while the outputs are attitude data of the spreader and container, including information such as position, angle, and orientation.

[0053] In some embodiments, transfer learning techniques can be used to train the BEV model to reduce training time and data requirements, or real-time data can be used for online learning and updates.

[0054] S203. Input the attitude data into the multilayer sensor and output the attitude deviation of the spreader relative to the target container.

[0055] Specifically, the attitude data output by the BEV model is further input into a multilayer perceptron (MLP) to calculate the attitude deviation of the spreader relative to the target container.

[0056] MLP (Multi-Level Processing) is used to learn the mapping relationship between input data and output results to predict the attitude differences between the spreader and the container. The input attitude data includes information such as the three-dimensional position and rotation angle of the spreader and the container. MLP uses this data to calculate attitude deviations, which reflect the gap between the current attitude of the spreader and the ideal attitude required to hold the container.

[0057] S204. Based on the attitude deviation, control the attitude of the spreader.

[0058] Specifically, the calculated attitude deviation is used to adjust the attitude of the spreader in real time to ensure that it can accurately align with and clamp the target container.

[0059] In some embodiments, the control system may employ advanced control algorithms such as proportional-integral-derivative (PID) controllers or model predictive control (MPC) to achieve high-precision adjustment of the spreader's attitude. PID controllers can quickly respond to attitude deviations and make corresponding adjustments by adjusting the proportional, integral, and derivative coefficients, while MPC optimizes the control input to achieve optimal control performance by predicting future system states.

[0060] The control system needs to receive sensor data and attitude deviation calculations in real time to ensure that the spreader can respond and adjust quickly in dynamic environments.

[0061] The spreader attitude control method provided in this invention receives detection images captured by a monocular camera, corrects the images based on the camera parameters, inputs the corrected images into a depth estimation network, and determines the corresponding 3D point cloud based on the network's output. Based on the 3D point cloud, obstacle data within the gantry crane's travel range is input into a bird's-eye view fusion model, which outputs attitude data for the spreader and the corresponding target container. This attitude data is then input into a multilayer perceptron, which outputs the spreader's attitude deviation relative to the target container. Based on this deviation, the spreader's attitude is controlled. Thus, by combining data from image sensors and LiDAR sensors, the stability and reliability of spreader attitude control are significantly improved. By projecting radar data onto an image coordinate system and inputting it into the bird's-eye view fusion model, the attitude data of the spreader and the target container can be accurately obtained. Subsequently, the attitude deviation is calculated using a multilayer perceptron, achieving precise control of the spreader's attitude. This multi-sensor fusion method effectively overcomes the environmental interference and insufficient robustness problems caused by relying solely on image recognition algorithms, improves the operational accuracy and safety of spreaders in complex scenarios, and ensures the accurate grabbing and placement of containers.

[0062] Figure 3 Flowchart of spreader attitude control provided by the present invention Figure 2 ,like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the specific process of the spreader attitude control method is described in detail, which includes:

[0063] S301. Perform time synchronization processing on the image sensor and the lidar sensor.

[0064] Specifically, this embodiment is used to explain the specific implementation method of the spreader attitude control process.

[0065] Specifically, before controlling the attitude of the spreader, it is necessary to debug the image sensor and the lidar sensor, including time synchronization processing and parameter calibration. Time synchronization processing is used to ensure that the data from the image sensor and the lidar sensor can be fused under the same time reference.

[0066] Because the sampling frequencies and data transmission delays of the two sensors may differ, unsynchronized data can lead to inconsistencies, affecting subsequent data fusion and attitude calculation. Time synchronization is typically achieved through a combination of hardware and software. Hardware synchronization can be achieved using dedicated synchronization modules or by sharing a clock signal (e.g., using GPS timing as a unified clock source for both the visual and LiDAR sensors), while software synchronization can be performed using timestamp alignment (e.g., when data acquisition is triggered, the timestamps of both devices should be within 10ms of each other).

[0067] S302. Perform parameter calibration on the image sensor and lidar sensor.

[0068] Among them, parameter calibration is used to obtain the intrinsic parameter matrix and the extrinsic parameter matrix.

[0069] Specifically, parameter calibration is to obtain the intrinsic and extrinsic parameter matrices of the image sensor and the LiDAR sensor in order to achieve accurate data fusion and coordinate system transformation between the data from the image sensor and the LiDAR sensor.

[0070] The intrinsic parameter matrix describes the sensor's internal characteristics, such as focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameter matrix describes the sensor's position and orientation in three-dimensional space. The calibration process typically involves capturing multiple images on a calibration board of known dimensions, using these images to calculate the sensor's intrinsic and extrinsic parameters. For LiDAR, the calibration process may involve using reflective markers or specific calibration targets to obtain the LiDAR's position relative to the image sensor. The accuracy of the calibration directly affects the precision of data fusion in subsequent steps; therefore, high-precision calibration tools and algorithms are required.

[0071] In some embodiments, periodic recalibration is also necessary to address environmental changes and device drift. Parameter calibration ensures that sensor data can be accurately projected and fused within the same coordinate system.

[0072] S303. Determine that the reprojection error in the lidar sensor and the image sensor meets the set error requirements.

[0073] Specifically, since lidar sensors and image sensors are different types and are used in complex environments, even after calibration, it is necessary to verify whether their calibration error meets the requirements, that is, to verify the reprojection error.

[0074] Reprojection error is a crucial indicator for evaluating sensor calibration accuracy. It refers to the deviation between the projected 3D point cloud data and the actual image data. To ensure data fusion accuracy, the reprojection error must meet set error requirements. This process typically involves calculating the pixel deviation of each projected point and optimizing calibration parameters by minimizing these deviations. If the reprojection error exceeds the set range, recalibration or adjustment of calibration parameters is necessary.

[0075] To improve the robustness of calibration, optimization algorithms such as the Levenberg-Marquardt algorithm can be used for nonlinear least squares optimization to precisely adjust the intrinsic and extrinsic parameter matrices.

[0076] In some embodiments, reprojection error checks are performed not only during the calibration of the lidar sensor and the image sensor, but also periodically during the application phase to ensure the accuracy and reliability of the spreader attitude deviation calculations based on the two sensors.

[0077] S304: Receive image data captured by the image sensor and radar data acquired by the lidar sensor.

[0078] Specifically, during the application phase, the control system receives real-time data from image sensors and LiDAR sensors. This data forms the basis for attitude calculation and control in subsequent steps.

[0079] S305. Based on the calibration parameters of the image sensor obtained from the calibration of the image sensor and the lidar sensor, the radar data is projected onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data.

[0080] The calibration parameters include the intrinsic and extrinsic parameter matrices of the image sensor. The extrinsic parameter matrices include rotation and translation matrices used to project radar data onto the coordinate system where the image data is located.

[0081] Specifically, using the previously calibrated intrinsic and extrinsic parameter matrices of the image sensor, the LiDAR data can be projected onto the coordinate system of the image data. This process involves converting 3D point cloud data into points on a 2D image plane to achieve effective data fusion.

[0082] The calibration parameters include extrinsic and intrinsic parameters. The extrinsic parameters further include rotation and translation matrices. The intrinsic parameters are used to correct image distortion and adjust the focal length, while the extrinsic parameters are used to transform radar data from its local coordinate system to the image coordinate system. The rotation matrix is ​​used to adjust the orientation of the radar data, and the translation matrix is ​​used to adjust its position. Through the operations of these matrices, the radar data can be accurately projected onto the image plane to form radar projection data.

[0083] For example, the formula for projecting the point cloud corresponding to the radar data onto the coordinate system corresponding to the image data using the extrinsic parameter matrix, intrinsic parameter matrix, and radar data can be expressed as:

[0084] ;

[0085] Where u and v are the two-dimensional coordinates of the image data, and x, y, and z are the three-dimensional point cloud coordinates of the radar data. K is the camera intrinsic parameter, and R and T are the rotation and translation matrices of the lidar to the camera, respectively.

[0086] In some embodiments, in order to improve the accuracy of projection, optimization algorithms can be used to correct the projection process, or multi-view data can be combined for comprehensive analysis to ensure that radar data and image data are fused in the same coordinate system, providing a unified basis for subsequent attitude calculation.

[0087] S306. Input radar projection data and image data into the bird's-eye view fusion model, and output the attitude data of the spreader and the corresponding target container.

[0088] Specifically, the radar projection data and image data that have undergone projection processing are input into the bird's-eye view fusion model to obtain the attitude data of the spreader and the target container.

[0089] In some embodiments, if the radar data is abnormal, the image data is input into the bird's-eye view fusion model, and the attitude data of the spreader and the corresponding target container are output; or, if the image data is abnormal, the radar projection data is input into the bird's-eye view fusion model, and the attitude data of the spreader and the corresponding target container are output.

[0090] Specifically, if radar data is abnormal, the model can rely solely on image data for calculation; conversely, if radar data is normal, this redundancy design improves the system's robustness.

[0091] In some embodiments, the attitude data includes the three-dimensional coordinates of the spreader’s center point, size information, rotation angle, the three-dimensional coordinates of the target container’s center point, size information, rotation angle, and container size type.

[0092] For example, the attitude data of the spreader and the target container can be represented as:

[0093] ;

[0094] Where X, Y, and Z are the center points of the target (such as spreader or target container), L, W, and H are the length, width, and height of the target. θ is the yaw angle of the target, and ID is the target category. Target categories include 20-foot containers, 40-foot containers, and 45-foot containers (the target container category is one of these three; in practical applications, other types can also be included, requiring only pre-configuration by management personnel) and spreaders.

[0095] S307. Input the horizontal coordinates of the center point of the spreader and the target container, the dimensions parallel to the horizontal plane, and the rotation angle from the attitude data into the multilayer sensor, and output the positional deviation between the spreader and the target container.

[0096] Specifically, the attitude data output by the bird's-eye view fusion model is input into a multilayer perceptron (MLP) to calculate the positional deviation between the spreader and the target container in three directions (such as parallel to the direction of travel of the spreader's corresponding yard crane trolley, parallel to the direction of travel of the spreader's corresponding trolley, and perpendicular to the horizontal plane; or, it can be two mutually perpendicular directions of movement of the spreader in a plane parallel to the direction of travel of the spreader's corresponding yard crane trolley and perpendicular to the direction of travel of the yard crane trolley).

[0097] In some embodiments, the multilayer perceptron can be a three-layer structure, with ten nodes in the input layer to receive the aforementioned types of input data, and three nodes in the output layer to represent positional deviations in three directions.

[0098] In some embodiments, image samples with the centers of standard 3D bounding boxes labeled can be used as training data to train the MLP model.

[0099] In some embodiments, the specific deviation direction can be two directions with an included angle in a plane that is parallel to the direction of travel of the trolley corresponding to the spreader and perpendicular to the direction of travel of the trolley (such as the direction of travel of the trolley corresponding to the spreader and the direction of movement of the spreader relative to the trolley).

[0100] S308. Based on the first deviation value in the attitude deviation that is parallel to the travel direction of the gantry crane corresponding to the spreader, send a first control command to the gantry crane.

[0101] The first control command is used to control the yard crane trolley to move based on the first deviation value.

[0102] Specifically, the calculated attitude deviations are used to generate control commands to control the movement of the yard crane trolley, trolley, and spreader. More specifically, the system generates a first control command based on a first deviation value in the attitude deviation that is parallel to the travel direction of the yard crane trolley corresponding to the spreader. This command is used to adjust the motion parameters of the yard crane trolley, such as speed and direction, to correct deviations and ensure that the spreader can accurately align with the target container.

[0103] S309. Based on the second and third deviation values ​​corresponding to the direction of spreader movement in the attitude deviation, send a second control command to the trolley.

[0104] The second control command is used to control the trolley to adjust the position of the lifting device based on the second deviation value and the third deviation value, wherein the corresponding directions of the second deviation value and the third deviation value are perpendicular to each other.

[0105] Specifically, based on the second and third deviation values ​​in the attitude deviation, the system can adjust the relative position of the trolley and the spreader to correct the deviation and ensure that the spreader can move along the correct path.

[0106] In some embodiments, the second deviation value can be the deviation of the spreader from the trolley's travel direction and the deviation of the spreader from the trolley's movement direction.

[0107] The spreader attitude control method provided in this disclosure significantly improves the accuracy and reliability of data fusion by synchronizing and calibrating the time and parameters of image sensors and LiDAR sensors. By ensuring that the reprojection error is within a set range, the system can achieve high-precision sensor data alignment, thereby improving the accuracy of attitude recognition. Combining a bird's-eye view fusion model and a multilayer perceptron, the system can effectively identify and calculate the attitude deviation between the spreader and the container in complex environments. By generating precise control commands, the system can adjust the movement of the gantry crane and its trolley in real time to ensure precise operation of the spreader. This solution not only improves the stability and safety of spreader attitude control but also enhances the system's adaptability in dynamic environments, optimizing the overall efficiency of port logistics.

[0108] Figure 4 This is a schematic diagram of the structure of the lifting device attitude control device provided by the present invention, as shown below. Figure 4 As shown, the lifting device attitude control device 400 provided in this embodiment includes:

[0109] The receiving module 410 is used to receive image data captured by the image sensor and radar data acquired by the lidar sensor, and project the radar data onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data. The image sensor and lidar sensor are set on the trolley corresponding to the lifting device, and there are at least two image sensors and at least two lidar sensors.

[0110] The processing module 420 is used to input radar projection data and image data into the bird's-eye view fusion model and output the attitude data of the spreader and the corresponding target container.

[0111] Analysis module 430 is used to input attitude data into the multilayer perceptron and output the attitude deviation of the spreader relative to the target container;

[0112] Output module 440 is used to control the attitude of the spreader based on attitude deviation.

[0113] In one possible implementation, the receiving module 410 is specifically used to receive image data captured by the image sensor and radar data acquired by the lidar sensor; based on the calibration parameters of the image sensor obtained by calibrating the image sensor and the lidar sensor, the radar data is projected onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data. The calibration parameters include the intrinsic parameter matrix and the extrinsic parameter matrix of the image sensor. The extrinsic parameter matrix includes a rotation matrix and a translation matrix used to project the radar data onto the coordinate system where the image data is located.

[0114] In one possible implementation, the receiving module 410 is further configured to receive image data captured by the image sensor and radar data acquired by the lidar sensor, and before projecting the radar data onto the coordinate system corresponding to the image data to obtain the corresponding radar projection data, perform time synchronization processing on the image sensor and the lidar sensor; and perform parameter calibration on the image sensor and the lidar sensor, the parameter calibration being used to obtain the intrinsic parameter matrix and the extrinsic parameter matrix.

[0115] In one possible implementation, the receiving module 410 is further configured to determine, after calibrating the parameters of the image sensor and the lidar sensor, that the reprojection error in the lidar sensor and the image sensor meets the set error requirements.

[0116] In one possible implementation, the processing module 420 is specifically used to: if the radar data is abnormal, input the image data into the bird's-eye view fusion model and output the attitude data of the spreader and the corresponding target container; or, if the image data is abnormal, input the radar projection data into the bird's-eye view fusion model and output the attitude data of the spreader and the corresponding target container.

[0117] In one possible implementation, the processing module 420 specifically includes attitude data including the three-dimensional coordinates of the spreader's center point, size information, rotation angle, the three-dimensional coordinates of the target container's center point, size information, rotation angle, and container size type.

[0118] In one possible implementation, the analysis module 430 is specifically used to input the horizontal plane coordinates of the center point of the spreader and the target container, the dimension information parallel to the horizontal plane, and the rotation angle from the attitude data into the multilayer sensor, and output the positional deviation between the spreader and the target container.

[0119] In one possible implementation, the output module 440 is specifically used to: send a first control command to the trolley based on a first deviation value in the attitude deviation that is parallel to the travel direction of the trolley corresponding to the spreader; the first control command is used to control the trolley to move based on the first deviation value; and send a second control command to the trolley based on a second deviation value and a third deviation value in the attitude deviation that correspond to the travel direction of the spreader; the second control command is used to control the trolley to adjust the position of the spreader based on the second deviation value and the third deviation value, the corresponding directions of the second deviation value and the third deviation value being perpendicular to each other.

[0120] The lifting device attitude control device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0121] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0122] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0123] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0124] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0125] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0126] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0127] like Figure 6 As shown, this is a schematic diagram of the structure of the field bridge provided by the present invention. The present invention also provides a field bridge, the field bridge 60 comprising as follows: Figure 4 The lifting device 610 in the illustrated embodiment is implemented in a manner similar to the methods described above, and its implementation principle and technical effects are not repeated here. This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above embodiments.

[0128] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above embodiments.

[0129] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0130] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0131] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0134] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0136] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0137] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present invention can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0138] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of the present invention can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0139] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0140] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0141] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0142] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0143] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method of controlling the attitude of a spreader, characterized by, The method comprises the following steps: receiving image data captured by an image sensor and radar data obtained by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a trolley corresponding to a spreader, the image sensor has at least two, and the laser radar sensor has at least two; inputting the radar projection data and the image data into an aerial view fusion model to output attitude data of the spreader and a corresponding target container; inputting the attitude data into a multi-layer perception machine to output attitude deviation of the spreader relative to the target container; controlling the attitude of the spreader based on the attitude deviation.

2. The method of claim 1, wherein, The receiving image data captured by an image sensor and radar data obtained by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data comprises: receiving image data captured by an image sensor and radar data obtained by a laser radar sensor; projecting the radar data into a coordinate system corresponding to the image data based on calibration parameters of the image sensor obtained by calibrating the image sensor and the laser radar sensor to obtain corresponding radar projection data, wherein the calibration parameters comprise an intrinsic matrix and an extrinsic matrix of the image sensor, and the extrinsic matrix comprises a rotation matrix and a translation matrix for projecting the radar data into the coordinate system in which the image data is located.

3. The method of claim 2, wherein, Before the receiving image data captured by an image sensor and radar data obtained by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, the method further comprises: performing time synchronization processing on the image sensor and the laser radar sensor; performing parameter calibration on the image sensor and the laser radar sensor, wherein the parameter calibration is used to obtain the intrinsic matrix and the extrinsic matrix.

4. The method of claim 3, wherein, After the performing parameter calibration on the image sensor and the laser radar sensor, the method further comprises: determining that a re-projection error in the image sensor and the laser radar sensor meets a set error requirement.

5. The method of claim 1, wherein, The inputting the radar projection data and the image data into an aerial view fusion model to output attitude data of the spreader and a corresponding target container further comprises: if the radar data is abnormal, inputting the image data into the aerial view fusion model to output the attitude data of the spreader and the corresponding target container; or if the image data is abnormal, inputting the radar projection data into the aerial view fusion model to output the attitude data of the spreader and the corresponding target container.

6. The method according to any one of claims 1 to 5, characterized in that, The attitude data comprises a three-dimensional coordinate of a center point of the spreader, size information, a rotation angle, a three-dimensional coordinate of a center point of the target container, size information, a rotation angle, and a container size category.

7. The method of claim 6, wherein, The inputting the attitude data into a multi-layer perception machine to output attitude deviation of the spreader relative to the target container comprises: The horizontal plane coordinates, size information parallel to the horizontal plane, and rotation angle of the center point of the target container in the spreader in the attitude data are input into a multi-layer perception machine, and a position deviation of the spreader from the target container is output.

8. The method of claim 7, wherein, The attitude control of the spreader attitude based on the attitude deviation comprises: Based on a first deviation value parallel to the travel direction of the corresponding yard crane of the spreader in the attitude deviation, a first control instruction is sent to the yard crane, and the first control instruction is used to control the yard crane to move based on the first deviation value; Based on a second deviation value and a third deviation value corresponding to the moving direction of the spreader in the attitude deviation, a second control instruction is sent to the trolley, and the second control instruction is used to control the trolley to adjust the position of the spreader based on the second deviation value and the third deviation value, and the corresponding directions of the second deviation value and the third deviation value are perpendicular to each other.

9. A spreader attitude control device, characterized by, Comprise: A receiving module is configured to receive image data captured by an image sensor and radar data obtained by a laser radar sensor, and project the radar data into a corresponding coordinate system of the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a corresponding trolley of a spreader, the image sensor has at least two, and the laser radar sensor has at least two; A processing module is configured to input the radar projection data and the image data into an aerial view fusion model to output attitude data of the spreader and a corresponding target container; An analysis module is configured to input the attitude data into a multi-layer perception machine to output an attitude deviation of the spreader relative to the target container; An output module is configured to control the attitude of the spreader attitude based on the attitude deviation.

10. An electronic device, comprising: Comprise: A processor, and a memory connected in communication with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A yard crane, characterized in that The yard crane comprises the spreader attitude control device according to claim 9.

Images