Lifting appliance attitude control method, device and equipment, storage medium and field bridge

Abstract

The embodiment of the invention provides a lifting appliance attitude control method, device and equipment, a storage medium and a field bridge, and relates to the technical field of port equipment automatic control. The method comprises the following steps: receiving image data shot by an image sensor and radar data acquired by a laser radar sensor, projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, inputting the radar projection data and the image data into a bird's-eye view fusion model to obtain a bird's-eye view fusion model; outputting attitude data of the lifting appliance and the corresponding target container; inputting the attitude data into a multi-layer perceptron, and outputting the attitude deviation of the lifting appliance relative to the target container; and controlling the posture of the lifting appliance based on the posture deviation. According to the method, the problem that a lifting appliance attitude control method based on an image recognition algorithm is insufficient in stability and reliability is effectively solved.

Description

Technical Field

[0001] The present application relates to the technical field of automatic control of port equipment, and in particular to a method, device, equipment, storage medium and field bridge for controlling the posture of a spreader. Background Art

[0002] In a container terminal or yard, a gantry crane is a main device for handling and stacking containers. The gantry crane moves horizontally on the yard through a yard crane trolley, grabs and unloads containers through a spreader, and drives the spreader to move horizontally and vertically in a direction perpendicular to the moving track of the yard crane trolley through a trolley. When grabbing and unloading containers, the control system in the trolley also controls the spreader's posture to ensure that the spreader accurately clamps the container and does not collide with the container.

[0003] In the related art, the control of the spreader posture is mainly based on the image recognition algorithm. Due to the large moving distance of the spreader, the non-fixed container size, the non-fixed position of the container relative to the trolley and other problems, the image recognition algorithm is easily affected by strong environmental interference and lacks robustness. Summary of the invention

[0004] The present application provides a method, device, equipment, storage medium and field bridge for controlling the posture of a sling, so as to solve the problem of insufficient stability and reliability of the sling posture control method based on image recognition algorithm in the related art.

[0005] In a first aspect, the present application provides a method for controlling a sling posture, comprising:

[0006] Receiving image data captured by an image sensor and radar data acquired by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a trolley corresponding to the sling, and there are at least two image sensors and at least two laser radar sensors;

[0007] The radar projection data and image data are input into the bird’s-eye view fusion model, and the posture data of the spreader and the corresponding target container are output;

[0008] The posture data is input into a multi-layer perceptron, and the posture deviation of the spreader relative to the target container is output;

[0009] Based on the posture deviation, the posture of the spreader is controlled.

[0010] In a second aspect, the present application provides a sling attitude control device, comprising:

[0011] A receiving module, used to receive image data captured by an image sensor and radar data acquired by a laser radar sensor, and project the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a trolley corresponding to the sling, and there are at least two image sensors and at least two laser radar sensors;

[0012] A processing module, used for inputting radar projection data and image data into a bird's-eye view fusion model, and outputting posture data of the spreader and the corresponding target container;

[0013] An analysis module, used to input the posture data into a multi-layer perceptron and output the posture deviation of the spreader relative to the target container;

[0014] The output module is used to control the posture of the spreader based on the posture deviation.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a field bridge, which includes a spreader attitude control device as described in the second aspect and / or various possible implementations of the second aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0021] The method, device, equipment, storage medium and field bridge provided by the embodiment of the present disclosure receive the image data taken by the image sensor and the radar data obtained by the laser radar sensor, and project the radar data into the coordinate system corresponding to the image data to obtain the corresponding radar projection data, input the radar projection data and the image data into the bird's-eye view fusion model, output the posture data of the spreader and the corresponding target container; input the posture data into the multi-layer perceptron, output the posture deviation of the spreader relative to the target container; and control the posture of the spreader based on the posture deviation. Thus, by combining the data of the image sensor and the laser radar sensor, the stability and reliability of the spreader posture control are significantly improved. By projecting the radar data into the image coordinate system and inputting it into the bird's-eye view fusion model, the posture data of the spreader and the target container can be accurately obtained. Subsequently, the posture deviation is calculated using the multi-layer perceptron to achieve precise control of the spreader posture. This multi-sensor fusion method effectively overcomes the environmental interference and insufficient robustness problems of relying solely on the image recognition algorithm, improves the operation accuracy and safety of the spreader in complex scenes, and ensures the accurate grasping and placement of the container. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A diagram of an application scenario of the method for controlling the posture of a sling provided in an embodiment of the present disclosure;

[0024] Figure 2 A flow chart of a method for controlling a hanger posture according to an embodiment of the present disclosure;

[0025] Figure 3 A flow chart of a method for controlling a hanger posture provided by another embodiment of the present disclosure;

[0026] Figure 4 A schematic structural diagram of a hanger posture control device provided by yet another embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the structure of a field bridge provided in one embodiment of the present disclosure.

[0029] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0033] It should be noted that the method, device, equipment, storage medium and field bridge for hoisting posture control provided in the present application can be used in the field of financial technology, and can also be used in any field other than the field of financial technology. The application field of the method, device, equipment, storage medium and field bridge for hoisting posture control in the present application is not limited.

[0034] Figure 1 A schematic diagram of an application scenario of the method for controlling the posture of a sling provided in this application, such as Figure 1 As shown, during the process of hoist posture control, when the gantry crane 100 grabs and releases the container, it will collect data of the hoist 130 and the target container 140 through the image sensor 110 and the lidar sensor 120 to control the movement of the yard crane 150, the trolley 160 and the hoist 130 of the gantry crane 100, and complete the interaction between the hoist 130 and the target container 140.

[0035] It should be noted that Figure 1The scene shown includes only one or a specific number of gantry cranes, image sensors, lidar sensors, spreaders, and target containers for illustration, but the present disclosure is not limited to this. In other words, the number of gantry cranes, image sensors, lidar sensors, spreaders, and target containers can be arbitrary.

[0036] In the field of port logistics, gantry cranes are the core equipment for container handling and stacking, and their efficient operation is crucial to the overall operational efficiency of the terminal. The gantry crane moves horizontally through the yard bridge trolley and grabs and unloads containers through the spreader. The spreader can also move in the vertical and lateral directions to adapt to different operating requirements. In this process, precise control of the spreader's posture is the key to ensuring safe and accurate handling of containers. However, traditional spreader posture control mainly relies on image recognition algorithms, which face many challenges in practical applications. First, the spreader moves a long distance, and the size and position of the container are not fixed, making the image recognition algorithm susceptible to interference in a dynamic environment. In addition, the port environment is complex and changeable, and the lighting conditions, weather changes, and interference from other mechanical equipment will affect the accuracy of image recognition. These factors lead to the lack of robustness and stability of the image recognition algorithm, which in turn affects the operating accuracy and safety of the spreader.

[0037] However, in actual research, image recognition often performs well in static and controlled environments, but its performance is often unsatisfactory in complex operating environments such as ports. Therefore, it is usually difficult to find possible problems in the spreader control process during the research and development stage. At the same time, when improving this solution, it is necessary to take into account the particularity of the port environment, including the durability of the equipment, the ease of use of the system, and the compatibility with the existing operating processes. Therefore, there is a lack of solutions in the relevant technologies that can meet the stability and accuracy requirements of port logistics for spreader control.

[0038] The method for controlling the attitude of the spreader provided in this application utilizes the data complementarity of the image sensor and the lidar sensor to project the radar data into the image coordinate system to form a comprehensive environmental perception view. The data is processed by the bird's-eye view fusion model to obtain the precise attitude information of the spreader and the container. Subsequently, the attitude deviation is calculated using a multi-layer perceptron to achieve precise control of the attitude of the spreader, effectively improving the robustness and accuracy of the spreader control in the port logistics environment.

[0039] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] Figure 2Schematic diagram of the process of the hanger posture control method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0041] S201, receiving image data captured by an image sensor and radar data acquired by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data.

[0042] Among them, the image sensor and the laser radar sensor are arranged on the trolley corresponding to the sling, and there are at least two image sensors and at least two laser radar sensors.

[0043] Specifically, this embodiment is used to provide an overall description of the main steps of the method for controlling the posture of the sling.

[0044] The executing entity in the embodiment of the present disclosure is a controller that controls the operation of the gantry crane and can obtain data collected by the image sensor and lidar sensor installed on the trolley of the gantry crane. Specifically, it can be a control system installed in the gantry crane, or it can be a cloud server that is communicatively connected to the control system of the gantry crane. For the convenience of explanation, they are collectively referred to as control systems here.

[0045] During the travel of the yard crane, the control system first receives data from multiple image sensors and lidar sensors installed on the trolley corresponding to the gantry crane's spreader. The image sensor is usually a high-resolution camera that can capture visual information about the spreader under the trolley and the surrounding environment of the target container corresponding to the spreader, while the lidar sensor provides accurate distance and depth information.

[0046] Since the image sensor and lidar sensor have different coordinate systems, in order to achieve effective data fusion, the radar data needs to be projected into the coordinate system of the image data. This process involves coordinate transformation and data alignment, and usually requires the use of sensor calibration parameters, including intrinsic and extrinsic parameters, to achieve accurate projection. The intrinsic parameters involve the focal length, principal point coordinates, and distortion coefficients of the sensor, while the extrinsic parameters describe the position and posture of the sensor in three-dimensional space.

[0047] Through these parameters, the three-dimensional point cloud in the radar data can be projected onto the image plane to form radar projection data. By projecting it into radar projection data, the radar data and image data can be processed uniformly.

[0048] In some embodiments, the image sensor and the laser radar sensor are arranged at the same position. If there are two image sensors and two laser radar sensors, the image sensors can be symmetrically arranged at two positions on the bottom of the vehicle that are offset from the center of the bottom of the vehicle, and the laser radar sensor and the image sensor are arranged at the same position.

[0049] S202: Input the radar projection data and the image data into a bird's-eye view fusion model, and output the posture data of the spreader and the corresponding target container.

[0050] Specifically, the radar projection data and image data after coordinate conversion are input into the bird's-eye view fusion model. The bird's-eye view fusion model (also known as the BEV fusion network model, BEV stands for Bird's-Eye View, meaning bird's-eye view, hereinafter referred to as the BEV model) is an advanced data processing model that aims to combine data from different sensors to provide a comprehensive environmental perception view. By fusing the visual information of the image and the depth information of the radar, the model can more accurately identify and locate the posture of the spreader and the target container.

[0051] Before actual application, the BEV model can be trained based on deep learning technology and a large amount of training data to ensure high-precision posture recognition in the complex environment of port logistics.

[0052] The input of the model includes preprocessed image data and radar projection data, and the output is the posture data of the spreader and container, including information such as position, angle and direction.

[0053] In some embodiments, the BEV model can be trained in combination with transfer learning technology to reduce training time and data requirements, or combined with real-time data for online learning and updating.

[0054] S203: Input the posture data into a multi-layer perceptron, and output the posture deviation of the spreader relative to the target container.

[0055] Specifically, the posture data output by the BEV model will be further input into a multi-layer perceptron (MLP) to calculate the posture deviation of the spreader relative to the target container.

[0056] MLP is used to learn the mapping relationship between input data and output results to predict the posture difference between the spreader and the container. The input posture data includes the three-dimensional position and rotation angle of the spreader and the container. MLP calculates the posture deviation from this data, which reflects the gap between the current posture of the spreader and the ideal posture required to clamp the container.

[0057] S204: Control the posture of the spreader based on the posture deviation.

[0058] Specifically, the calculated posture deviation is used to adjust the posture of the spreader in real time to ensure that it can accurately align with and clamp the target container.

[0059] In some embodiments, the control system can use advanced control algorithms such as proportional-integral-derivative (PID) controller or model predictive control (MPC) to achieve high-precision adjustment of the spreader posture. The PID controller can quickly respond to posture deviations and make corresponding adjustments by adjusting the proportional, integral and differential coefficients, while the MPC optimizes the control input to achieve the optimal control effect by predicting the future system state.

[0060] The control system needs to receive sensor data and calculation results of posture deviation in real time to ensure that the spreader can respond and adjust quickly in a dynamic environment.

[0061] The method for controlling the posture of the spreader provided in the embodiment of the present application receives the detection image taken by the monocular camera, and corrects the detection image based on the camera parameters of the monocular camera, inputs the corrected detection image into the depth estimation network, and determines the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network; based on the three-dimensional point cloud, the obstacle data within the driving range of the field bridge truck is determined and input into the bird's-eye view fusion model, and the posture data of the spreader and the corresponding target container are output; the posture data is input into the multi-layer perceptron, and the posture deviation of the spreader relative to the target container is output; based on the posture deviation, the posture of the spreader is controlled. Thus, by combining the data of the image sensor and the lidar sensor, the stability and reliability of the spreader posture control are significantly improved. By projecting the radar data into the image coordinate system and inputting it into the bird's-eye view fusion model, the posture data of the spreader and the target container can be accurately obtained. Subsequently, the posture deviation is calculated using the multi-layer perceptron to achieve precise control of the spreader posture. This multi-sensor fusion method effectively overcomes the environmental interference and lack of robustness problems of relying solely on image recognition algorithms, improves the operating accuracy and safety of spreaders in complex scenarios, and ensures the accurate grasping and placement of containers.

[0062] Figure 3 Schematic diagram of the process of the hanger attitude control provided for this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the specific process in the method for controlling the posture of the spreader is described in detail. The method includes:

[0063] S301: Perform time synchronization processing on the image sensor and the lidar sensor.

[0064] Specifically, this embodiment is used to illustrate the specific implementation method of the spreader posture control process.

[0065] Specifically, before performing the hoisting device posture control, it is necessary to debug the image sensor and the lidar sensor, including time synchronization processing and parameter calibration. Through time synchronization processing, it is ensured that the image sensor and the lidar sensor data can be fused under the same time reference.

[0066] Since the sampling frequency and data transmission delay of the two sensors may be different, unsynchronized data will lead to inconsistent information, which will affect the subsequent data fusion and attitude calculation. Time synchronization is usually achieved through a combination of hardware and software. Hardware synchronization can be achieved using a dedicated synchronization module or through a shared clock signal (for example, using GPS timing as a unified clock source for vision sensors and lidar sensors), while software synchronization can be achieved through timestamp alignment (for example, when triggering data collection, the timestamps of the two devices should be ensured to have an error of 10ms).

[0067] S302: Calibrate parameters of the image sensor and the lidar sensor.

[0068] Among them, parameter calibration is used to obtain the intrinsic parameter matrix and the extrinsic parameter matrix.

[0069] Specifically, parameter calibration is to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the image sensor and the lidar sensor to achieve accurate data fusion and realize the transformation of the coordinate system between the data of the image sensor and the lidar sensor.

[0070] The intrinsic parameter matrix describes the internal characteristics of the sensor, such as focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameter matrix describes the position and attitude of the sensor in three-dimensional space. The calibration process usually requires taking multiple images on a calibration plate of known size, from which the internal and external parameters of the sensor are calculated. For LiDAR, the calibration process may involve the use of reflective markers or specific calibration targets to obtain the position information of the LiDAR relative to the image sensor. The accuracy of the calibration directly affects the accuracy of data fusion in subsequent steps, so high-precision calibration tools and algorithms are required.

[0071] In some embodiments, in order to cope with environmental changes and device drift, regular recalibration is also necessary. Parameter calibration ensures that sensor data can be accurately projected and fused in the same coordinate system.

[0072] S303: Determine whether the reprojection errors in the laser radar sensor and the image sensor meet the set error requirements.

[0073] Specifically, due to the different types of lidar sensors and image sensors and the complex application environments in which they are located, even after calibration, it is necessary to verify whether the calibration error meets the requirements, that is, to verify the reprojection error.

[0074] Reprojection error is an important indicator for evaluating sensor calibration accuracy. Reprojection error refers to the deviation between the 3D point cloud data and the actual image data after it is projected onto the image plane. In order to ensure the accuracy of data fusion, the reprojection error must meet the set error requirements. This process is usually achieved by calculating the pixel deviation of each projection point and optimizing the calibration parameters by minimizing these deviations. If the reprojection error exceeds the set range, recalibration or adjustment of the calibration parameters is required.

[0075] In order to improve the robustness of the calibration, an optimization algorithm such as the Levenberg-Marquardt algorithm can be used for nonlinear least squares optimization to accurately adjust the internal and external parameter matrices.

[0076] In some embodiments, the reprojection error is checked not only during the calibration process of the lidar sensor and the image sensor, but also regularly during the application phase to ensure the accuracy and reliability of the calculation of the spreader posture deviation based on the two sensors.

[0077] S304: Receive image data captured by the image sensor and radar data acquired by the lidar sensor.

[0078] Specifically, in the application phase, the control system receives real-time data from image sensors and lidar sensors, which are the basis for posture calculation and control in subsequent steps.

[0079] S305. Based on the calibration parameters of the image sensor obtained by calibrating the image sensor and the laser radar sensor, the radar data is projected into a coordinate system corresponding to the image data to obtain corresponding radar projection data.

[0080] The calibration parameters include an intrinsic parameter matrix and an extrinsic parameter matrix of the image sensor, and the extrinsic parameter matrix includes a rotation matrix and a translation matrix used to project radar data to a coordinate system where image data is located.

[0081] Specifically, the lidar data can be projected into the coordinate system of the image data using the intrinsic and extrinsic matrix of the image sensor obtained previously. This process involves converting the three-dimensional point cloud data into points on the two-dimensional image plane to achieve effective data fusion.

[0082] The calibration parameters include the extrinsic matrix and the intrinsic matrix. The extrinsic matrix includes the rotation matrix and the translation matrix. The intrinsic matrix is ​​used to correct the image distortion and adjust the focal length, while the extrinsic matrix is ​​used to convert the radar data from its local coordinate system to the image coordinate system. The rotation matrix is ​​used to adjust the direction of the radar data, and the translation matrix is ​​used to adjust its position. Through the operation of these matrices, the radar data can be accurately projected onto the image plane to form radar projection data.

[0083] Exemplarily, the formula for projecting the point cloud corresponding to the radar data onto the coordinate system corresponding to the image data through the external parameter matrix, the internal parameter matrix, and the radar data can be expressed as:

[0084] ;

[0085] Among them, u and v are the two-dimensional coordinates of the image data, x, y, and z are the three-dimensional point cloud coordinates of the radar data, K is the camera internal parameter, and R and T are the rotation matrix and translation matrix from the lidar to the camera, respectively.

[0086] In some embodiments, in order to improve the accuracy of the projection, an optimization algorithm can be used to correct the projection process, or a comprehensive analysis can be performed in combination with multi-view data to ensure that the radar data and the image data are fused in the same coordinate system, providing a unified basis for subsequent posture calculations.

[0087] S306: Input the radar projection data and the image data into the bird's-eye view fusion model, and output the posture data of the spreader and the corresponding target container.

[0088] Specifically, the radar projection data and image data after projection processing are input into the bird's-eye view fusion model to obtain the posture data of the spreader and the target container.

[0089] In some embodiments, if the radar data is abnormal, the image data is input into a bird's-eye view fusion model, and the posture data of the spreader and the corresponding target container are output; or, if the image data is abnormal, the radar projection data is input into a bird's-eye view fusion model, and the posture data of the spreader and the corresponding target container are output.

[0090] Specifically, if the radar data is abnormal, the model can rely solely on image data for calculation; vice versa, this redundant design improves the robustness of the system.

[0091] In some embodiments, the posture data includes the three-dimensional coordinates of the center point of the spreader, size information, rotation angle, the three-dimensional coordinates of the center point of the target container, size information, rotation angle and container size type.

[0092] Exemplarily, the posture data of the spreader and the target container can be expressed as:

[0093] ;

[0094] Among them, X, Y, Z are the center points of the target (such as a spreader or a target container), L, W, H are the length, width, and height of the target. θ is the yaw angle of the target, and ID is the target category. The target categories include 20-foot containers, 40-foot containers, 45-foot containers (the target container category is one of these three. In actual applications, other categories can also be included, which only requires the administrator to pre-configure) and spreaders.

[0095] S307, input the horizontal plane coordinates of the center point of the spreader and the target container in the posture data, the dimension information parallel to the horizontal plane and the rotation angle into the multi-layer perceptron, and output the position deviation between the spreader and the target container.

[0096] Specifically, the posture data output by the bird's-eye view fusion model is input into a multi-layer perceptron (MLP) to calculate the position deviation between the spreader and the target container in three directions (such as parallel to the driving direction of the crane corresponding to the spreader, parallel to the driving direction of the trolley corresponding to the spreader, and perpendicular to the horizontal plane, or, it can also be two mutually perpendicular moving directions of the spreader in a plane parallel to the driving direction of the crane corresponding to the spreader and perpendicular to the driving direction of the trolley).

[0097] In some embodiments, the multilayer perceptron may be a three-layer structure, wherein the input layer has ten nodes for inputting the above-mentioned types of input data, and the output layer has three nodes, i.e., position deviations in three directions.

[0098] In some embodiments, image samples labeled with the center of standard 3D target box samples can be used as training data to train the MLP model.

[0099] In some embodiments, the specific deviation direction can be two directions with an angle in a plane parallel to the travel direction of the crane corresponding to the crane and perpendicular to the travel direction of the crane (such as the travel direction of the crane corresponding to the trolley and the movement direction of the crane relative to the trolley).

[0100] S308. Based on a first deviation value in the posture deviation that is parallel to the driving direction of the yard crane corresponding to the hoist, a first control instruction is sent to the yard crane.

[0101] The first control instruction is used to control the movement of the field crane vehicle based on the first deviation value.

[0102] Specifically, the calculated posture deviation is used to generate control instructions to control the movement of the crane, trolley and spreader. Specifically, the system generates a first control instruction based on the first deviation value in the posture deviation that is parallel to the travel direction of the crane corresponding to the spreader. This instruction is used to adjust the motion parameters of the crane, such as speed and direction, to correct the deviation and ensure that the spreader can accurately align with the target container.

[0103] S309: Based on the second deviation value and the third deviation value in the posture deviation corresponding to the moving direction of the spreader, send a second control instruction to the trolley.

[0104] The second control instruction is used to control the trolley to adjust the position of the sling based on the second deviation value and the third deviation, and the corresponding directions of the second deviation value and the third deviation value are perpendicular to each other.

[0105] Specifically, the system can adjust the relative positions of the trolley and the spreader according to the second deviation value and the third deviation value in the posture deviation to correct the deviation and ensure that the spreader can move along the correct path.

[0106] In some embodiments, the second deviation value may be a deviation of the sling relative to the driving direction of the trolley and a deviation of the sling relative to the moving direction of the trolley.

[0107] The spreader posture control method provided by the disclosed embodiment significantly improves the accuracy and reliability of data fusion through time synchronization and parameter calibration of image sensors and lidar sensors. By ensuring that the reprojection error is within the set range, the system can achieve high-precision sensor data alignment, thereby improving the accuracy of posture recognition. Combined with the bird's-eye view fusion model and the multi-layer perceptron, the system can effectively identify and calculate the posture deviation between the spreader and the container in a complex environment. By generating precise control instructions, the system can adjust the movement of the yard crane trolley and the trolley in real time to ensure the precise operation of the spreader. This solution not only improves the stability and safety of the spreader posture control, but also enhances the system's adaptability in a dynamic environment and optimizes the overall efficiency of port logistics.

[0108] Figure 4 The schematic diagram of the structure of the hanger attitude control device provided in this application is as follows: Figure 4 As shown, the hanger posture control device 400 provided in this embodiment includes:

[0109] The receiving module 410 is used to receive the image data captured by the image sensor and the radar data acquired by the laser radar sensor, and project the radar data into the coordinate system corresponding to the image data to obtain the corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on the trolley corresponding to the spreader, and there are at least two image sensors and at least two laser radar sensors;

[0110] The processing module 420 is used to input the radar projection data and the image data into the bird's-eye view fusion model and output the posture data of the spreader and the corresponding target container;

[0111] An analysis module 430, for inputting the posture data into a multi-layer perceptron and outputting a posture deviation of the spreader relative to the target container;

[0112] The output module 440 is used to control the posture of the spreader based on the posture deviation.

[0113] In one possible implementation, the receiving module 410 is specifically used to receive image data captured by an image sensor and radar data acquired by a lidar sensor; based on calibration parameters of the image sensor obtained by calibrating the image sensor and the lidar sensor, the radar data is projected into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the calibration parameters include an intrinsic parameter matrix and an extrinsic parameter matrix of the image sensor, and the extrinsic parameter matrix includes a rotation matrix and a translation matrix for projecting the radar data into the coordinate system where the image data is located.

[0114] In one possible implementation, the receiving module 410 is also used to receive image data captured by the image sensor and radar data acquired by the lidar sensor, and project the radar data into a coordinate system corresponding to the image data, and perform time synchronization processing on the image sensor and the lidar sensor before obtaining the corresponding radar projection data; perform parameter calibration on the image sensor and the lidar sensor, and the parameter calibration is used to obtain an intrinsic parameter matrix and an extrinsic parameter matrix.

[0115] In a possible implementation, the receiving module 410 is further used to determine whether the reprojection errors in the laser radar sensor and the image sensor meet the set error requirements after calibrating the parameters of the image sensor and the laser radar sensor.

[0116] In one possible implementation, the processing module 420 is specifically used to, if the radar data is abnormal, input the image data into the bird's-eye view fusion model, and output the posture data of the spreader and the corresponding target container; or, if the image data is abnormal, input the radar projection data into the bird's-eye view fusion model, and output the posture data of the spreader and the corresponding target container.

[0117] In a possible implementation, the processing module 420 specifically includes that the posture data includes the three-dimensional coordinates of the center point of the spreader, size information, rotation angle, the three-dimensional coordinates of the center point of the target container, size information, rotation angle and container size type.

[0118] In a possible implementation, the analysis module 430 is specifically used to input the horizontal plane coordinates of the center points of the spreader and the target container in the posture data, the dimension information parallel to the horizontal plane, and the rotation angle into a multi-layer perceptron, and output the position deviation between the spreader and the target container.

[0119] In a possible implementation, the output module 440 is specifically used to send a first control instruction to the bridge trolley based on a first deviation value in the posture deviation that is parallel to the driving direction of the bridge trolley corresponding to the spreader, and the first control instruction is used to control the movement of the bridge trolley based on the first deviation value; based on the second deviation value and the third deviation value in the posture deviation that correspond to the moving direction of the spreader, send a second control instruction to the trolley, and the second control instruction is used to control the trolley to adjust the position of the spreader based on the second deviation value and the third deviation, and the corresponding directions of the second deviation value and the third deviation value are perpendicular to each other.

[0120] The sling posture control device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be described in detail here.

[0121] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0122] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0123] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0124] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0125] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0126] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0127] like Figure 6 As shown, it is a schematic diagram of the structure of the field bridge provided by the present application. The present application also provides a field bridge, the field bridge 60 includes Figure 4 The specific implementation process of the sling attitude control device 610 in the illustrated embodiment can be referred to the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment. The present application also provides a computer program product, including a computer program, which implements the method in any of the above embodiments when executed by a processor.

[0128] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in any of the above embodiments is implemented.

[0129] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0130] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0131] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0134] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0137] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0138] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0139] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0140] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0143] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for controlling the posture of a sling, characterized in that: The steps include: Receiving image data captured by an image sensor and radar data acquired by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a trolley corresponding to the sling, and there are at least two image sensors and at least two laser radar sensors; Inputting the radar projection data and the image data into a bird's-eye view fusion model, and outputting posture data of the spreader and the corresponding target container; Inputting the posture data into a multi-layer perceptron to output a posture deviation of the spreader relative to the target container; Based on the posture deviation, the posture of the spreader is controlled.

2. The method according to claim 1, characterized in that The receiving of image data captured by the image sensor and radar data acquired by the laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, includes: Receive image data captured by the image sensor and radar data acquired by the lidar sensor; Based on the calibration parameters of the image sensor obtained by calibrating the image sensor and the lidar sensor, the radar data is projected into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the calibration parameters include an intrinsic parameter matrix and an extrinsic parameter matrix of the image sensor, and the extrinsic parameter matrix includes a rotation matrix and a translation matrix for projecting the radar data into the coordinate system where the image data is located.

3. The method according to claim 2, characterized in that The receiving of the image data captured by the image sensor and the radar data acquired by the laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain the corresponding radar projection data, further includes: Performing time synchronization processing on the image sensor and the laser radar sensor; Parameter calibration is performed on the image sensor and the lidar sensor, and the parameter calibration is used to obtain the intrinsic parameter matrix and the extrinsic parameter matrix.

4. The method according to claim 3, characterized in that After calibrating the parameters of the image sensor and the lidar sensor, the method further includes: It is determined that the reprojection error in the laser radar sensor and the image sensor meets the set error requirement.

5. The method according to claim 1, characterized in that The step of inputting the radar projection data and the image data into a bird's-eye view fusion model and outputting the posture data of the spreader and the corresponding target container further includes: If the radar data is abnormal, the image data is input into a bird's-eye view fusion model to output the posture data of the spreader and the corresponding target container; or, If the image data is abnormal, the radar projection data is input into a bird's-eye view fusion model, and the posture data of the spreader and the corresponding target container are output.

6. The method according to any one of claims 1 to 5, characterized in that The posture data includes the three-dimensional coordinates of the center point of the spreader, size information, rotation angle, the three-dimensional coordinates of the center point of the target container, size information, rotation angle and container size type.

7. The method according to claim 6, characterized in that The step of inputting the posture data into a multi-layer perceptron and outputting a posture deviation of the spreader relative to the target container comprises: The horizontal plane coordinates of the center points of the spreader and the target container in the posture data, the dimension information parallel to the horizontal plane and the rotation angle are input into a multi-layer perceptron, and the position deviation between the spreader and the target container is output.

8. The method according to claim 7, characterized in that The controlling of the posture of the spreader based on the posture deviation comprises: Based on a first deviation value in the posture deviation that is parallel to the driving direction of the field bridge trolley corresponding to the spreader, a first control instruction is sent to the field bridge trolley, wherein the first control instruction is used to control the field bridge trolley to move based on the first deviation value; Based on the second deviation value and the third deviation value corresponding to the moving direction of the sling in the posture deviation, a second control instruction is sent to the trolley, and the second control instruction is used to control the trolley to adjust the position of the sling based on the second deviation value and the third deviation, and the corresponding directions of the second deviation value and the third deviation value are perpendicular to each other.

9. A sling attitude control device, characterized in that: include: A receiving module, used for receiving image data captured by an image sensor and radar data acquired by a laser radar sensor, and projecting the radar data into a coordinate system corresponding to the image data to obtain corresponding radar projection data, wherein the image sensor and the laser radar sensor are arranged on a trolley corresponding to the sling, and there are at least two image sensors and at least two laser radar sensors; A processing module, used for inputting the radar projection data and the image data into a bird's-eye view fusion model, and outputting posture data of the spreader and the corresponding target container; An analysis module, used for inputting the posture data into a multi-layer perceptron, and outputting a posture deviation of the spreader relative to the target container; An output module is used to control the posture of the spreader based on the posture deviation.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A field bridge, characterized in that: The field bridge includes the spreader attitude control device as described in claim 9.

Images