A method for anti-collision and identification of rubber-tyred gantry crane integrating GNSS differential data

Abstract

The present invention relates to a method for anti-collision and identification of a rubber-tyred gantry crane integrating GNSS differential data. By installing sensors on the rubber-tyred gantry crane, highly automated and precise control of the rubber-tyred gantry crane operation is achieved, improving the operation efficiency and safety. GNSS signals are collected, and after processing, GNSS differential data and lane line information are obtained. The data received from the 3D lidar and binocular camera are filtered and denoised, improving the positioning accuracy and data quality, and providing a reliable basis for subsequent data processing. Deep learning technology is used to process the data collected by the binocular camera, extract obstacle and lane line information, and combine with the dynamic model to predict the motion state of the rubber-tyred gantry crane, improving the prediction accuracy of the system and the perception ability of the environment. Through 3D vision image recognition technology and point cloud data processing, the position and type of obstacles are identified, and a warning is issued when a potential collision is detected, improving the safety and reducing the occurrence of accidents.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile crane monitoring, and particularly relates to an anti-collision and identification method for mobile cranes integrating GNSS differential data. Background Art

[0002] Due to their strong mobility and wide adaptability, mobile cranes are widely used in container handling operations at ports and terminals. They can move flexibly within the terminal area for container stacking and handling operations.

[0003] After retrieval, a mobile crane anti-collision control system based on the GNSS positioning system with the publication number CN101464688B was disclosed on May 18, 2011. This product uses components such as a GNSS reference station subsystem, a TOS interface subsystem, a GNSS mobile station subsystem, and a central subsystem. Although it can achieve anti-collision control based on GNSS positioning, due to its dependence on GNSS positioning technology, it is easily affected by satellite signal interference and multipath effects, resulting in insufficient positioning accuracy and reliability under complex terrains and changing working conditions.

[0004] In addition, in the prior art, the following problems also exist:

[0005] 1. High dependence on manual operation: The existing mobile crane deviation correction mainly relies on the driver's manual operation. The driver needs to constantly monitor the position of the mobile crane in the lane, resulting in high labor intensity, low work efficiency, and being prone to safety accidents due to misjudgment or improper operation; Poor environmental adaptability: The ground marking and image recognition deviation correction methods cannot work properly in rainy and snowy weather, and the lack of lane lines has a great impact on recognition interference. The infrared or photoelectric ranging deviation correction methods are severely affected by external light interference and are difficult to correctly reflect the actual position of the equipment.

[0006] 2. Technical limitations: The anti-collision method based on two-dimensional and three-dimensional lidar has limited obstacles that can be protected. They must be high enough and within the detection plane, and are easily affected by noise points, with poor environmental adaptability and frequent false alarms.

[0007] 3. Sensor fusion problem: Although there is a mobile crane stereo anti-collision method based on the fusion of three-dimensional lidar and binocular cameras, how to effectively fuse the data of the two sensors to improve the automation degree and accuracy of detection is still a technical challenge.

[0008] 4. False alarm and missed detection problems: The data provided by a single sensor has a single feature, resulting in high false detection rates and missed detection rates, reducing the applicability and accuracy of the anti-collision function of mobile cranes.

[0009] The above problems indicate that the existing anti-collision and control products for rubber-tyred gantry cranes in the market are difficult to effectively meet the new requirements for high reliability and real-time response under complex terrains and changing working conditions. Therefore, the present invention provides a method for anti-collision and identification of rubber-tyred gantry cranes integrating GNSS differential data to overcome these deficiencies and provide a new solution that is more intelligent, efficient and adaptable to changing environments. Summary of the Invention

[0010] The present invention provides a method for anti-collision and identification of rubber-tyred gantry cranes integrating GNSS differential data, which solves the problems of unstable performance, complex operation, slow reaction speed, insufficient adaptability and reliability of existing anti-collision products for rubber-tyred gantry cranes under complex terrains and changing working conditions. The purpose of the present invention is to provide a new solution that is more intelligent, efficient and adaptable to changing environments to effectively improve the safety and reliability of rubber-tyred gantry cranes under complex terrains and changing working conditions.

[0011] The technical solution adopted by the present invention to solve the above technical problems is: a method for anti-collision and identification of rubber-tyred gantry cranes integrating GNSS differential data, including the following steps:

[0012] S1: System initialization and sensor integration, install and integrate a GNSS receiver, a 3D lidar, a binocular camera and an IMU on the rubber-tyred gantry crane, and ensure their synchronization with the control system of the rubber-tyred gantry crane;

[0013] S2: Data acquisition and preprocessing, collect GNSS signals, and obtain GNSS differential data and lane line information after processing, and perform filtering and denoising processing on the data received by the 3D lidar and the binocular camera;

[0014] S3: Sensor data fusion and processing, use deep learning technology to process the data collected by the binocular camera, extract obstacle and lane line information, and combine the dynamic model to predict the motion state of the rubber-tyred gantry crane to improve the prediction accuracy of the system;

[0015] S4: Collision detection and warning, identify the position and type of obstacles through 3D vision image recognition technology and point cloud data processing, and issue a warning when a potential collision is detected, and store the image and position information of the obstacle target for post-event analysis;

[0016] S5: Automatic deviation correction control, automatically adjust the driving direction and speed of the rubber-tyred gantry crane according to GNSS differential data and lane line information to correct the deviation;

[0017] S6: Dynamic path planning, simulate the motion state of the rubber-tyred gantry crane and the surrounding environment, and dynamically plan the optimal driving path;

[0018] S7: System feedback and optimization, collect system operation data, optimize the system, improve response speed and accuracy. At the same time, use the AI fusion method to reduce the number of sensors, reduce the time for automatic container loading, and improve the success rate of the first container loading;

[0019] S8: Emergency handling, when the system detects an emergency, automatically activate the emergency braking system to prevent collisions, and re-evaluate the environment and plan a new safe path after the emergency is handled.

[0020] Preferably, in S1, it specifically includes the following steps:

[0021] Sect. 1.1: The GNSS receiver is installed on the top of the rubber-tyred gantry crane, the 3D lidar and the binocular camera are installed on the front and side of the rubber-tyred gantry crane respectively, and the IMU is installed at the central position of the rubber-tyred gantry crane;

[0022] Sect. 1.2: Use a programmable logic controller (PLC) to coordinate the data of each sensor to achieve automatic control. The PLC is connected to each sensor through wired or wireless communication methods to ensure real-time data transmission.

[0023] Preferably, in S2, it specifically includes the following steps:

[0024] Sect. 2.1: Collect GNSS signals and combine with the ground base station to achieve centimeter-level positioning. The ground base station is connected to the GNSS receiver on the rubber-tyred gantry crane through wireless communication to provide GNSS differential data and lane line information in real time;

[0025] Sect. 2.2: Filter and denoise the data collected by the 3D lidar and the binocular camera. The filtering algorithm is as follows:

[0026] ;

[0027] Where, in the above formula is the state vector. For the 3D lidar, it mainly includes distance and angle information. For the binocular camera, it mainly includes the position and speed of the target in the image; is the posterior state estimate at time step That is, the optimal estimate of the system state after obtaining the actual measurement value at time k; is the prior state estimate at time step That is, before obtaining the actual measurement value at time k, the state predicted according to the state estimate at time k - 1 and the system model; K k Adjustment factor, used to determine the weight between the measurement value and the predicted value; Z k is the actual measurement value at time step k, obtained from the data collected by the 3D lidar and the binocular camera; is the observation matrix that maps the state variables to the measurement space and determines how the state variables affect the measurement values.

[0028] Preferably, in S3, it specifically includes:

[0029] Use the video detection and segmentation technology of deep learning to process the video data collected by the binocular camera, and extract the information of obstacles and lane lines in the environment. Specifically, the YOLO detection model is used in deep learning, and its loss function is:

[0030] ;

[0031] Among them, L in the above formula is the total loss. The smaller the value of the total loss L, the closer the predicted obstacles and lane lines of the model are to the actually labeled obstacles and lane lines, and the better the performance of the model; is a weight parameter used to balance the contribution of the coordinate loss in the total loss; it controls the influence degree of the accuracy of the bounding box position prediction on the overall loss; represents the number of bounding boxes predicted by each grid cell in the object detection task. Different grid cells correspond to predicting different numbers of bounding boxes, and different numbers of bounding boxes correspond to dynamic scenes and obstacles; B represents the maximum number of bounding boxes that can be predicted in each grid cell. To cover multiple possible obstacles and lane lines, B the value of needs to be large enough; is an indicator function. When there is an object in the grid cell and the bounding box , its value is 1, otherwise it is 0, so that only the bounding boxes containing the information of obstacles and lane lines contribute to the loss function; and respectively represent the true values of the x coordinate and y coordinate of the center of the bounding box; and respectively represent the predicted values of the x coordinate and y coordinate of the center of the bounding box; and respectively represent the true values of the width and height of the bounding box, and help the model understand the sizes of obstacles and lane lines; and respectively represent the predicted values of the width and height of the bounding box; is another weight parameter to identify the areas without obstacles or lane lines; represents the true value of the confidence of the existence of an object in the bounding box; represents the predicted value of the confidence of the existence of an object in the bounding box; represents the true value of the probability that the bounding box belongs to the category c ; Indicates that the bounding box belongs to the category c classes is the total number of obstacle and lane line information categories.

[0032] Preferably, S3 further includes:

[0033] The dynamic model technology is combined to model and predict the motion state of the tire crane, predict the future position and driving trajectory of the tire crane, and improve the prediction accuracy of the system. The GRU model is specifically used, and its equation is as follows:

[0034] ;

[0035] Among them, To update the value of the gate, it determines how much proportion of the hidden state at the previous moment The hidden state that will be retained until the current moment middle; is the weight matrix of the update gate, is the bias term of the update gate, and σ is the sigmoid activation function, which limits the output value to between 0 and 1; To reset the value of the gate, it determines how much of the previous moment's hidden state will be reset to 0; is the weight matrix of the reset gate, is the bias term for the reset gate; Is the value of the candidate hidden state, which is a candidate for a new hidden state and is partially or completely used to update the hidden state at the current moment ; is the weight matrix of the candidate hidden states, is the bias term for the candidate hidden state; tanh is the hyperbolic tangent activation function, which limits the output value to between -1 and 1; the symbol ⊙ represents the Hadamard product (element-wise multiplication); According to the value of the update gate To update the hidden state at the current moment , is the hidden state vector at time step t, which represents the internal state of the tire crane and contains the predicted position and speed information of the tire crane; is the input vector at time step t, which contains the current speed, acceleration, and direction information of the tire crane.

[0036] Preferably, in S4, the following steps are specifically included:

[0037] S4.1: Solve the obstacle avoidance and warning problems of large mechanical equipment such as tire cranes through 3D visual image recognition technology. After the data collected by the 3D laser radar and binocular camera are fused, a 3D point cloud image is generated. The location and type of obstacles are identified through point cloud clustering and segmentation algorithms. The point cloud clustering uses the DBSCAN algorithm, and its parameters include the minimum number of points. and neighborhood radius minPts;

[0038] S4.2: Realize the storage of obstacle target alarm images to provide traceability for accident analysis. The image and location information of the obstacle target are stored in the vehicle memory for post-analysis and troubleshooting.

[0039] Preferably, in S5, the following steps are specifically included:

[0040] S5.1: Automatically adjust the travel direction and speed of the RTG to correct deviations based on GNSS differential data and lane information. The CPU calculates the RTG's current position and deviation based on GNSS differential data and lane information, and adjusts the RTG's travel direction and speed using a PID controller.

[0041] S5.2: Use the BP neural network to identify the images captured by the binocular camera, simplify them into discrete points, and perform line detection using a hybrid Hough transform and least squares method to develop a correction strategy.

[0042] Preferably, in S6, the steps include:

[0043] The trajectory and position of the container are predicted and simulated, the optimal path is dynamically planned, the movement state and surrounding environment of the tire crane are simulated in real time, and the optimal driving path is calculated through the optimization algorithm. The cost function is as follows:

[0044] ;

[0045] in, The total cost of the node, which means the total cost of the node from the starting point to the target point The estimated cost of , in the scenario where the digital twin simulates a rubber-tyred crane, this represents the total cost from the current position of the rubber-tyred crane to the target position;

[0046] In the above formula The actual cost from the starting point to the node, which is specifically the actual driving distance or time cost of the tire crane moving from the initial position to the current node position;

[0047] In the above formula $g(n)$ is the estimated cost from the current node to the target node. This is a heuristic estimate used to predict the cost of the shortest path from the current node to the target node. Specifically, it is based on the shortest estimated distance from the current position of the RTG to the target position, and the Euclidean distance or Manhattan distance is used for calculation;

[0048] In the above formula, $n$ represents the current node. Each node represents the state of the RTG at a specific position and time, including its position coordinates, speed, and direction state information.

[0049] Preferably, in S7, it specifically includes the following steps:

[0050] S7.1: Collect system operation data, optimize the system, and improve the system's response speed and accuracy. The data collection module records sensor data and control instructions in real time, and improves the system's performance and reliability through big data analysis and optimization algorithms;

[0051] S7.2: Use the AI fusion method to reduce the number of spreader sensors, reduce the time for automatic container landing, and improve the success rate of the first container landing. The AI fusion method improves the system's perception ability and decision-making accuracy through multi-modal data fusion.

[0052] Preferably, in S8, it specifically includes the following steps:

[0053] S8.1: When the system detects an emergency, automatically activate the emergency braking system to prevent collisions. The emergency braking system includes a braking control unit and a brake. The braking control unit quickly activates the brake according to the instructions of the central processor to ensure the safety of the RTG in an emergency;

[0054] S8.2: After the emergency is handled, the system will re-evaluate the environment and plan a new safe path. The re-evaluation module recalculates the new driving path by re-collecting environmental data to ensure the safety and efficiency of the RTG.

[0055] Advantages of the present invention:

[0056] Through the integration of various sensors, the application of deep learning technology, dynamic models, optimization algorithms, etc., the present invention realizes the automation, intelligence, and safety of RTG operations. Specifically, this method improves the positioning accuracy, data quality, prediction accuracy, obstacle detection ability, path planning efficiency, system response speed and accuracy of the RTG, as well as the safety and emergency response ability in case of emergencies. Through the integrated application of these technologies, the operation efficiency and safety of the RTG are significantly improved, the operation cost is reduced, and the reliability and economy of the system are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a schematic diagram of the overall structure of the present invention. Detailed implementation manners

[0058] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0059] It should be noted that if terms such as "first" and "second" are involved in the description, claims and the above-mentioned drawings of this application, they are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances for the embodiments of this application described herein. In addition, if terms such as "comprising" and "having" and any variations thereof are involved, the intention is to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0060] In this application, if terms such as "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal", etc. are involved, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the drawings. These terms are mainly used to better describe this application and its embodiments, and are not used to limit that the indicated device, element or component must have a specific orientation or be constructed and operated in a specific orientation.

[0061] Moreover, in addition to being able to represent an orientation or positional relationship, some of the above terms may also be used to represent other meanings. For example, the term "upper" may also be used to represent a certain attachment relationship or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0062] In addition, in this application, if terms such as "installation", "setting", "provided with", "connection", "connected", "socketed", etc. are involved, they should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or there can also be internal communication between two devices, elements or components. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0063] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the accompanying drawings and combine with the embodiments to elaborate on this application in detail.

[0064] By providing a method for anti-collision and identification of rubber-tyred gantry cranes that integrates GNSS differential data in an embodiment of the present invention, the problems of unstable performance, complex operation, slow reaction speed, insufficient adaptability and reliability of existing rubber-tyred gantry crane anti-collision products in complex terrains and variable working conditions are solved. The present invention aims to provide a new solution that is more intelligent, efficient and adaptable to variable environments, so as to effectively improve the safety and reliability of rubber-tyred gantry cranes in complex terrains and variable working conditions.

[0065] In order to better understand the above technical solution, the following will elaborate on the above technical solution in detail in combination with the drawings of the specification and specific implementation manners.

[0066] Refer to Figure 1 , a method for anti-collision and identification of rubber-tyred gantry cranes that integrates GNSS differential data, includes the following steps:

[0067] S1: System initialization and sensor integration. Install and integrate a GNSS receiver, a 3D lidar, a binocular camera and an IMU on the rubber-tyred gantry crane, and ensure their synchronization with the control system of the rubber-tyred gantry crane; specifically including:

[0068] S1.1: The GNSS receiver is installed on the top of the rubber-tyred gantry crane, the 3D lidar and the binocular camera are respectively installed on the front and side of the rubber-tyred gantry crane, and the IMU is installed at the central position of the rubber-tyred gantry crane;

[0069] S1.2: Use a programmable logic controller (PLC) to coordinate the data of each sensor to achieve automatic control. The PLC is connected to each sensor through wired or wireless communication methods to ensure real-time data transmission;

[0070] In this embodiment, by installing and integrating a GNSS receiver, a 3D lidar, a binocular camera and an IMU on the rubber-tyred gantry crane and ensuring their synchronization with the control system of the rubber-tyred gantry crane, highly automated and precise control of the rubber-tyred gantry crane operation is achieved, improving the operation efficiency and safety.

[0071] S2: Data acquisition and preprocessing. Acquire GNSS signals, and after processing, obtain GNSS differential data and lane line information, and perform filtering and denoising processing on the data received by the 3D lidar and the binocular camera; specifically including the following steps:

[0072] S2.1: Collect GNSS signals and achieve centimeter-level positioning in combination with a ground base station. The ground base station is connected to the GNSS receiver on the RTG through wireless communication to provide GNSS differential data and lane line information in real time.

[0073] S2.2: Filter and denoise the data collected by the 3D lidar and binocular camera. The filtering algorithm is as follows:

[0074] ;

[0075] where is the state vector. For the 3D lidar, it mainly includes distance and angle information. For the binocular camera, it mainly includes the position and speed of the target in the image. is the posterior state estimate at time step , that is, the optimal estimate of the system state after obtaining the actual measurement value at time k. is the prior state estimate at time step , that is, the state predicted according to the state estimate at time k - 1 and the system model before obtaining the actual measurement value at time k; K k is the adjustment factor used to determine the weight between the measurement value and the predicted value; Z k is the actual measurement value at time step k, obtained from the data collected by the 3D lidar and binocular camera. is the observation matrix that maps the state variable to the measurement space and determines how the state variable affects the measurement value.

[0076] In this embodiment, this equation combines the predicted state estimate and the actual measurement value to generate a more accurate state estimate and quantify the uncertainty of the estimate. In this way, the noise in the data of the 3D lidar and binocular camera can be effectively reduced, providing more reliable data for subsequent processing. Collecting GNSS signals and achieving centimeter-level positioning in combination with a ground base station improves the positioning accuracy, provides accurate basic data for subsequent path planning and obstacle detection, and filters and denoises the data collected by the 3D lidar and binocular camera, reducing the noise in the data and improving the reliability of the data, providing high-quality input for subsequent data processing.

[0077] S3: Sensor data fusion and processing. Use deep learning technology to process the data collected by the binocular camera, extract obstacle and lane line information, and combine the dynamic model to predict the motion state of the RTG to improve the prediction accuracy of the system. Specifically, it includes:

[0078] The video data collected by the binocular camera is processed using video detection and segmentation technology based on deep learning to extract information about obstacles and lane lines in the environment. Specifically, the YOLO detection model is adopted for deep learning, and its loss function is as follows:

[0079] ;

[0080] where L in the above formula is the total loss. The smaller the value of the total loss L, the closer the predicted obstacles and lane lines of the model are to the actually labeled obstacles and lane lines, and the better the performance of the model; is a weight parameter used to balance the contribution of the coordinate loss to the total loss; it controls the influence degree of the accuracy of the predicted bounding box position on the overall loss; represents the number of bounding boxes predicted by each grid cell in the object detection task. Different grid cells predict different numbers of bounding boxes, and different numbers of bounding boxes correspond to dynamically changing scenes and obstacles; B represents the maximum number of bounding boxes that can be predicted in each grid cell. To cover multiple possible obstacles and lane lines, B the value of needs to be large enough; is an indicator function. When there is an object in the grid cell and the bounding box and respectively represent the true values of the x - coordinate and y - coordinate of the center of the bounding box; and respectively represent the predicted values of the x - coordinate and y - coordinate of the center of the bounding box; and respectively represent the true values of the width and height of the bounding box, and help the model understand the sizes of obstacles and lane lines; and respectively represent the predicted values of the width and height of the bounding box; is another weight parameter to identify areas without obstacles or lane lines; represents the true value of the confidence of the presence of an object in the bounding box; represents the predicted value of the confidence of the presence of an object in the bounding box; represents the true value of the probability that the bounding box belongs to the class c ; represents the predicted value of the probability that the bounding box belongs to the class c . classes is the total number of categories of obstacle and lane line information.

[0081] In this embodiment, the video detection and segmentation technology of deep learning is used to process the video data collected by the binocular camera, extract the information of obstacles and lane lines in the environment, improve the accuracy of obstacle detection and the recognition ability of lane lines, and adopt the YOLO detection model. The model performance is optimized through a specific loss function, and the accuracy and efficiency of target detection are improved.

[0082] In step S3, it further includes:

[0083] Combined with the kinetic model technology, the motion state of the rubber-tyred gantry crane is modeled and predicted, and the future position and driving trajectory of the rubber-tyred gantry crane are predicted to improve the prediction accuracy of the system. Specifically, the GRU model is adopted, and its equation is as follows:

[0084] ;

[0085] where, in the above formula is the value of the update gate, which determines what proportion of the previous hidden state will be retained to the current hidden state ; is the weight matrix of the update gate, is the bias term of the update gate, and σ is the sigmoid activation function, which limits the output value between 0 and 1; is the value of the reset gate, which determines what proportion of the previous hidden state will be reset to 0; is the weight matrix of the reset gate, is the bias term of the reset gate; is the value of the candidate hidden state, which is a candidate for a new hidden state and is partially or fully used to update the current hidden state ; is the weight matrix of the candidate hidden state, is the bias term of the candidate hidden state; tanh is the hyperbolic tangent activation function, which limits the output value between -1 and ǀ; the symbol ⊙ represents the Hadamard product (element-wise multiplication); According to the value of the update gate to update the current hidden state , is the hidden state vector at time step t, representing the internal state of the rubber-tyred gantry crane, including the predicted position and speed information of the rubber-tyred gantry crane; is the input vector at time step t, including the current speed, acceleration, and direction information of the rubber-tyred gantry crane.

[0086] In this embodiment, the dynamic model technology is combined to model and predict the motion state of the rubber-tyred gantry crane, which improves the prediction accuracy of the system, makes the operation of the rubber-tyred gantry crane safer and more efficient. The GRU model is adopted, and through the update gate and reset gate mechanisms, it effectively processes sequence data and improves the accuracy of predicting the motion state of the rubber-tyred gantry crane.

[0087] S4: Collision detection and warning. Through 3D vision image recognition technology and point cloud data processing, the position and type of obstacles are identified, and a warning is issued when a potential collision is detected. At the same time, the images and position information of the obstacle targets are stored for post-event analysis. The specific steps are as follows:

[0088] S4.1: Solve the anti-collision and warning problems of obstacles for large-scale mechanical equipment such as rubber-tyred gantry cranes through 3D vision image recognition technology. After the data collected by the 3D lidar and binocular cameras are fused, a 3D point cloud map is generated. The position and type of obstacles are identified through point cloud clustering and segmentation algorithms. The DBSCAN algorithm is used for point cloud clustering, and its parameters include the minimum number of points and the neighborhood radius minPts;

[0089] S4.2: Implement the storage of alarm images of obstacle targets to provide traceability for accident analysis. The images and position information of the obstacle targets are stored in the vehicle-mounted memory for post-event analysis and fault troubleshooting.

[0090] In this embodiment, the 3D vision image recognition technology is used to solve the anti-collision and warning problems of obstacles for large-scale mechanical equipment such as rubber-tyred gantry cranes, which improves the accuracy of obstacle detection and the timeliness of warning. The DBSCAN algorithm is used for point cloud clustering and segmentation, which improves the ability to identify the position and type of obstacles and provides reliable data support for collision warning.

[0091] S5: Automatic deviation correction control. According to GNSS differential data and lane line information, automatically adjust the driving direction and speed of the rubber-tyred gantry crane to correct the deviation. The specific steps are as follows:

[0092] S5.1: According to GNSS differential data and lane line information, automatically adjust the driving direction and speed of the rubber-tyred gantry crane to correct the deviation. The central processor calculates the current position and deviation of the rubber-tyred gantry crane according to GNSS differential data and lane line information, and adjusts the driving direction and speed of the rubber-tyred gantry crane through a PID controller;

[0093] S5.2: Use a BP neural network to recognize the images collected by the binocular cameras, simplify them into discrete points, and perform line detection through a hybrid Hough transform and least squares method to formulate a deviation correction strategy.

[0094] In this embodiment, according to the GNSS differential data and lane line information, the traveling direction and speed of the rubber-tyred gantry crane are automatically adjusted to correct the deviation, improving the automation level and accuracy of the operation. The BP neural network is used to recognize the images collected by the binocular camera, which are simplified into discrete points, and the line detection is carried out by the method of hybrid Hough transform and least squares, improving the accuracy and efficiency of the deviation correction strategy.

[0095] S6: Dynamic path planning, simulating the motion state and the surrounding environment of the rubber-tyred gantry crane, and dynamically planning the optimal traveling path; specifically including:

[0096] Predict and simulate the landing trajectory and position, dynamically plan the optimal path, real-time simulate the motion state and the surrounding environment of the rubber-tyred gantry crane, calculate the optimal traveling path through the optimization algorithm, and its cost function is as follows:

[0097] ;

[0098] Where, The total cost of the node, representing the estimated cost from the starting point to the target point passing through the node In the scenario of digital twin simulating the rubber-tyred gantry crane, this represents the total cost from the current position of the rubber-tyred gantry crane to the target position;

[0099] In the above formula, Is the actual cost from the starting point to the node, which is specifically the actual traveling distance or time cost for the rubber-tyred gantry crane to move from the initial position to the current node position;

[0100] In the above formula, Is the estimated cost from the node to the target point, which is a heuristic estimate used to predict the shortest path cost from the current node to the target node. Specifically, it is based on the shortest estimated distance from the current position of the rubber-tyred gantry crane to the target position, and the Euclidean distance or Manhattan distance is used for calculation;

[0101] In the above formula, Represents the current node, and each node represents the state of the rubber-tyred gantry crane at a specific position and time, including its position coordinates, speed, and direction state information.

[0102] In this embodiment, the landing trajectory and position are predicted and simulated, and the optimal path is dynamically planned, improving the efficiency and adaptability of the path planning, making the operation of the rubber-tyred gantry crane safer and more efficient. By calculating the optimal traveling path through the optimization algorithm, the accuracy and efficiency of the path planning are improved.

[0103] S7: System feedback and optimization, collecting the system operation data, optimizing the system, and improving the response speed and accuracy. At the same time, using the AI fusion method to reduce the number of sensors, reduce the time for automatic landing, and improve the one-time landing success rate; specifically including the following steps:

[0104] S7.1: Collect the system operation data, optimize the system, and improve the response speed and accuracy of the system. The data collection module records the sensor data and control instructions in real time, and improves the performance and reliability of the system through big data analysis and optimization algorithms;

[0105] S7.2: Use the AI fusion method to reduce the number of spreader sensors, reduce the time for automatic container landing, and improve the success rate of the first container landing. The AI fusion method improves the perception ability and decision-making accuracy of the system through multi-modal data fusion.

[0106] S8: Emergency handling. When the system detects an emergency, automatically activate the emergency braking system to prevent collisions, and re-evaluate the environment and plan a new safe path after the emergency is handled; The specific steps are as follows:

[0107] S8.1: When the system detects an emergency, automatically activate the emergency braking system to prevent collisions. The emergency braking system includes a braking control unit and a brake. The braking control unit quickly activates the brake according to the instructions of the central processing unit to ensure the safety of the rubber-tyred gantry crane in an emergency;

[0108] S8.2: After the emergency is handled, the system will re-evaluate the environment and plan a new safe path. The re-evaluation module recovers the environmental data and calculates a new driving path to ensure the safety and efficiency of the rubber-tyred gantry crane.

[0109] In this embodiment, when the system detects an emergency, the emergency braking system is automatically activated to prevent collisions, and the environment is re-evaluated and a new safe path is planned after the emergency is handled, enhancing the safety and emergency response ability of the system.

[0110] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tire crane collision prevention and identification method integrating GNSS differential data, characterized in that: The following steps are involved: S1: System initialization and sensor integration: installing and integrating the GNSS receiver, 3D LiDAR, binocular camera, and IMU on the RTG; S2: Data acquisition and preprocessing: Collect GNSS signals, process them to obtain GNSS differential data and lane information, and filter and denoise the received 3D lidar and binocular camera data. This process specifically includes the following steps: S2.1: Collects GNSS signals and achieves centimeter-level positioning in conjunction with a ground base station. The ground base station connects to the GNSS receiver on the tire crane via wireless communication, providing real-time GNSS differential data and lane information. S2.2: Filter and denoise the data collected by the 3D LiDAR and binocular camera. The filtering algorithm is as follows: ； Among them, is the state vector, which includes distance and angle information for 3D lidar and position and velocity of the target in the image for binocular camera; is in the time step The posterior state estimate is the optimal estimate of the system state after obtaining the actual measurement value at time k; is in the time step The prior state estimate is the state predicted based on the state estimate at time k-1 and the system model before the actual measurement value at time k is obtained; K k Adjustment factor, used to determine the weight between the measured value and the predicted value; Z k is the actual measurement value at time step k, obtained from the data collected by the 3D lidar and binocular camera; The observation matrix maps the state variables to the measurement space and determines how the state variables affect the measurement values; S3: Sensor data fusion and processing: Using deep learning technology to process data collected by binocular cameras, extract obstacle and lane line information, and combine it with dynamic models to predict the motion state of the tire crane; S4: Collision detection and warning, which uses 3D visual image recognition technology and point cloud data processing to identify the location and type of obstacles, and issues a warning when a potential collision is detected, while also storing the image and location information of the obstacle target; S5: Automatic deviation correction control, automatically adjusting the travel direction and speed of the tire crane based on GNSS differential data and lane line information; S6: Dynamic path planning, simulating the movement state and surrounding environment of the tire crane to dynamically plan the optimal driving path; S7: System feedback and optimization: collect system operation data, optimize the system, and use AI fusion methods to reduce the number of sensors; S8: Emergency handling: When the system detects an emergency, it automatically activates the emergency braking system to prevent a collision, and reassesses the environment and plans a new safe path after the emergency is handled.

2. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1 is characterized in that: In S1, the following steps are specifically included: S1.1: The GNSS receiver is installed on the top of the RTG, the 3D LiDAR and binocular camera are installed on the front and side of the RTG, and the IMU is installed at the center of the RTG. S1.2: Use a programmable controller (PLC) to coordinate the data of each sensor. The PLC is connected to each sensor via wired or wireless communication.

3. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1 is characterized in that: In S3, specifically including: The video data collected by the binocular camera is processed using deep learning video detection and segmentation technology to extract information about obstacles and lane lines in the environment. Deep learning specifically uses the YOLO detection model, and its loss function is: ； In the above formula, L is the total loss. The smaller the total loss L is, the closer the model's predicted obstacles and lane lines are to the actual marked obstacles and lane lines, and the better the model performance is. Is a weight parameter used to balance the contribution of coordinate loss in the total loss; Indicates the number of bounding boxes predicted by each grid cell in the target detection task. Different grid cells correspond to different numbers of bounding boxes, and different numbers of bounding boxes correspond to dynamically changing scenes and obstacles. B Indicates the maximum number of bounding boxes that can be predicted in each grid cell to cover multiple obstacles and lane lines that may appear. B The value of needs to be large enough; is an indicator function, when the grid cell and bounding box When there is a target in , its value is 1, otherwise it is 0, so that only the bounding box containing obstacle and lane line information contributes to the loss function; and Represents the true value of the x-coordinate and y-coordinate of the center of the bounding box respectively; and Represents the predicted values of the x-coordinate and y-coordinate of the center of the bounding box respectively; and represent the true values of the width and height of the bounding box, respectively, and Help the model understand the size of obstacles and lane lines; and Represent the predicted values of the width and height of the bounding box respectively; is another weight parameter that identifies areas without obstacles or lane lines; The true value representing the confidence level of the object in the bounding box; The predicted value representing the confidence level of the object in the bounding box; Indicates that the bounding box belongs to the category c The true value of the probability of Indicates that the bounding box belongs to the category c classes is the total number of obstacle and lane line information categories.

4. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 3 is characterized in that: In S3, it also includes: The dynamic model technology is combined to model and predict the motion state of the tire crane, and predict the future position and driving trajectory of the tire crane. The GRU model is specifically used, and its equation is as follows: ； Among them, To update the value of the gate, it determines how much proportion of the hidden state at the previous moment The hidden state that will be retained until the current moment middle; is the weight matrix of the update gate, is the bias term of the update gate, and σ is the sigmoid activation function, which limits the output value to between 0 and 1; To reset the value of the gate, it determines how much of the previous moment's hidden state will be reset to 0; is the weight matrix of the reset gate, is the bias term for the reset gate; Is the value of the candidate hidden state, which is a candidate for a new hidden state and is partially or completely used to update the hidden state at the current moment ; is the weight matrix of the candidate hidden states, is the bias term of the candidate hidden state. tanh is the hyperbolic tangent activation function, which limits the output value to between -1 and 1. The symbol ⊙ represents the Hadamard product. According to the value of the update gate To update the hidden state at the current moment , is the hidden state vector at time step t, which represents the internal state of the tire crane and contains the predicted position and speed information of the tire crane; is the input vector at time step t, which contains the current speed, acceleration, and direction information of the tire crane.

5. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1, characterized in that: In S4, the following steps are specifically included: S4.1: After the data collected by the 3D laser radar and the binocular camera are fused, a 3D point cloud is generated. The location and type of obstacles are identified through point cloud clustering and segmentation algorithms. The point cloud clustering uses the DBSCAN algorithm, whose parameters include the minimum number of points. and neighborhood radius minPts; S4.2: Implement obstacle target alarm image storage. The image and location information of the obstacle target are stored in the vehicle memory for subsequent analysis and troubleshooting.

6. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1, characterized in that: In S5, the following steps are specifically included: S5.1: Automatically adjusts the travel direction and speed of the tire crane based on GNSS differential data and lane information. The central processing unit calculates the current position and deviation of the tire crane based on GNSS differential data and lane information, and adjusts the travel direction and speed of the tire crane through the PID controller. S5.2: Use the BP neural network to identify the images captured by the binocular camera, simplify them into discrete points, and perform line detection using a hybrid Hough transform and least squares method to develop a correction strategy.

7. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1, characterized in that: In S6, specifically including: The trajectory and position of the container are predicted and simulated, the optimal path is dynamically planned, the movement state and surrounding environment of the tire crane are simulated in real time, and the optimal driving path is calculated through the optimization algorithm. The cost function is as follows: ； in, The total cost of the node, which means the total cost of the node from the starting point to the target point The estimated cost of , in the scenario where the digital twin simulates a rubber-tyred crane, this represents the total cost from the current position of the rubber-tyred crane to the target position; In the above formula The actual cost from the starting point to the node, which is specifically the actual driving distance or time cost of the tire crane moving from the initial position to the current node position; In the above formula The estimated cost from the node to the target point is a heuristic estimate used to predict the shortest path cost from the current node to the target node. It is based on the shortest estimated distance from the current position of the tire crane to the target position and is calculated using Euclidean distance or Manhattan distance. In the above formula Represents the current node. Each node represents the state of the tire crane at a specific location and time, including its position coordinates, speed, and direction.

8. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1, characterized in that: In S7, the specific steps include: S7.1: Collect system operation data and optimize the system; the data collection module records sensor data and control instructions in real time, and optimizes the algorithm through big data analysis; S7.2: Use AI fusion methods to reduce the number of spreader sensors; AI fusion methods are based on multimodal data fusion.

9. The tire crane anti-collision and identification method integrating GNSS differential data according to claim 1, characterized in that: In S8, the following steps are specifically included: S8.1: When the system detects an emergency, it automatically activates the emergency braking system, which includes a brake control unit and a brake. The brake control unit quickly activates the brake according to the instructions of the central processor. S8.2: After handling an emergency, the system will reassess the environment and plan a new safe path. The reassessment module calculates the new driving path by recollecting environmental data.

Images