Multi-machine cooperation method and system for multimodal transport cargo receiving, unloading and transferring

Abstract

The invention discloses a multimodal transport cargo receiving, unloading and transferring multi-machine cooperation method and system. The multimodal transport cargo receiving, unloading and transferring multi-machine cooperation method comprises the steps of obtaining freight basic elements, constraint conditions and real-time environment information, and assigning a matched number of unmanned transfer vehicles and unmanned aerial vehicle sets; generating an initial optimal path; and the local path of the initial optimal path is adjusted in real time, the unmanned transfer vehicle and the unmanned aerial vehicle set advance to the unloading destination according to the global optimal path, and multi-vehicle cooperative cargo unloading is completed according to the conflict-free motion trail. A multi-machine cooperation technology is adopted, advanced technologies such as unmanned driving, the Internet of Things and an artificial intelligence algorithm are fused, the unmanned transfer trolley, the unmanned aerial vehicle and the intelligent gantry crane can work efficiently and cooperatively, various operation scenes such as indoor and outdoor switching, place layout difference and extreme weather can be dealt with, cargoes can be rapidly and accurately received, unloaded and transferred, and the working efficiency is improved. And the efficiency of multimodal transport is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cargo handling, and particularly to a multi-machine cooperation method and system for multimodal transport cargo unloading and transfer. Background Art

[0002] Under the background of the accelerating global economic integration, the logistics industry, as a key link connecting production and consumption, its efficiency and service quality are crucial for the stable development of the global economy. Multimodal transport, as the core of the modern logistics system, realizes seamless connection and efficient transportation of goods by integrating multiple transportation modes such as highway, railway, waterway, and air, greatly promoting the prosperity of global trade. However, the multimodal transport process involves various complex cargo unloading and transfer scenarios such as rail-water intermodal transport, road-rail intermodal transport, and air-water intermodal transport, and these scenarios pose extremely high requirements for efficiency, accuracy, and safety.

[0003] Existing unloading and transfer technologies have obvious deficiencies when facing these complex and changeable environments: on the one hand, the ability to cope with complex scenarios is limited. Traditional manual operations and simple automation systems are difficult to quickly adapt to the special needs of different scenarios, resulting in low transfer efficiency, frequent cargo damage and transfer errors, increasing logistics costs, and reducing service quality. On the other hand, the intelligent cooperation ability is insufficient. Existing technologies mostly operate with single devices or simple systems, lacking effective cooperation among advanced devices such as unmanned transfer vehicles, unmanned aerial vehicles, and intelligent gantry cranes, resulting in insufficient utilization of resources, unsmooth operation processes, unable to give full play to the advantages of each device, restricting the improvement of the overall operation efficiency, and difficult to meet the requirements of the modern logistics industry for efficient, accurate, and intelligent unloading and transfer. Summary of the Invention

[0004] Embodiments of the present invention provide a multi-machine cooperation method and system for multimodal transport cargo unloading and transfer to solve the technical problems of low efficiency, accuracy, and safety in multimodal transport cargo unloading and transfer in the prior art.

[0005] A multi-machine cooperation method for multimodal transport cargo unloading and transfer includes:

[0006] S10. Obtain the basic freight elements, constraint conditions, and real-time environmental information, and assign a matching number of unmanned transfer vehicles and unmanned aerial vehicle groups according to the basic freight elements, constraint conditions, and environmental information; the basic freight elements include the starting location, unloading destination, cargo size, and cargo weight; the constraint conditions include time window, battery endurance time, and load balance condition; the real-time environmental information includes traffic conditions, weather conditions, obstacles, and the geographical layout of the loading and unloading area;

[0007] S20. Generate an initial optimal path according to the basic freight elements and real-time environmental information;

[0008] S30. When the unmanned transfer vehicle and the unmanned aerial vehicle group travel along the initial optimal path, obtain the real-time positioning information collected by the unmanned transfer vehicle and the real-time visual data collected by the unmanned aerial vehicle group, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions;

[0009] S40. The unmanned transfer vehicle and the unmanned aerial vehicle group travel to the unloading and loading destination along the global optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aerial vehicle group. The semantic map data includes cargo labels and pallet positions;

[0010] S50. According to the point cloud data and the semantic map data, pre-enact the loading and unloading actions in the Isaac Sim simulation environment, generate a safe operation sequence, and calculate the optimal loading and unloading order;

[0011] S60. According to the safe operation sequence and the optimal loading and unloading order, through the assistance of the unmanned aerial vehicle for positioning, guide the gantry crane to accurately dock with the container corner fittings of the cargo, use the CBS algorithm to plan the non-conflicting movement trajectories among the unmanned transfer vehicle, the unmanned aerial vehicle group and the gantry crane boom, and complete the multi-machine collaborative cargo unloading and loading according to the non-conflicting movement trajectories.

[0012] A multi-machine collaborative system for multimodal transport cargo unloading and transfer, comprising:

[0013] Assignment module: used to obtain the basic freight elements, constraint conditions and real-time environment information, and assign a matching number of unmanned transfer vehicles and unmanned aerial vehicle groups according to the basic freight elements, constraint conditions and environment information;

[0014] Initial optimal path generation module: used to generate an initial optimal path according to the basic freight elements and real-time environment information;

[0015] Global optimal path generation module: used to obtain the real-time positioning information collected by the unmanned transfer vehicle and the real-time visual data collected by the unmanned aerial vehicle group when the unmanned transfer vehicle and the unmanned aerial vehicle group travel along the initial optimal path, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions;

[0016] Obtaining module: used for the unmanned transfer vehicle and the unmanned aerial vehicle group to travel to the unloading and loading destination along the global optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aerial vehicle group. The semantic map data includes cargo labels and pallet positions;

[0017] Simulation and calculation module: used to pre-enact the loading and unloading actions in the Isaac Sim simulation environment according to the point cloud data and the semantic map data, generate a safe operation sequence, and calculate the optimal loading and unloading order;

[0018] The unloading module is used to accurately dock the gantry crane with the container corner fittings of the goods through the assistance of drones for positioning according to the safe operation sequence and the optimal loading and unloading order, plan the non-conflicting movement trajectories among the automated guided vehicle, the drone group and the gantry crane boom by using the CBS algorithm, and complete the multi-machine collaborative goods unloading according to the non-conflicting movement trajectories.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0020] (1) It significantly improves the transfer and unloading efficiency of multimodal transport and realizes efficient logistics operation: The present invention adopts a global-local double-layer planning algorithm, combines real-time environmental information to finely plan the transfer path, determines the general route framework through global planning, and dynamically adjusts according to the real-time situation through local planning to find the optimal global optimal path. It also solves the problems of dynamic obstacle avoidance and efficiency optimization in the scenario of the connection of transportation modes. The automated guided vehicle, the drone group and the intelligent gantry crane achieve seamless connection through information sharing and coordination, constructing a trinity multi-machine collaborative system of "air-ground-crane". The automated guided vehicle is responsible for short-distance ground transportation and loading and unloading, the drone monitors the goods and provides assistance for positioning, and the intelligent gantry crane precisely hoists the container, reducing manual intervention and operation time, shortening the dispatching response time of the straddle carrier to within 200 milliseconds, and improving the goods transfer efficiency.

[0021] (2) The present invention comprehensively guarantees the transfer safety and reduces the accident risk: It integrates the drone vision data and the radar data of the automated guided vehicle to construct an all-round environmental perception system. The drone monitors in a large range, and the radar precisely detects the obstacles at close range. When an obstacle is detected, the system starts the obstacle avoidance algorithm to dynamically avoid the collision risk in real time. With the assistance of drones for positioning, it provides accurate guidance for the docking of the gantry crane and the container. The drone obtains the position and attitude information of the container and transmits it to the gantry crane control system to achieve precise docking with an error less than 2 mm, improving the loading and unloading efficiency.

[0022] (3) It has strong adaptability to complex environments: The vehicle-machine cooperation technology enables the transfer system to adapt to complex scenarios such as indoor-outdoor switching, different transfer site layouts and extreme weather. It automatically switches the sensing device according to the lighting conditions, enables the economical multi-spectral imaging unit when the lighting is sufficient, and switches to the high-cost-performance solid-state LiDAR in low-illumination environments, improving the sensing ability and operation reliability of the system under different lighting conditions. By integrating advanced sensor technology, intelligent control algorithms and protection measures, the system has the ability to operate all-weather. Description of the Drawings

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 is a schematic flowchart of a multi - machine collaborative method for multimodal transport cargo unloading, transfer and transshipment in an embodiment of the present invention;

[0025] Figure 2 is a structural diagram of a multi - machine collaborative system for multimodal transport cargo unloading, transfer and transshipment in an embodiment of the present invention;

[0026] Figure 3 is a principle structural diagram of a multi - machine collaborative method for multimodal transport cargo unloading, transfer and transshipment in an embodiment of the present invention;

[0027] Figure 4 is a structural diagram of the DON model of a multi - machine collaborative method for multimodal transport cargo unloading, transfer and transshipment in an embodiment of the present invention;

[0028] Figure 5 is a working process flowchart of a multi - machine collaborative system for multimodal transport cargo unloading, transfer and transshipment in an embodiment of the present invention. Specific embodiments

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0030] The multi - machine collaborative method for multimodal transport cargo unloading, transfer and transshipment provided by the embodiments of the present invention can be applied in the application environment as Figure 1 shown. Specifically, the multi - machine collaborative method for multimodal transport cargo unloading, transfer and transshipment is applied in a multi - machine collaborative system for multimodal transport cargo unloading, transfer and transshipment. The multi - machine collaborative system for multimodal transport cargo unloading, transfer and transshipment includes a client and a server as Figure 2 shown. The client communicates with the server through a network. Among them, the client, also known as the user side, refers to a program that provides local services for the client corresponding to the server. The client can be installed on but not limited to various personal computers, laptop computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0031] In one embodiment, as Figure 1 shown, a multi-machine cooperation method for multimodal cargo unloading, transferring and transporting is provided, including the following steps:

[0032] S10. Obtain the basic freight elements, constraint conditions and real-time environment information, and assign a matching number of unmanned transfer vehicles and unmanned aircraft groups according to the basic freight elements, constraint conditions and environment information; the basic freight elements include the starting location, unloading destination, cargo size and cargo weight; the constraint conditions include time window, battery endurance time and load balance condition; the real-time environment information includes traffic conditions, weather conditions, obstacles and the geographical layout of the loading and unloading area. Understandably, for the obtaining and assignment strategy, the executing entity can be a logistics management platform with intelligent scheduling function, which establishes communication connections with multiple unmanned transfer vehicles and multiple unmanned aircraft groups respectively, and can obtain their status information in real time and issue scheduling instructions.

[0033] S20. Generate an initial optimal path according to the basic freight elements and real-time environment information. Understandably, by comprehensively considering the basic freight elements (such as cargo type, weight, volume, starting and ending locations, etc.) and real-time environment information (such as traffic flow, weather conditions, road construction, etc.), the generated initial optimal path can avoid congested sections and areas affected by bad weather, reduce transportation time and improve vehicle turnover rate.

[0034] S30. When the unmanned transfer vehicles and unmanned aircraft groups are traveling according to the initial optimal path, obtain the real-time positioning information collected by the unmanned transfer vehicles and the real-time visual data collected by the unmanned aircraft groups, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions. Understandably, when the unmanned transfer vehicles and unmanned aircraft groups are traveling according to the initial optimal path, the environment is dynamically changing, and sudden dynamic obstacles such as pedestrians, other vehicles, falling objects, road collapse, water accumulation, construction, etc. may suddenly appear. By obtaining the real-time positioning information of the unmanned transfer vehicles and the real-time visual data of the unmanned aircraft groups, these obstacles can be detected in time, and the local path can be adjusted in real time to avoid collision accidents and ensure the safety of the cargo, equipment and surrounding personnel and property.

[0035] S40. The unmanned transfer vehicle and the drone group travel to the loading and unloading destination along the globally optimal path, obtaining the three-dimensional point cloud data of the loading and unloading area collected by the gantry crane, and obtaining the semantic map data collected by the drone group. The semantic map data includes cargo labels and pallet positions. It can be understood that obtaining the three-dimensional point cloud data of the loading and unloading area collected by the gantry crane can accurately construct a three-dimensional spatial model, clearly presenting details such as obstacles and cargo stacking heights, and assisting in the rational planning of loading and unloading actions and paths; obtaining the semantic map data containing cargo labels and pallet positions collected by the drone group can accurately identify cargo information and pallet distribution, achieve accurate matching of cargo and pallets, and ensure that loading and unloading operations are carried out efficiently, accurately, and orderly.

[0036] S50. Rehearse loading and unloading actions in the Isaac Sim simulation environment based on point cloud data and semantic map data, generate a safe operation sequence, and calculate the optimal loading and unloading sequence. Understandingly, rehearsing loading and unloading actions in the Isaac Sim simulation environment based on point cloud and semantic map data can identify potential collision and interference risks in advance, generating a safe operation sequence to ensure actual operation safety. Calculating the optimal loading and unloading sequence can significantly reduce equipment movement distance and time, improve loading and unloading efficiency, and reduce operating costs.

[0037] S60: Based on the safe operation sequence and the optimal loading and unloading sequence, drone-assisted positioning is used to guide the gantry crane to precisely dock with the container corner fittings. The CBS algorithm is used to plan collision-free motion trajectories between the unmanned transfer vehicle, the drone group, and the gantry crane boom, and the multi-machine coordinated cargo loading and unloading is completed according to the collision-free motion trajectory. It can be understood that based on the safe operation sequence and the optimal loading and unloading sequence, drone-assisted positioning can greatly improve the accuracy of the docking between the gantry crane and the container corner fittings and reduce docking errors. The CBS algorithm is used to plan collision-free motion trajectories, effectively preventing collisions between the unmanned transfer vehicle, the drone group, and the gantry crane boom during operation, ensuring the safe operation of all equipment.

[0038] In the present invention, an innovative multi-machine collaborative method of "air-ground-crane" is constructed, with drones as the key force in the air. With their flexible flying capabilities and wide field of view, they can obtain environmental information of the transfer site in real time, and provide accurate navigation and positioning services for ground equipment and gantry cranes. As the main means of transportation on the "ground", unmanned transfer vehicles are responsible for short-distance transportation and loading and unloading operations of goods in the transfer site. Their strong carrying capacity and adaptability can meet the transportation needs in different scenarios. As the core equipment of the "crane", the intelligent gantry crane, with its precise lifting capabilities and efficient operating performance, can realize the rapid and safe transfer of goods between different means of transportation.

[0039] In one embodiment, step S10 includes:

[0040] Obtain the basic freight elements through the shipper; obtain the road congestion situation in real time through 5G / C-V2X technology; obtain the weather conditions along the way through the meteorological API; obtain the precise geographical coordinates and road layout of the loading and unloading area through ArcGIS or Google Maps API; detect and identify road obstacles in real time through the millimeter-wave radar / LiDAR carried by the unmanned transfer vehicle. The above sub-steps are not in any particular order. Understandably, as Figure 3 shown, the unmanned transfer vehicle, the unmanned vehicle group and the gantry crane form the moving part of the transfer system, the radar systems carried by the unmanned vehicle and the gantry crane and the vision modules carried by the unmanned vehicle group form the reconnaissance part of the transfer system, and the decision-making part refers to the large model that collects information, processes information and sends control signals.

[0041] Among them, the unmanned transfer vehicle is responsible for transferring goods between different transportation modes and is equipped with an advanced radar system for detecting the surrounding environment in real time to ensure safe driving. In addition, the unmanned transfer vehicle is also equipped with two drone recovery compartments. The recovery compartments provide protection for the drones under adverse weather conditions (such as thunderstorms and strong winds) and are equipped with a wireless charging module to provide energy support for the continuous operation of the drones, ensuring the efficient operation of the entire system.

[0042] The main task of the drones is to fly with the unmanned transfer vehicle and provide real-time road condition information for the decision-making part. Especially in complex road conditions, it helps the unmanned transfer vehicle reach the destination safely. During the process of loading and unloading goods, the drones also provide key information to ensure the smooth progress of the loading and unloading work, thereby reducing manual intervention. The lower part of the drones is equipped with replaceable customized modules, including a camera module suitable for good light conditions and a lidar module suitable for low visibility environments. Usually, two drones work and charge alternately to ensure continuous operation. In complex road conditions, two drones can take off simultaneously to provide more comprehensive road condition information.

[0043] The gantry crane is mainly responsible for accurately placing the container on the unmanned transfer vehicle. For this purpose, the gantry crane is equipped with a radar system that can provide accurate information for the decision-making part during the loading and unloading process to ensure the accurate placement of the container, thereby improving the efficiency and safety of the entire transfer process.

[0044] The decision-making part is the control center of the entire transfer system. It can be deployed indoors, receive environmental information from the unmanned transfer vehicle, drones and gantry crane, and control the movement of the unmanned transfer vehicle based on this information; it is also responsible for controlling the accompanying flight of the drones to obtain information from multiple angles to ensure the accuracy and safety of the goods loading and unloading process. By centrally processing and analyzing data from various devices, the decision-making part can make optimal decisions in real time and coordinate the efficient operation of the entire transfer system.

[0045] In one embodiment, step S20 includes the following sub-steps: S201. Obtain the geographical coordinates of the starting location and the unloading destination, and generate a preliminary geographical path based on ArcGIS or Google Maps API; according to the cargo size and cargo weight, screen out the passable paths that match the unmanned transfer vehicle from the geographical path; S202. Construct a multi-objective optimization model, define the constraint conditions as optimization objectives, set the weight coefficients, and adjust the priorities of each objective according to actual requirements; S203. Use the multi-objective evolutionary algorithm to optimize the passable path, and continuously iterate and adjust the passable path until an initial optimal path that meets all constraint conditions is found; S204. Verify the initial optimal path through the simulation software Isaac Sim, and simulate the operation of the unmanned transfer vehicle and the unmanned aircraft group on the initial optimal path; S205. Send the verified initial optimal path to the unmanned transfer vehicle and the unmanned aircraft group. Understandably, by comprehensively considering various factors such as geographical information, cargo characteristics, and constraint conditions, and using advanced algorithms and simulation software for initial optimal path planning and verification, it provides a reliable navigation basis for the operation of unmanned transportation equipment.

[0046] In one embodiment, step S30 includes:

[0047] S301. Obtain the positioning information of the unmanned transfer vehicle through the Beidou positioning module, and the positioning information includes the real-time position, speed, and direction; the vision module carried by the unmanned aircraft group detects the surrounding obstacles of the initial optimal path;

[0048] S302. Use the deep learning model Mask R-CNN to perform semantic segmentation on the vision data, and extract the obstacle type, position, and motion state; S303. Align the positioning information of the unmanned transfer vehicle and the vision data of the unmanned aircraft group on the time axis, and input them into the DQN model; S304. The DQN model selects the optimal local path according to the current state and sends it to the unmanned transfer vehicle and the robot group to adjust the operation actions;

[0049] S305. When the adjustment of the optimal local path cannot meet the global objective, use the graph neural network or genetic algorithm to trigger the intelligent large model to re-search for a new global optimal path that meets all constraint conditions on the 3D semantic map. Understandably, by obtaining the positioning information of the unmanned transfer vehicle and the detection data of the surrounding obstacles by the unmanned aircraft group, using the deep learning model to extract the obstacle features, combining with the reinforcement learning model to select the optimal local path, and triggering the global path re-search when necessary, to achieve real-time dynamic path planning and adjustment during the operation of the unmanned transfer vehicle and the unmanned aircraft group.

[0050] In another embodiment, as Figure 4As shown, step S304 is that the DQN model selects the optimal local path according to the current state and sends it to the unmanned transfer vehicle and the robot group to adjust the operation actions, and further includes the following sub-steps:

[0051] S3041. Construct a DRL network for training the DQN model. The DRL network includes a first convolutional layer conv1, a second convolutional layer conv2, a first fully connected layer fc1, and a second fully connected layer fc2:

[0052] The convolutional kernel size of conv1 is 3x3, and the stride is 2; the convolutional kernel size of conv2 is 3x3, and the stride is 2; fc1 has 128 neurons; fc2 has 3 neurons, and the output of fc2 corresponds to the Q values corresponding to 3 actions;

[0053] S3042. The DQN training samples can be obtained by using the experience replay mechanism, and then the network parameters are updated by using the stochastic gradient descent method. Specifically:

[0054] The agent saves the state transition sequence e at each time point t, that is, <current state, action, reward, next state>;

[0055] The state transition sequence e is stored in the replay pool D with a length of N, D = {e1, e2,..., eN};

[0056] Randomly sample from the replay pool D and use it as the training data for updating the DRL network;

[0057] S3043. The ε-greedy policy model is used to complete the action selection of the unmanned transfer vehicle, gantry crane, and robot group, and randomly select an action with a probability of ε. The expression of the ε-greedy policy model is:

[0058]

[0059] Among them, N(A) represents the number of actions of the mobile robot in the action set A, s represents the current system state, and P(a max , s) represents the probability of taking action a max on the basis of state s;

[0060] S3044. According to the action selection strategy, determine the best action to be executed by the unmanned transfer vehicle in the current state, and send the determined optimal action instruction to the control system of the unmanned transfer vehicle to adjust its operation actions;

[0061] S3045. Perform loop iteration, repeat the steps of S3041 to S3044 above, so that the unmanned transfer vehicle continuously perceives, makes decisions, and executes actions in the environment, and updates the DQN model according to the new state.

[0062] Understandably, steps S3041 - S3045 circumvent the limitations of relying on a single sensor or data source for path planning and decision-making. By fusing data from multiple sensors such as lidar, visual SLAM, inertial navigation, and Beidou positioning, it provides a more comprehensive and accurate environmental perception ability, and dynamically adjusts the route in real-time according to environmental changes and sensor data.

[0063] Understandably, the sub-steps of steps S20 - S30 are used to solve the problems of dynamic obstacle avoidance and efficiency optimization in the scenario of multi-modal transportation connection, and adopt a global-local double-layer planning algorithm based on multi-modal fusion. In practical application scenarios, such as when an unmanned transfer vehicle transfers containers at a port-railway freight yard, the unmanned transfer vehicle can provide a 3D semantic map constructed by lidar and Beidou positioning information, while the unmanned aerial vehicle group can provide real-time visual data, such as identifying congested areas in the yard through aerial photography. The intelligent large model will combine these two aspects of information provided by the unmanned transfer vehicle and the unmanned aerial vehicle group, and use an optimization algorithm to generate the globally optimal path. Then, with the help of a reinforcement learning model (such as DQN), the local path is adjusted in real-time to avoid dynamic obstacles, such as temporarily parked AGVs or manually operated equipment. In addition, the algorithm also embeds multi-objective constraints, covering aspects such as time windows, battery life, and load balance, so as to ensure that the path planning can take into account both efficiency and safety. In a typical scenario, the scheduling response time of cross-transport vehicles can be shortened to within 200 milliseconds.

[0064] In one embodiment, step S40 is that the unmanned transfer vehicle and the unmanned aerial vehicle group travel to the unloading destination according to the globally optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aerial vehicle group. The semantic map data includes cargo labels and pallet positions, and includes the following sub-steps:

[0065] S401. Establish a position conversion model to convert the radar point coordinates of the three-dimensional space of the loading and unloading area obtained by the radar carried by the gantry crane into the camera coordinate system. The mathematical expression of the position conversion model is:

[0066]

[0067] where r is the distance from the radar to the target point in the loading and unloading area, θ is the horizontal azimuth angle measured by the radar, f x 、f y are the focal lengths of the camera, z is the depth of the target point in the camera coordinate system, t z is the translation amount in the z-axis direction of the installation positions of the radar and the camera, c x 、c y are the coordinates of the camera optical center on the image plane;

[0068] S402. Obtain the point cloud data of the three-dimensional space based on the radar point coordinates in the camera coordinate system, convert the original point cloud coordinates into compensated point cloud coordinates, and the calculation formula is:

[0069] P corrected = R·P raw + T (2)

[0070] where P corrected is the compensated point cloud coordinate, P raw is the original point cloud coordinate, R is the rotation matrix provided by the IMU, which is used to describe the attitude change of the radar sensor during the acquisition of point cloud data, and T is the translation vector provided by the IMU, which is used to represent the displacement of the radar sensor during the acquisition of point cloud data;

[0071] S403. Based on the compensated point cloud coordinates, use the static background filtering and dynamic target enhancement methods to calculate the point cloud difference set, reducing the amount of information transmission. The specific calculation process is as follows:

[0072] [[ID=?]]Dynamic_t = Cloud_t Θ Cloud_{t - 1} (3)

[0073] where Cloud_t represents the point cloud data of the t-th frame, Cloud_{t - 1} represents the point cloud data of the (t - 1)-th frame, Dynamic_t represents the point cloud set of dynamic targets in the t-th frame, and Θ represents the point cloud difference set calculation;

[0074] S404. Compare the points of two adjacent frames of point clouds point by point. If the displacement is greater than a certain threshold, it is marked as dynamic; otherwise, it is marked as static. Remove the overlapping points and retain the non-overlapping points;

[0075] S405. Build a speed verification model for verifying dynamic targets. The expression of the speed verification model is:

[0076]

[0077] where v represents the speed, Δt represents the time difference between two adjacent frames, Centroid t represents the centroid position at time t, and Centroid t-1 represents the centroid position at time t - 1;

[0078] If the calculated v matches the actual speed, it is marked as a dynamic target, and position data is obtained based on the dynamic target.

[0079] Note: There seems to be a formatting or content issue with the formula in step S403. The formula "Dynamic_t = Cloud_t Θ Cloud_{t - 1} (3)" might be incorrect or incomplete in the original Chinese. I've translated it as is, but it might need further clarification in the source material.Understandably, the processing of the position data is completed by steps S401 - S405. Through a series of data processing and conversion operations, the original radar point cloud data and semantic map data are transformed into a high-precision environment model that can be used for loading and unloading action rehearsal and path planning. This not only improves the accuracy and reliability of the data but also provides a solid foundation for subsequent simulation rehearsal, path planning, and multi-robot collaborative operations. By optimizing the data processing flow and improving the data accuracy, the efficiency and safety of multimodal cargo unloading and transfer are significantly enhanced.

[0080] In one embodiment, step S50 includes: S501, unifying the timestamps of the point cloud data and the semantic map data to eliminate coordinate system deviations; ensuring the temporal consistency of the point cloud data and the semantic map data.

[0081] S502, loading the gantry crane, unmanned transfer vehicle, and unmanned aerial vehicle group in Isaac Sim, rehearsing the loading and unloading actions by combining the point cloud data and the semantic map data, using the collision detection algorithm FCL to detect the interference risk between the unmanned transfer vehicle and obstacles in real time, and adjusting the trajectory to avoid collisions; simulating the loading and unloading operation process in the Isaac Sim environment to verify the feasibility of the path planning and the loading and unloading sequence, using the FCL algorithm to detect the interference risk between the unmanned transfer vehicle and obstacles in real time. Through simulation rehearsal, potential path conflicts and collision risks can be discovered.

[0082] S503, using the ROS2TF2 toolchain to align the coordinate transformations of the gantry crane, unmanned vehicle, and unmanned aerial vehicle in real time, dynamically adjusting the observation perspective of the unmanned aerial vehicle, and combining the point cloud and semantic data to ensure that the information is unobstructed and there are no blind spots; S504, using the mixed integer programming algorithm MIP to solve the optimal loading and unloading sequence and combining the branch and bound method to handle constraint conflicts. The dynamic adjustment of the observation perspective enables the unmanned aerial vehicle to obtain environmental information more flexibly.

[0083] In another embodiment, when the unmanned transfer vehicle completes the cargo unloading according to the conflict-free movement trajectory, it further includes: detecting the obstacles that occur when the unmanned transfer vehicle is running on the globally optimal path in real time, generating an emergency avoidance path and updating the vehicle's speed and steering to safely avoid the obstacles. Understandably, by fusing multi-source data, the unmanned transfer vehicle can more comprehensively understand the surrounding environment. During the real-time obstacle avoidance process, the unmanned transfer vehicle can not only generate an emergency avoidance path but also dynamically adjust the subsequent path planning according to the current state and the target position, thus realizing the multi-robot collaborative effect during the multimodal cargo unloading and transfer process.

[0084] Understandably, the sub-steps of steps S40 - S50 adopt a combined solution of Beidou + lidar + visual SLAM + inertial navigation, providing centimeter-level accuracy and high-robustness positioning and navigation capabilities for the unmanned transfer vehicle during the multimodal cargo handling and transfer process. This solution achieves precise positioning and efficient operation through a series of well-defined steps: First, the Beidou RTK differential positioning technology is used to obtain the global accurate coordinates of the unmanned transfer vehicle; Second, combined with the lidar point cloud matching and the semantic segmentation technology of visual SLAM, the surrounding environment is sensed in real time and the pose information of the unmanned transfer vehicle is corrected; Subsequently, the drone uses a wide-angle visual sensor to scan the panorama and transmit it back to the decision-making department, and a global dynamic map is constructed through edge computing; Finally, the decision-making department sends the dynamic map to the unmanned transfer vehicle to assist its positioning system in eliminating the cumulative error and achieving higher-precision positioning and navigation.

[0085] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0086] In one embodiment, a multimodal cargo handling and transfer system is provided, and this multimodal cargo handling and transfer system corresponds one-to-one with the multimodal cargo handling and transfer method in the above embodiment. As Figure 2 shown, this multimodal cargo handling and transfer system includes:

[0087] Assignment module: used to obtain the basic freight elements, constraint conditions and real-time environment information, and assign a matching number of unmanned transfer vehicles and unmanned aerial vehicle groups according to the basic freight elements, constraint conditions and environment information;

[0088] Initial optimal path generation module, used to generate an initial optimal path according to the basic freight elements and real-time environment information;

[0089] Global optimal path generation module, used to obtain the real-time positioning information collected by the unmanned transfer vehicle and the real-time visual data collected by the unmanned aerial vehicle group when the unmanned transfer vehicle and the unmanned aerial vehicle group travel according to the initial optimal path, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions;

[0090] Acquisition module: used for the unmanned transfer vehicle and the unmanned aerial vehicle group to travel to the handling destination according to the global optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aerial vehicle group. The semantic map data includes cargo labels and pallet positions;

[0091] Simulation and calculation module: used to pre-enact the loading and unloading actions in the Isaac Sim simulation environment according to the point cloud data and the semantic map data, generate a safe operation sequence, and calculate the optimal loading and unloading sequence;

[0092] The unloading module is used to accurately dock the gantry crane with the container corner fittings of the goods through drone-assisted positioning according to the safe operation sequence and the optimal loading and unloading order, plan the non-conflicting movement trajectories among the automated guided vehicle, the drone group and the gantry crane boom by using the CBS algorithm, and complete the multi-machine collaborative goods unloading according to the non-conflicting movement trajectories.

[0093] In the system of the above embodiments of the present invention, as Figure 5 shown, it is the working process of the system of a specific embodiment:

[0094] The stage of traveling to the destination: During the process of traveling to the destination, according to the input information such as the unloading and transfer location, destination, goods size and weight, etc., considering the constraint conditions, real-time traffic and weather conditions comprehensively, an appropriate number of automated guided vehicles and drone groups are dispatched. The automated guided vehicle starts the millimeter wave radar / LiDAR in the unloaded state, detects obstacles in real time, measures the distance and constructs a 3D environment model of the surrounding area. At the same time, it establishes a real-time interactive connection with the decision-making department through 5G / C-V2X technology, provides road condition information and receives control signals, and moves according to the signals. Two drones are located in the protection bin and take off under the control of the decision-making department in case of complex road conditions. The camera module is used to identify the road conditions by using algorithms such as YOLO and OpenCV, and the picture is transmitted back to the decision-making department with low latency by using H.265 encoding and RTSP protocol. The decision-making department gives a safe passage signal accordingly.

[0095] The stage of arriving at the transfer location and unloading: After arriving at the transfer location, the gantry crane starts the millimeter wave radar array to scan the three-dimensional space of the loading and unloading area, generates point cloud data with an accuracy of ±5 cm and transmits it back to the decision-making department. The drone enables the depth camera to construct a 3D semantic map of the loading and unloading point, identifies the goods label and the pallet position, and also transmits the data back to the decision-making department. Based on this information, the decision-making department rehearses the loading and unloading actions in Isaac Sim, detects the collision risk, and generates a safe operation sequence. The coordinate systems of each device are aligned in real time through the ROS2TF2 tool chain, and the observation angle of the drone is dynamically adjusted to ensure comprehensive and accurate information. The mixed integer programming algorithm is used to calculate the optimal loading and unloading order, and balance the moving energy consumption of the gantry crane and the waiting time of the automated guided vehicle. When the unloading starts, the decision-making department sends a control signal to make a drone hover above the gantry crane spreader, and realizes the accurate docking of the spreader and the container corner fittings (error <2 mm) through ArUco marker-assisted positioning. At the same time, another drone flies along to extract key features, and uses the CBS algorithm to plan non-conflicting movement trajectories for each device.

[0096] Transportation process stage: During transportation, the unmanned transfer vehicle transmits millimeter-wave radar data back to the decision-making department. Two unmanned aerial vehicles fly alternately to provide information about the surrounding environment. The decision-making department fuses radar point clouds, IMU pose data, and GPS positioning information through Kalman filtering to construct a high-precision dynamic obstacle map within 10 meters around the vehicle. It calculates potential hazards such as potholes on the road surface and tilted goods in real time, and reconstructs the 3D semantic environment in combination with the NeRF algorithm. Based on large model decision-making, it solves the optimal speed-steering sequence within the next 3 seconds, satisfies the dynamic constraints, and transmits the control signal back to the unmanned transfer vehicle. If the radar detects a suddenly emerging moving obstacle ahead, such as an intruding animal, the emergency mode is activated: the unmanned aerial vehicle switches to high-frequency scanning at 30Hz to update the obstacle movement trajectory; the large model generates an emergency avoidance path within 50ms to ensure the safe and efficient transportation process.

[0097] For the specific limitations of the multimodal cargo unloading and transfer system, reference can be made to the limitations of the multimodal cargo unloading and transfer method described above, which will not be elaborated here. Each module in the above multimodal cargo unloading and transfer system can be implemented in whole or in part by software, hardware, and their combinations. The above-mentioned modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0098] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multi-machine cooperation method for multimodal transport cargo unloading and transfer, characterized in that Including: S10. Obtain the basic freight elements, constraint conditions and real-time environment information, and assign a matching number of unmanned transfer vehicles and unmanned aerial vehicle groups according to the basic freight elements, constraint conditions and environment information; the basic freight elements include the starting location, unloading destination, cargo size and cargo weight; the constraint conditions include time window, battery endurance time and load balance condition; the real-time environment information includes traffic conditions, weather conditions, obstacles and the geographical layout of the loading and unloading area; S20. Generate an initial optimal path according to the basic freight elements and real-time environment information; S30. When the unmanned transfer vehicle and the unmanned aerial vehicle group travel according to the initial optimal path, obtain the real-time positioning information collected by the unmanned transfer vehicle and the real-time visual data collected by the unmanned aerial vehicle group, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions; S40. The unmanned transfer vehicle and the unmanned aerial vehicle group travel to the unloading destination according to the global optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aerial vehicle group, and the semantic map data includes cargo labels and pallet positions; S50. According to the point cloud data and semantic map data, preview the loading and unloading actions in the Isaac Sim simulation environment, generate a safe operation sequence, and calculate the optimal loading and unloading order; S60. According to the safe operation sequence and the optimal loading and unloading order, through the assistance of the unmanned aerial vehicle for positioning, guide the gantry crane to accurately dock with the container corner fittings of the cargo, use the CBS algorithm to plan the non-conflicting movement trajectories among the unmanned transfer vehicle, the unmanned aerial vehicle group and the gantry crane boom, and complete the multi-machine collaborative cargo unloading according to the non-conflicting movement trajectories.

2. The multi-machine collaborative method for multimodal transport cargo unloading and transshipment according to claim 1, wherein, The step S10 includes: Obtain the basic freight elements through the shipper; Obtain the road congestion situation in real time through 5G / C-V2X technology; Obtain the weather conditions along the way through the meteorological API; Obtain the accurate geographical coordinates and road layout of the loading and unloading area through the ArcGIS or Google Maps API; Detect and identify road obstacles in real time through the millimeter wave radar / LiDAR carried by the unmanned transfer vehicle.

3. The multi-machine collaborative method for multimodal transport cargo unloading and transshipment according to claim 1, characterized in that, The step S20 includes: S201. Obtain the geographical coordinates of the starting location and the unloading destination, and generate a preliminary geographical path based on the ArcGIS or Google Maps API; according to the cargo size and cargo weight, screen out the passing paths that match the unmanned transfer vehicle from the geographical path; S202. Build a multi-objective optimization model, define the constraint conditions as optimization objectives, set the weight coefficients, and adjust the priorities of each objective according to actual needs; S203. Use the multi-objective evolutionary algorithm to optimize the passing path, and continuously iterate and adjust the passing path until an initial optimal path that meets all constraint conditions is found; S204. Verify the initial optimal path through the simulation software Isaac Sim, and simulate the operation of the unmanned transfer vehicle and the unmanned aerial vehicle group on the initial optimal path; S205. Send the verified initial optimal path to the unmanned transfer vehicle and the unmanned aerial vehicle group.

4. The multi-machine cooperation method for multimodal transport cargo unloading and transshipment according to claim 1, wherein The step S30 includes: S301. Obtain the positioning information of the unmanned transfer vehicle through the Beidou positioning module. The positioning information includes real-time position, speed, and direction. The vision module carried by the unmanned aerial vehicle group detects the surrounding obstacles of the initial optimal path. S302. Use the deep learning model Mask R-CNN to perform semantic segmentation on the visual data, and extract the obstacle type, position, and motion state. S303. Align the positioning information of the unmanned transfer vehicle with the visual data of the unmanned aerial vehicle group on the time axis, and input them into the DQN model. S304. The DQN model selects the optimal local path according to the current state and sends it to the unmanned transfer vehicle and the robot group to adjust the operation actions. S305. When the adjustment of the optimal local path cannot meet the global goal, use the graph neural network or genetic algorithm to trigger the intelligent large model to re-search for a new global optimal path that meets all the constraint conditions on the 3D semantic map.

5. The multi-machine cooperation method for multimodal transport cargo unloading and transshipment according to claim 1, wherein The step S40 further includes: S401. Establish a position conversion model to convert the radar point coordinates in the three-dimensional space of the loading and unloading area obtained by the radar carried by the gantry crane into the camera coordinate system. The mathematical expression of the position conversion model is: Among them, r is the distance from the radar to the target point in the loading and unloading area, θ is the horizontal azimuth angle measured by the radar, f x and f y are the focal lengths of the camera, z is the depth of the target point in the camera coordinate system, t z is the translation amount in the z-axis direction of the installation positions of the radar and the camera, c x and c y are the coordinates of the camera optical center on the image plane; S402. Based on the radar point coordinates in the three-dimensional space under the camera coordinate system, obtain the point cloud data of the three-dimensional space, convert the original point cloud coordinates into compensated point cloud coordinates, and the calculation expression is: P corrected = R·P raw + T(2) Among them, P corrected is the compensated point cloud coordinate, P raw is the original point cloud coordinate, R is the rotation matrix provided by the IMU, which is used to describe the attitude change of the radar sensor during the acquisition of point cloud data, and T is the translation vector provided by the IMU, which is used to represent the displacement of the radar sensor during the acquisition of point cloud data; S403. Based on the compensated point cloud coordinates, use the static background filtering and dynamic target enhancement method to perform point cloud difference set calculation to reduce the amount of information transmission. The specific calculation process is: Dynamic_t = Cloud_t Θ Cloud_{t-1} (3) Where Cloud_t represents the point cloud data of the t-th frame, Cloud_{t-1} represents the point cloud data of the (t - 1)-th frame, Dynamic_t represents the point cloud set of dynamic targets in the t-th frame, and Θ represents point cloud difference set calculation. S404. Compare the point clouds of two adjacent frames point by point. If the displacement is greater than a certain threshold, it is marked as dynamic, otherwise it is marked as static. Remove the overlapping points and retain the non-overlapping points. S405. Construct a speed verification model for verifying dynamic targets. The expression of the speed verification model is: where v represents speed, Δt represents the time difference between two adjacent frames, Centroid t represents the centroid position at time t, Centroid t-1 represents the centroid position at time t−1; If the calculated v matches the actual speed, it is marked as a dynamic target, and position data is obtained based on the dynamic target.

6. The multi-machine cooperation method for multimodal transport cargo unloading and transshipment according to claim 4, wherein, The step S304 further includes: S3041. Construct a DRL network for training the DQN model. The DRL network includes a first convolutional layer conv1, a second convolutional layer conv2, a first fully connected layer fc1, and a second fully connected layer fc2: The convolution kernel size of conv1 is 3x3, and the stride is 2; the convolution kernel size of conv2 is 3x3, and the stride is 2; fc1 has 128 neurons; fc2 has 3 neurons, and the output of fc2 corresponds to the Q values of 3 actions. S3042. The experience reuse mechanism can be used to obtain DON training samples, and then the network parameters are updated using the stochastic gradient descent method. Specifically: The agent saves the state transition sequence e at each time point t, that is, <current state, action, reward, next state>. The state transition sequence e is stored in the reuse pool D with a length of N, D = {e1, e2, …, eN}; Randomly sample from the reuse pool D and use it as the training data for updating the DRL network; S3043. Use the ε-greedy policy model to complete the action selection of the unmanned transfer vehicle, gantry crane, and robot group, and randomly select an action with a probability of ε. The expression of the ε-greedy policy model is: Among them, N(A) represents the number of actions of the mobile robot in the action set A, s represents the current system state, and P(a max , s) represents the probability of taking action a max on the basis of state s; S3044. According to the action selection strategy, determine the best action to be executed by the unmanned transfer vehicle in the current state, and send the determined optimal action instruction to the control system of the unmanned transfer vehicle to adjust its operating action; S3045. Perform loop iteration, repeat the steps of S3041 to S3044 above, so that the unmanned transfer vehicle continuously perceives, makes decisions, and executes actions in the environment, and updates the DQN model according to the new state.

7. The multi-machine collaborative method for multimodal transport cargo unloading and transshipment according to claim 1, wherein The step S50 includes: S501. Unify the timestamps of the point cloud data and the semantic map data to eliminate the coordinate system deviation; S502. Load the gantry crane, unmanned transfer vehicle, and unmanned aircraft group in Isaac Sim, combine the point cloud data and the semantic map data to preview the loading and unloading actions, use the collision detection algorithm FCL to detect the interference risk between the unmanned transfer vehicle and obstacles in real time, and adjust the trajectory to avoid collisions; S503. Use the ROS2TF2 toolchain to align the coordinate transformations of the gantry crane, unmanned vehicle, and unmanned aircraft in real time, dynamically adjust the observation perspective of the unmanned aircraft, and combine the point cloud and semantic data to ensure that the information is unobstructed and there are no blind spots; S504. Use the mixed integer programming algorithm MIP to solve the optimal loading and unloading sequence, and combine the branch and bound method to handle constraint conflicts.

8. The multi-machine collaborative method for multimodal transport cargo unloading and transshipment according to claim 1, wherein, When the unmanned transfer vehicle completes the cargo receiving and discharging according to the conflict-free movement trajectory, it also includes: Real-time detect obstacles that occur when the unmanned transfer vehicle is running on the global optimal path, generate an emergency avoidance path and update the vehicle's speed and steering to safely avoid obstacles.

9. A multi-machine collaborative system for multimodal transport cargo unloading and transshipment, characterized in that, It includes: Assignment module: used to obtain the basic freight elements, constraint conditions, and real-time environment information, and assign a matching number of unmanned transfer vehicles and unmanned aircraft groups according to the basic freight elements, constraint conditions, and environment information; Initial optimal path generation module, used to generate an initial optimal path according to the basic freight elements and real-time environment information; Global optimal path generation module, used to obtain the real-time positioning information collected by the unmanned transfer vehicle and the real-time visual data collected by the unmanned aircraft group when the unmanned transfer vehicle and the unmanned aircraft group are moving according to the initial optimal path, adjust the local path of the initial optimal path in real time, and generate a global optimal path to avoid dynamic obstacles or complex road conditions; Acquisition module: used for the unmanned transfer vehicle and the unmanned aircraft group to travel to the receiving and discharging destination according to the global optimal path, obtain the point cloud data of the three-dimensional space of the loading and unloading area collected by the gantry crane, and obtain the semantic map data collected by the unmanned aircraft group. The semantic map data includes cargo labels and pallet positions; Simulation and calculation module: used to preview the loading and unloading actions in the Isaac Sim simulation environment according to the point cloud data and the semantic map data, generate a safe operation sequence, and calculate the optimal loading and unloading sequence; The unloading module is used to accurately dock the gantry crane with the container corner fittings of the goods through the assistance of drones for positioning according to the safe operation sequence and the optimal loading and unloading order, plan the non-conflicting movement trajectories among the automated guided vehicle, the drone group, and the gantry crane boom using the CBS algorithm, and complete the multi-machine collaborative unloading of the goods according to the non-conflicting movement trajectories.

Images