Embodied intelligent robot cargo loading and unloading collaborative control method based on multi-modal perception

Abstract

This invention discloses a multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method, relating to the field of cargo loading and unloading collaborative control technology. The method includes registering and calibrating the status of multiple embodied intelligent robots to make them schedulable; detecting the objects to be transported in real time through a visual sensor network and determining their quantity; and performing transportation analysis based on a target time, including: obtaining the shortest path from the starting point to the target point, identifying congestion nodes and determining their allowable values ​​based on the relationship between the path width and the travel width of the transport object, and obtaining the number of simultaneous departures. This application significantly improves system throughput and efficiency in complex bottleneck environments. By introducing a congestion node identification and allowable value quantification analysis mechanism, the width constraint of the physical path is accurately transformed into scheduling parameters. Based on this, a batch departure and interval adjustment strategy is designed, preventing congestion in narrow areas from the source, enabling the multi-robot system to maintain efficient and orderly flow even in environments with inherent bottlenecks.

Description

Technical Field

[0001] This invention belongs to the field of cargo loading and unloading collaborative control technology, specifically a multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method. Background Technology

[0002] As the logistics and warehousing industry accelerates its evolution towards automation and intelligence, embodied AIrobots, due to their high degree of autonomy and flexibility, have become key execution units in the loading, unloading, and handling of goods. To achieve efficient operations, multiple robots typically need to be coordinated. Currently, multi-robot collaborative control methods mainly rely on a central control system for global path planning and task allocation, combined with local sensor data for obstacle avoidance.

[0003] However, in real-world, complex warehouse environments, existing technologies still face significant challenges: First, most collaborative strategies are based on idealized global maps and constant throughput assumptions, failing to adequately consider long-term or temporary path bottlenecks in the actual physical space (such as narrow passages or temporary stacking areas). When multiple robots are simultaneously dispatched to such areas, congestion or even deadlocks can easily occur, leading to a sharp drop in overall system efficiency. Second, existing scheduling systems lack dynamic response capabilities. When high-priority urgent tasks are suddenly inserted, the system often lacks a rapid and accurate assessment of the progress of tasks in transit and global resources, making it difficult to achieve optimal resource reuse and smooth task transitions without causing severe congestion or significant delays to existing tasks. Furthermore, most real-time collaborative avoidance mechanisms between robots are based on simple distance thresholds or preset fixed priorities (such as allowing heavy-load robots to pass to empty ones), lacking flexible judgment criteria in complex, dynamic, multi-task scenarios. For example, they fail to consider the global urgency of the task itself as the primary factor in avoidance decisions.

[0004] Therefore, there is an urgent need for a collaborative control method that can deeply integrate multimodal environmental perception information and make integrated intelligent decisions on path bottlenecks, dynamic task insertion, and multi-robot collaborative obstacle avoidance, so as to improve the operational efficiency, robustness, and adaptability of multi-robot systems in real and dynamic warehouse scenarios. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art;

[0006] Therefore, this invention proposes a multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method, including:

[0007] Register and label the status of multiple intelligent robots to make them schedulable;

[0008] The system uses a visual sensor network to detect objects to be moved in real time and determine their quantity.

[0009] Transportation analysis based on target time includes: obtaining the shortest path of the transported goods from the origin to the destination; identifying congestion points and determining their allowable values ​​based on the relationship between path width and the travel width of the transported goods; and obtaining the number of goods leaving the same destination.

[0010] Conduct transportation simulations to obtain the number of single transfers and the time taken per trip, calculate the total transportation demand, and combine the number of outgoing shipments with the number of available transport vehicles to determine whether the target time is met;

[0011] If these conditions are not met, a transshipment plan is generated based on the location of the congestion point, loading time, and transportation speed by adjusting the batch quantity and departure interval.

[0012] During the transfer process, a short-range communication device is activated for each batch of transported goods, and real-time status information is exchanged when they enter each other's communication range, triggering an avoidance algorithm to achieve dynamic collaborative control.

[0013] Furthermore, the allowable value is determined as follows:

[0014] If the path width is less than twice the single standard width, the allowable value is 1; if it is less than three times the single standard width, the allowable value is 2; and so on. The single standard width is 1.2 to 1.5 times the driving width of the transport vehicle.

[0015] Furthermore, the single trip time is the actual time spent transporting a vehicle multiplied by an increment, where the increment ranges from 1.1 to 1.5 and increases with the number of vehicles being transported.

[0016] Furthermore, the methods for generating transit plans include:

[0017] The transportation targets are dispatched in batches based on the loading time as the initial interval; it is verified whether different batches will meet at the congestion point. If they do meet, the interval is gradually reduced until the time requirement is met.

[0018] Furthermore, in the obstacle avoidance algorithm, low-priority transport targets actively avoid obstacles, and the priority is determined based on the load weight or the urgency of the task.

[0019] Furthermore, the short-range communication device uses UWB or Wi-FiDirect, with a communication distance set at 3 meters.

[0020] Furthermore, it also includes steps for dynamically inserting emergency tasks during transportation:

[0021] Suspend subsequent batches of the current mission;

[0022] Reassess available transport resources;

[0023] Plan routes and determine the number of co-originating points for emergency missions;

[0024] Perform multi-task joint optimization simulation to generate resource reuse schemes.

[0025] Furthermore, the multi-task joint optimization simulation includes: phased scheduling of transportation targets, combined with resource recovery prediction, to achieve seamless connection between tasks.

[0026] Furthermore, during the execution of an emergency mission, if the vehicle encounters a regular mission transport target, the emergency mission will give way based on mission priority.

[0027] Furthermore, the system sets a minimum retention quantity of transport targets to 20% of the total number of robots, with a minimum of 2 robots.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] This application significantly improves system throughput and efficiency in complex bottleneck environments. By introducing a congestion node identification and allowable value quantification analysis mechanism, the width constraint of the physical path is accurately transformed into scheduling parameters. Based on this, a batch departure and interval adjustment strategy is designed to prevent congestion in narrow areas from the source, enabling the multi-machine system to maintain efficient and orderly flow in environments with inherent bottlenecks.

[0030] Meanwhile, this application has powerful dynamic task insertion and global resource optimization capabilities. Through task progress evaluation, resource recovery prediction, and multi-task joint optimization simulation, the system can quickly freeze existing plans, conduct a global inventory, and simulate various resource reuse schemes when an emergency task arrives, so as to achieve optimal connection between new tasks and existing tasks and dynamic resource reallocation, which greatly enhances the system's emergency response and multi-task parallel processing capabilities. Attached Figure Description

[0031] Figure 1 The method flowchart provided by the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1:

[0034] Please see Figure 1 This application provides a multimodal perception-based collaborative control method for loading and unloading cargo using an embodied intelligent robot;

[0035] As an embodiment of this application, it specifically includes:

[0036] Register and label the status of multiple intelligent robots to make them schedulable;

[0037] The system uses a visual sensor network to detect objects to be moved in real time and determine their quantity.

[0038] Transportation analysis based on target time includes: obtaining the shortest path of the transported goods from the origin to the destination; identifying congestion points and determining their allowable values ​​based on the relationship between path width and the travel width of the transported goods; and obtaining the number of goods leaving the same destination.

[0039] Conduct transportation simulations to obtain the number of single transfers and the time taken per trip, calculate the total transportation demand, and combine the number of outgoing shipments with the number of available transport vehicles to determine whether the target time is met;

[0040] If these conditions are not met, a transshipment plan is generated based on the location of the congestion point, loading time, and transportation speed by adjusting the batch quantity and departure interval.

[0041] During the transfer process, a short-range communication device is activated for each batch of transported goods, and real-time status information is exchanged when they enter each other's communication range, triggering an avoidance algorithm to achieve dynamic collaborative control.

[0042] Example 2:

[0043] This embodiment is based on Embodiment 1, but differs in that it describes the entire process of overall planning, scheduling, and collaborative control of a batch of known goods handling tasks in a standard warehouse environment.

[0044] Step 1: First, register and calibrate the status of the multiple embodied intelligent robots deployed in the warehouse to make them schedulable. For ease of description, we will refer to them as transport targets here.

[0045] The system uses a network of visual sensors, such as RGB-D cameras, deployed above the warehouse to detect in real time the objects that need to be moved on the current shelves. These objects can be a batch of outbound boxes. After analysis, a total of 30 boxes that need to be moved were identified. Of course, the user can also pre-enter the corresponding number of objects.

[0046] Step Two: The administrator inputs the target time (let's assume it's 30 minutes) through the central control system interface; then, transportation analysis is performed based on the target time. The specific analysis method is as follows:

[0047] Automatically obtain the shortest path from the starting point to the location of the transported object;

[0048] The width information of all path nodes is pre-stored in the map;

[0049] Obtain the required driving width for any transport object to travel safely, and mark it as the single standard width; here, the single standard width can generally be 1.2 to 1.5 times the width of the transport object;

[0050] These paths are automatically detected. When the passage width of any path segment is less than twice the single standard width, the continuous path segment is marked as a blockage node. At the same time, the allowable value of the corresponding blockage node is set to 1, indicating that only one transport object is allowed to pass through the blockage node.

[0051] If any path has a passage width less than three times the single standard width, then that continuous path is marked as a blocked node, and the allowable value of the corresponding blocked node is marked as 2. Similarly, if it is less than four times the single standard width, then the allowable value of the corresponding blocked node is 3. Based on this principle, all blocked nodes and their corresponding allowable values ​​are obtained.

[0052] Find the minimum value of the allowed value and mark it as the same out number;

[0053] Step 3: Conduct transportation simulation. The specific simulation method is as follows:

[0054] Controlling the actual transportation of any transport target allows for the stable capture and transport of a single transport process, marking the number of containers as the single transfer count. Here, we assume a single transfer count of 2 containers. Simultaneously, the time for a complete transport trip is obtained and multiplied by an increment to obtain the time, which is marked as the single trip time. The increment ranges from 1.1 to 1.5, typically 1.2. However, this value depends on the number of transport targets currently in operation; the more transport targets in operation, the larger the increment, indicating a greater likelihood of confusion due to the increased number of transport targets. The single trip time includes round-trip travel time and loading / unloading time.

[0055] There are currently 8 robots available in the warehouse. To cope with possible emergency tasks, the system sets a minimum retention rate of 20%, and it must not be lower than 2 robots. Therefore, the number of robots to be retained at this time is 2. Thus, the maximum number of robots that can be deployed for this task is 6.

[0056] The total transportation demand is automatically calculated as follows: 30 cargo boxes ÷ 2 (boxes / unit / trip) = 15 units / trip.

[0057] Then, obtain the number of vehicles with the same output. Divide 15 by the number of vehicles with the same output. If there is a decimal, round it up. If it is an integer, do not process it. This gives you the number of transfer trips. Multiply it by the time of a single trip to get the total time required when the corresponding number of vehicles with the same output are working at the same time. If the time required at this time is less than or equal to the target time, then dispatch the corresponding number of vehicles with the same output for transfer.

[0058] Otherwise, optimization will be performed, specifically as follows:

[0059] Obtain the blocking node where the allowed value is located corresponding to the same number of exits, and mark it as a card node. Obtain the location of all card nodes in the entire path.

[0060] Then set the interval time, where the interval time is less than or equal to the corresponding loading time, which is the time required for the transported goods to be loaded with the goods.

[0061] First, based on the loading time as the interval, two batches of transport objects with the same number of units are issued successively, and the objects are transferred. The total time of the handling order for each batch to be handled once is calculated.

[0062] Total time for a handling order = Time for one trip + Interval time;

[0063] Divide the transfer trip by 2 and multiply by the total time of the transport order to get the required transport time. Then compare it with the target time. If it is less than the target time, mark the above method as a transfer plan. If it is not less than the target time;

[0064] Divide the loading time by 2 to get the time mark as the interval time. Then, during the loading time, three batches of the same number of transport objects are allowed to be transferred. That is, after the first batch is sent out, the second batch is sent out after the interval time, and then the third batch is sent out after the interval time.

[0065] After the third batch is dispatched, it is necessary to verify whether the third batch of transported goods will meet at the card node. Regardless of the circumstances of the meeting, this is determined by the length and location of the card node, as well as the loading time and the average transport speed of the transported goods. If they will meet at the card node, the interval time will be reduced sequentially according to the set time difference. Here, the time difference is set to 15 seconds, but it can be set to other values.

[0066] Then calculate the required handling time and compare it with the target time. If it is still greater than the target time, dispatch four batches of transport objects with the same number of units to transfer the objects. Similarly, first divide the loading time by 3 to get the interval time. Then determine whether any two different batches of transport objects will meet at the card node. If they meet, reduce the interval time. Determine the handling time according to the same principle and compare it with the target time. If it is still greater than the target time, continue to add batches until the handling time is less than or equal to the target time.

[0067] If the required transported goods cannot meet the minimum retention of two units during this process, either request the relocation of transported goods or relocate them according to the largest batch that can be dispatched, and provide feedback on the required handling time in real time.

[0068] Step 4: When each batch of transported goods departs, the system activates a short-range communication device, such as UWB or Wi-FiDirect, and sets the communication distance to 3 meters; at the same time, the planned detailed path is backed up to the central control system server and the robot's local control unit respectively.

[0069] Step 5: During the execution of the mission, when any two batches of transported goods enter each other's communication distance of less than 3 meters, the two parties exchange real-time location, speed and mission intent through the communication link.

[0070] Triggering the avoidance algorithm: Lower priority batches of transport goods actively avoid each other. Here, "low priority" can refer to lightly loaded batches. The algorithm finds the nearest feasible avoidance point, such as a short stop in a groove on the side of the channel, to allow higher priority batches of transport goods to pass first. Alternatively, it can avoid any batch of transport goods, with only one batch being avoided between two batches.

[0071] After the reversal, both parties continued along the original path or the path dynamically adjusted by the system.

[0072] Step Six: Through the above process, even with path bottlenecks, the system still efficiently and safely completed the batch transportation task by coordinating multiple robots through precise departure scheduling, real-time communication, and dynamic obstacle avoidance, while maintaining two robots as emergency backup during the task.

[0073] Example 3:

[0074] This embodiment is based on Embodiment 2, but differs in that it aims to demonstrate how the system can quickly respond, replan, and coordinate existing transportation resources to achieve dynamic scheduling and seamless connection between multiple tasks when a new high-priority task is suddenly inserted while the system is executing a predetermined task.

[0075] S1. Initial task execution and status monitoring;

[0076] Assume that the system is executing a normal priority task, denoted as task α, according to the process of Example 2: 40 cargo boxes need to be moved, 4 transport vehicles are put into operation, denoted as R1-R4, and the number of vehicles dispatched at the same time is 2, that is, 2 vehicles are dispatched at the same time in each batch.

[0077] Mission α is running smoothly. R1 and R2, as the first batch, have already departed, while R3 and R4, as the second batch, are waiting in the loading area for the preset interval before departing. The short-range communication distance for all transported items is set to 3 meters and is operating normally.

[0078] S2, Emergency Task Insertion and Global Resource Reassessment

[0079] At this moment, the central control system received an emergency task β: eight precision instrument components, i.e., the target objects, needed to be moved immediately within 10 minutes. Analysis revealed that this task required high stability during transport, with only one component allowed for each transport operation.

[0080] S3. The system immediately initiates the dynamic scheduling process:

[0081] 1. Freeze the issuance of new batches of mission α: Immediately suspend the departure commands for R3 and R4.

[0082] 2. Global Resource Inventory: The total number of available transport targets remains at 8; Task α has already occupied 4 units R1-R4, with a minimum reserve of 2 units; therefore, theoretically, the maximum number that can be allocated to emergency Task β is: 8 units - 4 units (occupied) - 2 units (reserved) = 2 units. However, the progress of Task α needs to be considered.

[0083] 3. Task α progress assessment and resource recovery prediction: The system calculates the real-time location and status of R1 and R2, and predicts the time when they return to the loading area and complete unloading, which is recorded as Tr; it is found that R1 and R2 will return in 3 minutes and enter an idle state that can be rescheduled.

[0084] 4. Emergency Task β Path and Bottleneck Analysis: Plan the path to the precision instrument warehouse; analysis reveals a critical passage in the path, whose width can only accommodate 1.5 times the single standard width, and is therefore marked as a bottleneck node with a passability value of 1. Hence, the number of simultaneous exits for Task β is 1.

[0085] Step 3: Conduct multi-task joint optimization simulation:

[0086] Option A: Immediate Response: Immediately dispatch one unit from the waiting R3 and R4 units to ensure a simultaneous outgoing quantity of 1, and execute the first transport of task β; calculate the time per trip, including fine loading / unloading. Since the simultaneous outgoing quantity is 1, 8 units are needed for 8 transports of 8 items. If only one unit is dispatched, 8 trips are required, and the total time far exceeds the 10-minute target.

[0087] Option B: Resource Relay. Based on resource recovery forecasts, the plan is as follows:

[0088] Phase 1: 0-3 minutes, immediately dispatch R3 to perform the first transport of mission β, with a co-transport count of 1.

[0089] Phase 2: After 3 minutes: R1 and R2 return, idle. At this time, there are 7 items remaining in task β, and the available transport targets are R1, R2, and R4. R3 is still in task β. Since the number of simultaneous departures is 1, but the allowable value of the blocking node is 1, it only limits the number of simultaneous passages and does not prohibit sequential fast passage. The system can plan for R1, R2, and R4 to depart sequentially with the minimum safe interval, such as 45 seconds, to execute the subsequent transport of task β.

[0090] Resumption of Task α: After Task β is satisfied first, the released transport target will continue to execute the remaining part of Task α.

[0091] The total time taken by the system to calculate scheme B meets the 10-minute target for task β, and the delay for task α is within an acceptable range. Therefore, scheme B is adopted.

[0092] Step 4: During the execution of Plan B, R3, which is performing task β, and R2, which is returning from task α, may meet on a common path. When they enter a 3-meter communication range, the avoidance algorithm is activated. The algorithm uses task priority as the primary criterion: task β is urgent, while task α is ordinary. Therefore, even if R2 is fully loaded and R3 is unloaded, R2 must actively find a way to allow R3, which is performing the urgent task, to pass first. This demonstrates that in multi-task collaboration, the avoidance strategy has evolved from comparing individual vehicle attributes to comparing the global priority of tasks.

[0093] Step 5: Task Recovery and Completion

[0094] After task β is completed, the status of all transport targets is updated. The system recalculates the optimal subsequent transport plan for task α. At this point, the number and location of available transport targets have changed. Schedulers R1-R4 continue to complete the remaining work and reassess the overall completion time.

[0095] This embodiment demonstrates that when faced with the insertion of dynamic tasks, the system can quickly interrupt existing plans, comprehensively assess resources, perform multi-task joint scheduling and resource reuse planning, and ensure that high-priority tasks are executed efficiently and safely through a cross-task collaborative avoidance mechanism based on task priority, reflecting the dynamic adaptability and scheduling intelligence of the method.

[0096] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method, characterized in that, include: Register and label the status of multiple intelligent robots to make them schedulable; The system uses a visual sensor network to detect objects to be moved in real time and determine their quantity. Transportation analysis based on target time includes: obtaining the shortest path of the transported goods from the origin to the destination; identifying congestion points and determining their allowable values ​​based on the relationship between path width and the travel width of the transported goods; and obtaining the number of goods leaving the same destination. Conduct transportation simulations to obtain the number of single transfers and the time taken per trip, calculate the total transportation demand, and combine the number of outgoing shipments with the number of available transport vehicles to determine whether the target time is met; If these conditions are not met, a transshipment plan is generated based on the location of the congestion point, loading time, and transportation speed by adjusting the batch quantity and departure interval. During the transfer process, a short-range communication device is activated for each batch of transported goods, and real-time status information is exchanged when they enter each other's communication range, triggering an avoidance algorithm to achieve dynamic collaborative control.

2. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, The allowable value is determined as follows: If the path width is less than twice the single standard width, the allowable value is 1; if it is less than three times the single standard width, the allowable value is 2; and so on. The single standard width is 1.2 to 1.5 times the driving width of the transport vehicle.

3. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, The single trip time is the actual time spent transporting a vehicle multiplied by an increment, where the increment ranges from 1.1 to 1.5 and increases with the number of vehicles being transported.

4. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, The methods for generating a transit plan include: The transportation targets are dispatched in batches based on the loading time as the initial interval; it is verified whether different batches will meet at the congestion point. If they do meet, the interval is gradually reduced until the time requirement is met.

5. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, In the obstacle avoidance algorithm, low-priority transport objects are actively avoided, and the priority is determined based on the load weight or the urgency of the task.

6. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, The short-range communication device uses UWB or Wi-FiDirect, with a communication distance set at 3 meters.

7. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, It also includes the step of dynamically inserting emergency tasks during transportation: Suspend subsequent batches of the current mission; Reassess available transport resources; Plan routes and determine the number of co-originating points for emergency missions; Perform multi-task joint optimization simulation to generate resource reuse schemes.

8. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 7, characterized in that, Multi-task joint optimization simulation includes: phased scheduling of transportation targets, combined with resource recovery prediction, to achieve seamless connection between tasks.

9. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 7, characterized in that, During the execution of an emergency mission, if the vehicle encounters a vehicle intended for ordinary mission transport, the vehicle will be given priority and given the right of way based on mission priority.

10. The multimodal perception-based embodied intelligent robot cargo loading and unloading collaborative control method according to claim 1, characterized in that, The system sets the minimum number of transport targets to be retained to be 20% of the total number of robots, and no less than 2 robots.

Images