Multi-robot cooperation task processing method and system and electronic equipment

Abstract

The embodiment of the invention provides a multi-robot cooperation task processing method and system and electronic equipment, and relates to the technical field of robots, and the method comprises the steps: generating a time sequence path of each to-be-transported object from an initial pose to a target pose based on the initial poses and target poses of a plurality of to-be-transported objects; dividing the time sequence path of each to-be-transported object into a plurality of path sections, and determining a partial order relationship between the path sections of the to-be-transported objects; the partial order relation is used for representing the execution sequence of the subtasks; allocating a corresponding robot to each subtask based on the state information of the plurality of robots and the partial order relationship among the subtasks; for each sub-task, generating a motion plan for the robot executing the sub-task based on the starting pose and the ending pose of the sub-task; and sending the motion plan to the corresponding robot, so that the robot executes a transportation task of the corresponding subtask based on the motion plan, and processing of a multi-robot cooperative task in a complex environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, in particular to the field of multi-robot collaborative operation technology, and specifically to a method, system and electronic equipment for processing multi-robot collaborative tasks. Background Art

[0002] In robotics, pushing, a non-grasping skill, is particularly valuable for low-cost mobile robots without manipulator arms, such as ground vehicles and quadrupeds. This skill allows robots to clear obstacles and move objects to designated locations. By simultaneously applying force from different points, multiple robots can push the same object simultaneously, improving the feasibility and efficiency of robotic operations while overcoming the load-bearing limitations of individual robots.

[0003] Related Technology In the task of multiple robots collaborating to push an object, multiple robots (whose size may be much smaller than the object to be pushed) push the same object synchronously from different points. Each robot uses its own camera to determine whether it can see the target position of the object directly. If the robot determines that it cannot see the target position directly, that is, the target position is blocked, the robot pushes the object along the normal direction of the blocked area; if the robot determines that it can see the target position, that is, the target position is not blocked, the robot will circle along the outside of the object or the search route until it enters the blocked area of ​​the target position. Multiple robots push the object using this strategy, so that the vector sum of the thrust acting on the object always approximately points to the target position, and each step away from the target position is strictly reduced, thereby ensuring that any convex object can be pushed to the predetermined target position in a planar environment without additional obstacles.

[0004] However, the strategy used in the aforementioned multi-robot collaborative object-pushing task, as long as a robot determines that the target location is obstructed, assumes that the object is in front of it and pushes it. This can easily cause the robot to apply force to obstacles such as walls or corners of narrow passages instead of transferring force to the object, causing one end of the object to be stuck against the obstacle and unable to move. When multiple robots make misjudgments, the simultaneous pushing of the object by multiple robots causes the direction of the thrust transferred to the object to constantly change, which can easily get the object stuck in place. Obviously, the strategy used in the aforementioned multi-robot collaborative object-pushing task is suitable for simple, open spaces, but is prone to deadlock in complex spaces with obstacles. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a method, system, and electronic device for processing multi-robot collaborative tasks, so as to realize the processing of multi-robot collaborative tasks in complex environments and improve the execution efficiency and safety of multi-robot collaborative tasks in uncertain environments. The specific technical solutions are as follows:

[0006] An embodiment of the present invention provides a method for processing a multi-robot collaborative task, the method comprising:

[0007] Based on the initial positions and target positions of multiple objects to be transported, a temporal path from the initial position to the target position of each object to be transported is generated; the temporal path of each object to be transported does not have collisions in the time and space dimensions; the temporal path of each object to be transported is divided into multiple path segments, and the partial order relationship between the path segments of each object to be transported is determined; wherein, each path segment represents a subtask, and the partial order relationship is used to represent the execution order between subtasks; based on the status information of multiple robots and the partial order relationship between the subtasks, a corresponding robot is assigned to each subtask; for each subtask, based on the starting position and ending position of the subtask, a motion plan is generated for the robot that executes the subtask; the motion plan is sent to the corresponding robot so that the robot executes the transportation task of the corresponding subtask based on the motion plan.

[0008] Optionally, the method of dividing the time sequence path of each object to be transported into multiple path segments and determining the partial order relationship between the path segments of each object to be transported includes: determining whether there is an intersection between the time sequence paths of different objects to be transported; based on the intersection, determining the segmentation point contained in the time sequence path of the object to be transported where the intersection is located; based on the segmentation point, dividing the time sequence path of each object to be transported into multiple path segments; and determining the partial order relationship between the path segments based on the time sequence relationship between the path segments.

[0009] Optionally, the timing path contains the postures of the objects to be transported at different time stamps arranged in time sequence; the timing path of each object to be transported is divided into multiple path segments, and the partial order relationship between the path segments of each object to be transported is determined, including: for each timing path, comparing the postures contained in the timing path with the postures contained in other timing paths one by one, and determining whether there is an intersection between the timing path and the other timing paths based on the comparison result; for each intersection of the timing path with other timing paths, determining whether the first time stamp corresponding to the intersection in the timing path is later than the second time stamp corresponding to the intersection in the other timing paths; if so, determining the intersection as the segmentation point of the timing path; dividing the timing path into path segments based on the determined segmentation point; and determining the partial order relationship between the path segments based on the timing relationship between the path segments of each object to be transported.

[0010] Optionally, the method of assigning a corresponding robot to each subtask based on the status information of multiple robots and the partial order relationship between the subtasks includes: initializing and generating a task assignment search tree; selecting a subtask to be assigned as the root node of the task assignment search tree based on the partial order relationship between the subtasks and the status of each subtask, and assigning an idle robot to the root node; constructing the task assignment search tree based on a selection expansion strategy; determining a target task assignment scheme from the constructed task assignment search tree; wherein the target task assignment scheme includes the identifier of the subtask and the robot assigned to the subtask.

[0011] Optionally, for each subtask, based on the starting posture and ending posture of the subtask, a motion plan is generated for the robot performing the subtask, including: for each subtask, the starting posture and ending posture of the subtask are used as the starting keyframe and the target keyframe respectively; the starting keyframe is the keyframe of the propulsion mode to be assigned; determining whether there is a keyframe of the propulsion mode to be assigned; if there is a keyframe of the propulsion mode to be assigned, selecting a keyframe whose estimated cost meets a preset condition; for the first keyframe of the selected keyframes to be assigned a propulsion mode, determining whether the arc segment between the keyframe and the next adjacent keyframe intersects with the obstacle; if so, inserting an intermediate keyframe between the keyframe and the next adjacent keyframe, and returning to the step of determining whether there is a keyframe of the propulsion mode to be assigned; if not, assigning the corresponding propulsion mode to the keyframe.

[0012] Optionally, assigning a corresponding push mode to the key frame includes: querying a preset action set, and determining a corresponding predicted push mode for the key frame based on the correspondence between the key frames and push modes contained in the preset action set; performing a feasibility analysis on the predicted push mode to determine whether the predicted push mode is feasible; if the predicted push mode is feasible, determining the predicted push mode as the push mode corresponding to the key frame; if the predicted push mode is not feasible, generating a corresponding push mode for the key frame through a sparse optimization method.

[0013] Optionally, the motion plan includes a key frame and a pushing pattern sequence corresponding to the subtask; sending the motion plan to the corresponding robot so that the robot performs the transportation task of the corresponding subtask based on the motion plan includes: for each subtask, sending the motion plan corresponding to the subtask to the corresponding robot; determining a reference frame based on the arc trajectory corresponding to the current frame and the next frame of the current frame in the subtask; calculating the contact point and contact direction of the robot on the object to be transported at the reference frame based on the pushing pattern from the current frame to the next frame of the current frame, and sending the contact point and contact direction to the robot so that the robot transports the object to be transported according to the contact point and contact direction; detecting whether the object to be transported deviates from the motion plan corresponding to the subtask; if deviated, taking the current frame as the starting posture of the subtask, and returning to execute the step of generating a motion plan for the robot performing the subtask based on the starting posture and ending posture of the subtask for each subtask; when it is detected that the subtask is completed, marking the status of the subtask as task completed, and marking the status of the robot that completes the subtask as idle.

[0014] Optionally, the method of generating a temporal path from the initial pose to the target pose of each object to be transported based on the initial poses and target poses of multiple objects to be transported includes: based on the initial poses and target poses of multiple objects to be transported, using a multi-agent path planning algorithm to generate a temporal path from the initial pose to the target pose of each object to be transported.

[0015] An embodiment of the present invention further provides a multi-robot collaborative task processing system, the system comprising:

[0016] A path generation module is used to generate a time sequence path for each object to be transported from the initial position to the target position based on the initial position and the target position of the multiple objects to be transported; the time sequence path of each object to be transported is collision-free in the time and space dimensions;

[0017] A path partitioning module is used to divide the sequential path of each object to be transported into multiple path segments and determine a partial order relationship between the path segments of each object to be transported; wherein each path segment represents a subtask, and the partial order relationship is used to represent the execution order of the subtasks;

[0018] A task assignment module is used to assign a corresponding robot to each subtask based on the status information of multiple robots and the partial order relationship between the subtasks;

[0019] A plan generation module is used to generate a motion plan for the robot performing each subtask based on the starting and ending poses of the subtask;

[0020] The task execution module is used to send the motion plan to the corresponding robot so that the robot executes the transportation task of the corresponding subtask based on the motion plan.

[0021] An embodiment of the present invention also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-mentioned multi-robot collaborative task processing methods when executing the programs stored in the memory.

[0022] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for processing multi-robot collaborative tasks.

[0023] Beneficial effects of the embodiments of the present invention:

[0024] The embodiments of the present invention provide a method, system, and electronic device for processing multi-robot collaborative tasks. Based on the initial positions and target positions of multiple objects to be transported, a time sequence path is generated for each object to be transported from the initial position to the target position. The generated time sequence path for each object to be transported does not have collisions in the spatiotemporal dimension, so that in the process of generating the time sequence path, not only the shape of the object but also environmental obstacles are considered, and thus the method can be used for processing multi-robot collaborative tasks in complex environments. The time sequence path of each object to be transported is divided into multiple path segments, and the partial order relationship between the path segments of each object to be transported is determined to ensure the rationality of the task execution order and that no collisions occur during the task execution. Furthermore, based on the status information of multiple robots and the partial order relationship between each subtask, a corresponding robot is assigned to each subtask, so that the task assignment can follow the partial order relationship, thereby improving the execution efficiency and safety of multi-robot collaborative tasks in uncertain environments. For each subtask, a motion plan is generated for the robot that performs the subtask based on the starting and ending positions of the subtask, and the motion plan is sent to the corresponding robot so that the robot can perform the transportation task of the corresponding subtask based on the motion plan, realizing the processing of multi-robot collaborative tasks in complex environments.

[0025] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0027] Figure 1 A schematic flow chart of a method for processing a multi-robot collaborative task provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a flow chart of a path segment division method provided in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of another flow chart of a path segment division method provided in an embodiment of the present invention;

[0030] Figure 4 A flowchart of a task allocation method provided by an embodiment of the present invention;

[0031] Figure 5 A schematic diagram of a flow chart of a motion planning generation method provided by an embodiment of the present invention;

[0032] Figure 6 A flowchart of a task execution method provided by an embodiment of the present invention;

[0033] Figure 7a A schematic diagram of multi-robot collaborative task processing provided by an embodiment of the present invention;

[0034] Figure 7b Another schematic diagram of multi-robot collaborative task processing provided by an embodiment of the present invention;

[0035] Figure 7c Another schematic diagram of multi-robot collaborative task processing provided by an embodiment of the present invention;

[0036] Figure 8 A schematic structural diagram of a multi-robot collaborative task processing system provided by an embodiment of the present invention;

[0037] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0039] In multi-robot collaborative tasks, such as pushing an object, different modes—long-side pushing, short-side pushing, diagonal pushing, and frame pushing—can generate translational and rotational motions. However, the quality of the motion varies depending on the object's physical properties, such as mass distribution, shape, and friction coefficient. Therefore, when implementing multi-robot collaborative pushing of multiple objects in complex environments, considerations must be given to: the strong coupling of object trajectories in time and space, which can easily lead to collisions and deadlocks; and the inherent uncertainty in the feasibility of pushing different objects when executing pushing tasks online.

[0040] Through consideration of the above issues, in order to realize the processing of multi-robot collaborative tasks in complex environments and improve the execution efficiency and safety of multi-robot collaborative tasks in uncertain environments, an embodiment of the present invention provides a method, system and electronic device for processing multi-robot collaborative tasks. The method for processing multi-robot collaborative tasks provided by the embodiment of the present invention can be applied to scenarios such as the collaborative task processing of multi-robot collaborative transportation (such as pushing or moving) of objects, where the objects can be multiple objects or a single object. The method for processing multi-robot collaborative tasks according to the embodiment of the present invention can be applied to electronic devices, such as servers, controllers, etc.

[0041] The following describes in detail a method for processing a multi-robot collaborative task provided by an embodiment of the present invention.

[0042] like Figure 1 As shown, an embodiment of the present invention provides a method for processing a multi-robot collaborative task, including:

[0043] S101, based on the initial positions and target positions of a plurality of objects to be transported, generating a temporal path from the initial position to the target position of each object to be transported;

[0044] Among them, the temporal path of each object to be transported does not have collision in the space-time dimension;

[0045] S102, dividing the time sequence path of each object to be transported into multiple path segments, and determining a partial order relationship between the path segments of each object to be transported;

[0046] Each path segment represents a subtask, and the partial order relationship is used to represent the execution order between subtasks;

[0047] S103, assigning a corresponding robot to each subtask based on the status information of the multiple robots and the partial order relationship between the subtasks;

[0048] S104, for each subtask, generating a motion plan for the robot performing the subtask based on the starting pose and ending pose of the subtask;

[0049] S105 , sending the motion plan to the corresponding robot, so that the robot performs the transportation task of the corresponding subtask based on the motion plan.

[0050] In an embodiment of the present invention, a temporal path is generated for each object to be transported, based on the initial and target positions of multiple objects to be transported. The generated temporal paths for each object to be transported are collision-free in both time and space, allowing for the generation of temporal paths that not only consider the object shape but also environmental obstacles. This approach can be used to handle multi-robot collaborative tasks in complex environments. The temporal path for each object to be transported is divided into multiple path segments, and a partial order relationship is determined between the path segments of each object to be transported to ensure the rationality of the task execution sequence and the avoidance of collisions during task execution. Furthermore, based on the status information of multiple robots and the partial order relationship between each subtask, a corresponding robot is assigned to each subtask, ensuring that task assignment follows the partial order relationship, thereby improving the efficiency and safety of multi-robot collaborative tasks in uncertain environments. For each subtask, a motion plan is generated for the robot performing the subtask based on the subtask's starting and ending positions. The motion plan is then sent to the corresponding robot, allowing the robot to perform the transportation task of the corresponding subtask based on the motion plan, thus enabling the handling of multi-robot collaborative tasks in complex environments.

[0051] In S101, the object to be transported can be any object that needs to be transported. Transporting here can mean pushing, moving, carrying, etc. In one example, the initial position and target position of the object to be transported represent the initial position and initial angle, and the target position and target angle, etc., of the object to be transported, respectively. In this embodiment of the present invention, the specific shape of the object is not limited; the position of the object can be coordinate information, or latitude and longitude information, etc.

[0052] In one possible implementation, a multi-agent path finding (MAPF) algorithm can be used to generate a temporal path from the initial position to the target position of each object to be transported based on the initial positions and target positions of multiple objects to be transported.

[0053] Wherein, the multi-agent path planning algorithm is a method for planning a collision-free path for a plurality of robots from a starting position to a target position. The main goal of the multi-agent path planning algorithm is to ensure that all robots can move simultaneously, avoid collisions, and optimize the overall running time and cost.

[0054] Exemplarily, in the case of knowing the initial pose and the target pose of the object to be transported m, wherein, representing a set of objects to be transported, using a multi-agent path planning algorithm, in combination with the position information of obstacles in the scene where the multi-robot cooperative task is located and the shape of the object to be transported m, a time sequence path of the object to be transported m from the initial pose to the target pose is generated The generated time sequence path contains a plurality of poses (including position and angle, etc.) of the object to be transported m in time sequence from the initial pose to the target pose. The set of time sequence paths of a plurality of objects to be transported is represented as Each time sequence path of the object to be transported generated using the multi-agent path planning algorithm does not have a collision in the time-space dimension, thereby ensuring that the robots transporting the objects to be transported on the time sequence path will not collide with each other or with obstacles. Here, the time interval corresponding to the time sequence can be set according to the actual situation, for example, it can be set as a fixed time interval, such as 0.05 seconds, 0.1 seconds, etc., and it can also be set as a variable step time interval, such as a step of increasing 0.01 seconds each time. It should be noted that the time interval (i.e. time step) corresponding to the time sequence is not the actual time interval during task execution.

[0055] In the embodiment of the present application, the multi-agent path planning algorithm is used to generate a time sequence path of each object to be transported from the initial pose to the target pose, and each time sequence path of the object to be transported generated does not have a collision in the time-space dimension, so that not only the shape of the object is considered in the process of generating the time sequence path, but also the environmental obstacles are considered, thereby being able to be used for processing of multi-robot cooperative tasks in complex environments, and also ensuring that the robots transporting the objects to be transported on the time sequence path will not collide with each other or with obstacles.

[0056] In S102, for each time sequence path of the object to be transported, the time sequence path is decomposed into a plurality of path segments, and then a partial order relationship between the path segments of each object to be transported is established according to the time-space intersection between the path segments of each object to be transported. In the embodiment of the present application, each path segment is represented as a subtask, and accordingly, the partial order relationship between the path segments represents the execution order between the subtasks, that is, the decomposed path segments have a precedence relationship in execution order.

[0057] In S103, based on the determined partial order relationship between the subtasks and the status information of multiple robots, a corresponding robot or robot group is dynamically assigned to each subtask, under the conditions that: each robot has enough time to navigate between consecutive subtasks and the partial order relationship between the subtasks is respected, so as to improve the execution efficiency and safety of multi-robot collaborative tasks in uncertain environments.

[0058] After assigning a corresponding robot or robot group to each subtask, a motion plan is generated for the robot or robot group performing each subtask based on the starting and ending poses of each subtask in S104. The generated motion plan may include the propulsion mode used by the robot to perform the subtask and the propulsion speed within the propulsion mode. In one example, the propulsion mode includes the contact points of the robot transporting the object to be transported.

[0059] In one embodiment of the present invention, the generation of a motion plan can be achieved based on the quasi-static force balance assumption, that is, assuming that the speed of the object to be transported is slow enough, its acceleration is approximately zero, the inertial force can be ignored, and the object to be transported is always in a state of static equilibrium. Under this assumption, the robot's propulsion force and the ground friction force satisfy the equilibrium relationship: q_push + q_friction = 0, where q_push is the generalized propulsion force applied by the robot to the object to be transported, and q_friction is the generalized resistance generated by ground friction. Through this equilibrium relationship, the feasibility of the movement of the object to be transported under different propulsion modes can be predicted and verified, so as to better generate motion plans.

[0060] In S105, for each subtask, the motion plan corresponding to the subtask is sent to the robot or robot group that executes the subtask. Upon receiving the motion plan, the robot or robot group executes the transportation task of the path segment corresponding to the subtask according to the pushing mode and corresponding pushing speed contained in the received motion plan.

[0061] In one possible implementation, Figure 2 As shown, the above step S102 divides the temporal path of each object to be transported into multiple path segments and determines the partial order relationship between the path segments of each object to be transported, which may include:

[0062] S201, determining whether there is an intersection between the time sequence paths of different objects to be transported.

[0063] The temporal path of the object to be transported contains multiple postures arranged in time from the initial posture to the target posture. For each object to be transported, the posture contained in the temporal path of the object to be transported is compared with the postures contained in the temporal paths of other objects to be transported by traversal. If there is a posture overlap (that is, the position and angle are the same), it means that the temporal paths of the two objects to be transported have an intersection, otherwise there is no intersection, so as to determine whether the temporal paths of different objects to be transported have an intersection.

[0064] S202 : Based on the intersection, determine the segmentation points included in the time sequence path of the object to be transported where the intersection is located.

[0065] For any intersection of the time sequence paths of two objects to be transported, whether the intersection is a segmentation point in the time sequence paths of the objects to be transported is determined based on the order in which the objects to be transported arrive at the intersection.

[0066] In an example, there is an intersection between the timing paths of the objects m1 and m2 to be transported. The time point corresponding to the intersection in the timing path of the object m1 to be transported is t1, and the time point corresponding to the intersection in the timing path of the object m2 to be transported is t2. t1 is earlier than t2, that is, the object m1 to be transported reaches the intersection first in time and space. Then, for the object m1 to be transported, the intersection is not a segmentation point, but for the object m2 to be transported, the intersection is a segmentation point. Based on this principle, whether the intersection in the timing paths of different objects to be transported is a segmentation point is determined, and the segmentation points contained in the timing paths of each object to be transported are obtained.

[0067] S203 : Divide the time sequence path of each object to be transported into multiple path segments based on the segmentation points.

[0068] For example, the time sequence path of the object m to be transported is After determining the segmentation points included in the temporal path of the object to be transported, the path between the two segmentation points is divided into a path segment. For example, the temporal posture between the two segmentation points is placed in a path segment set to obtain a path segment, thereby dividing the temporal path into Decomposed into multiple path segments represented as Represents the timing path The first path segment of Represents the timing path The kth path segment of .

[0069] S204: Determine a partial order relationship between the path segments based on the temporal order relationship between the path segments.

[0070] Because the temporal path of the object to be transported contains multiple postures arranged in time from the initial posture to the target posture, and the segmentation point is determined according to the order in which the object to be transported arrives at the intersection, then correspondingly, each path segment in the temporal path of the object to be transported obtained by division corresponds to time information, and then directly based on the temporal relationship between each path segment, the partial order relationship between each path segment can be determined. This partial order relationship characterizes the execution order between the subtasks corresponding to the path segment, so as to ensure that the object to be transported and the robot transporting the object to be transported will not collide with each other during the execution of subsequent tasks. Exemplarily, the partial order relationship between each path segment can be expressed as sequence data of the order between different path segments (subtasks), such as subtask 1 precedes subtask 2 and subtask 3, subtask 2 precedes subtask 4, subtask 3 precedes subtask 5 and subtask 6, etc.

[0071] In an embodiment of the present invention, a split point is determined according to the order in which the object to be transported and other objects to be transported arrive at the intersection point in the time sequence path, and the time sequence path is divided according to the split point, so that there is a strict partial order relationship between the divided path segments, and the partial order relationship between the path segments of each object to be transported is determined in this way, which can ensure the rationality of the task execution order and prevent collisions during the task execution process.

[0072] In a possible implementation, the above-mentioned time sequence path includes the positions of the objects to be transported at different time stamps arranged in time sequence; Figure 3 As shown, the above step S102 divides the time sequence path of each object to be transported into multiple path segments, and determines the partial order relationship between the path segments of each object to be transported, including:

[0073] S301 , for each timing path, comparing the postures included in the timing path with the postures included in other timing paths one by one, and determining whether there is an intersection between the timing path and the other timing paths based on the comparison results.

[0074] The temporal path of the object to be transported includes the postures of the object to be transported at different time stamps arranged in time sequence. For example, it can be represented as a sequence of (time stamp, posture). The temporal paths of the object to be transported are traversed, and for each temporal path of the object to be transported, the postures included in the temporal path are compared one by one with the postures included in other temporal paths to determine whether there are overlapping postures. If there are overlapping postures, it indicates that the temporal path has an intersection with the other temporal paths. Otherwise, the temporal path does not have an intersection with the other temporal paths.

[0075] S302 : For each intersection point in the timing path with other timing paths, determine whether a first timestamp corresponding to the intersection point in the timing path is later than a second timestamp corresponding to the intersection point in other timing paths.

[0076] S303: If the first timestamp corresponding to the intersection in the timing path is later than the second timestamp corresponding to the intersection in other timing paths, determine the intersection as a split point of the timing path.

[0077] Each intersection point in the timing path with other timing paths is traversed to determine whether the intersection point is a segmentation point of the timing path based on the order of the timestamps of the intersection point in the timing path and other timing paths, and the end point in the timing path (i.e., the target posture) is also determined as a segmentation point.

[0078] Because each intersection point of each timing path with other timing paths is traversed, if the first timestamp corresponding to the intersection point in the timing path is not later than the second timestamp corresponding to the intersection point in other timing paths, no processing is required.

[0079] S304: Divide the timing path into path segments based on the determined segmentation points.

[0080] Exemplarily, the timing path is represented as a sequence of (timestamp, posture). Accordingly, after determining the segmentation points contained in the timing path, the (timestamp, posture) between the two segmentation points is added to a path segment set to form multiple path segment sets, thereby realizing the division of the timing path into multiple path segments.

[0081] S305 : Determine a partial order relationship between the path segments based on the temporal order relationship between the path segments of the objects to be transported.

[0082] The implementation process of this step can refer to the implementation process of the above-mentioned step S204, and will not be repeated here in this embodiment of the present invention.

[0083] In an embodiment of the present invention, the time sequence path of each object to be transported is traversed, and a split point is determined according to the order in which the object to be transported and other objects to be transported arrive at the intersection point in the time sequence path. The time sequence path is divided by using this split point, so that there is a strict partial order relationship between the divided path segments, and the partial order relationship between the path segments of each object to be transported is determined by this, which can ensure the rationality of the task execution order and prevent collisions during task execution.

[0084] In one possible implementation, after decomposing the sequential paths of the objects to be transported and determining the partial order relationship between the path segments of the objects to be transported, a tree search method can be used to combine the status information of multiple robots to assign a corresponding robot or robot group to each subtask. The tree search method is a method for searching and solving problems through a tree structure. Figure 4As shown, the above step S103 allocates a corresponding robot to each subtask based on the status information of multiple robots and the partial order relationship between the subtasks, including:

[0085] S401, initializing and generating a task allocation search tree.

[0086] When task assignment is required, an empty task assignment search tree is initialized. For example, the task assignment search tree can be expressed as Tr = ({V}, E), where {V} represents the node set, E represents the edge set, and the initial node V0 is in an empty assignment state.

[0087] S402 : Based on the partial order relationship between the subtasks and the status of the subtasks, a subtask to be assigned is selected as the root node of the task assignment search tree, and an idle robot is assigned to the root node.

[0088] In this embodiment of the present invention, task allocation adheres to the following two conditions: first, each robot has sufficient time to navigate between consecutive subtasks; second, the partial order relationship between all subtasks is respected. Based on these conditions, in this embodiment of the present invention, each path segment of each object to be transported is represented as a subtask. At the beginning of task allocation, each subtask is in the state of pending assignment. During the task allocation process, if a corresponding robot or robot group has been assigned to a subtask, the subtask's state is updated to assigned subtask. After task allocation is completed, during task execution, if a subtask has been completed, the subtask's state is updated to completed subtask.

[0089] In one example, during the task assignment process, a list of subtasks to be assigned can be maintained, and each subtask to be assigned can be cached in the list of subtasks to be assigned according to the partial order relationship between the subtasks. Furthermore, the first subtask to be assigned in the list of subtasks to be assigned is selected as the root node of the task assignment search tree, and an idle robot or robot group is assigned to the root node, that is, an idle robot or robot group is assigned to the first subtask to be assigned, and then the selected first subtask to be assigned is deleted from the list of subtasks to be assigned, and the assigned robot or robot group is marked as non-idle, or the assigned robot or robot group is marked as non-idle during the time period corresponding to the execution of the selected first subtask to be assigned. The time period corresponding to the execution of the selected first subtask to be assigned is estimated based on the length of the path segment corresponding to the first subtask to be assigned and the number of assigned robots.

[0090] Exemplarily, one or a group of robots can be randomly selected from the robots in the idle state and assigned to the root node. Preferably, one or a group of robots can be randomly selected from the robots in the idle state and not assigned to other subtasks and assigned to the root node.

[0091] S403: Construct a task allocation search tree based on the selected expansion strategy.

[0092] After selecting a root node and assigning a corresponding robot to it, a task assignment search tree is constructed by selecting subtasks from the waiting list based on the partial order between subtasks and the estimated spread values ​​of the subnodes. These subtasks are then assigned to idle robots until a pre-defined termination condition is met. Each node represents a task assignment state, recording the identifiers of the assigned subtasks and the robots to which they were assigned.

[0093] For example, the task assignment search tree generated above is represented as Tr=({V}, E), which consists of a node set {V} and an edge set E: {V→V +}, V→V + Indicates that from node V to node V + Each node V in the task allocation search tree represents a partial task allocation state, recording the currently assigned subtasks and the identities of the robots assigned to them. Initially, node V0 is in an empty allocation state. As the search progresses, the tree gradually expands to form multiple possible allocation solutions.

[0094] The process of constructing a task assignment search tree can include two stages: selection and expansion. First, in the selection stage, the estimated expansion value V is selected from the existing node set {V} * The largest child node, where the estimated expansion value is calculated by the following expression:

[0095]

[0096] Among them, V * Indicates the estimated expansion value corresponding to the child node, represents the kth subtask of the object m to be transported, S V Indicates the assigned subtask set corresponding to the child node, represents the length of the path segment corresponding to the kth subtask of the object m to be transported, represents the robot assigned to the kth subtask of transporting object m, Represents a robot Complete subtasks Estimated end time.

[0097] Then in the expansion phase, from the remaining unassigned subtasks set Select a subtask to be assigned The subtask satisfies the partial order constraint: That is, all predecessor subtasks have been allocated in the current node. Then for the selected subtask Randomly select one or a group of robots from the robots that are in the idle state or currently in the idle state for allocation, and create the next child node of the current node.

[0098] Repeat the above selection-expansion process until the preset termination condition is reached. The preset termination condition can be any of the following: the number of assigned subtasks corresponding to the current node reaches the preset value H, which can be expressed as |S V |≥H; or, there is no unassigned subtask, i.e. Alternatively, the search time reaches the budgeted time. In one example, the preset value is set in advance based on the number of subtasks to be assigned and the number of robots. The search time represents the time spent on building the task assignment search tree, and the budgeted time is set based on the actual situation.

[0099] S404: Determine a target task allocation solution from the constructed task allocation search tree.

[0100] Each node in the constructed task allocation search tree represents a task allocation state. Then, the task allocation state corresponding to the node with the largest estimated expansion value can be selected from the task allocation search tree as the target task allocation scheme, which includes the subtask and the identifier of the robot assigned to the subtask.

[0101] To reduce the amount of calculation, the estimated expansion value of each child node calculated during the construction of the task allocation search tree can be recorded, and then when determining the target task allocation plan, the child node corresponding to the maximum estimated expansion value can be directly selected.

[0102] For example, the target task allocation scheme corresponding to the node with the largest estimated expansion value may include the local task plan τ of each robot. j ,in, τ j Represents the subtask sequence that robot j needs to perform in sequence And the start time t1, t2, ... of each subtask. The local task plan τ h This allows the robot to clearly understand: the order of each subtask, the start time of each subtask, and the robots involved in each subtask.

[0103] In an embodiment of the present invention, task allocation is performed through a tree search method. The task allocation is implemented based on the partial order relationship between the subtasks, so that the allocation of the latter subtask depends on the previous subtask. The partial order relationship between the subtasks can be respected, thereby improving the execution efficiency and safety of multi-robot collaborative tasks in uncertain environments.

[0104] In one possible implementation, Figure 5 As shown, the above step S104 generates a motion plan for the robot performing each subtask based on the starting posture and ending posture of the subtask, including:

[0105] S501 : For each subtask, the starting pose and the ending pose of the subtask are used as the starting key frame and the target key frame respectively.

[0106] The starting key frame is the key frame to be assigned the push mode.

[0107] For example, the kth subtask of transporting object m The starting pose is expressed as The ending pose is expressed as Correspondingly, the starting key frame is The target keyframe is In the initial stage, initialize the kth subtask of the object m to be transported Corresponding motion planning It can be expressed as Indicates empty, that is, from the starting key frame To target keyframe The push mode is empty.

[0108] S502: Determine whether there is a key frame to be assigned a push mode.

[0109] Determine whether there is a key frame for the propulsion mode to be assigned in the subtask. If so, it means that the motion plan for the robot executing the subtask has not been generated, and then execute step S503; if not, it means that the motion plan for the robot executing the subtask has been generated, and then execute step S507.

[0110] S503: Select a key frame whose estimated cost meets a preset condition.

[0111] When there is a key frame to be assigned a push mode, the estimated cost corresponding to the current motion plan is calculated for the subtask, and the key frame whose estimated cost meets a preset condition (eg, the lowest estimated cost) is selected.

[0112] In an example, the estimated cost corresponding to a subtask is calculated by the following expression:

[0113]

[0114] wherein θ * represents the estimated cost of motion planning corresponding to the subtask, l represents the identification of the key frame in the subtask, L θ represents the total number of key frames in the subtask, ξ l represents the pushing mode corresponding to the lth key frame, represents the arc segment between the lth key frame and the (l+1)th key frame, represents the direction feasibility estimation value of the object m to be transported under the pushing mode ξ l and the arc segment , ω1 represents the mode switching weight coefficient, and ξ l+1 represents the pushing mode corresponding to the (l+1)th key frame, represents the mode switching cost value of the pushing mode of the object m to be transported converted from ξ l to ξ l+1 , ω2 represents the navigation weight coefficient, s l represents the lth key frame, s l+1 represents the (l+1)th key frame, represents the navigation cost value of the object m to be transported switched from the lth key frame to the (l+1)th key frame. The direction feasibility is used to evaluate the feasibility of pushing the object in different directions, the mode switching cost is used to evaluate the difficulty of switching the object from one pushing mode to another pushing mode, and the navigation cost is used to evaluate the difficulty of moving the object along the path segment.

[0115] wherein and The specific forms of ω1 and ω2 can be set according to actual conditions. Exemplarily,

[0116] The mode switching weight coefficient ω1 and the navigation weight coefficient ω2 can be a value greater than 0, and can be set according to actual conditions, for example, ω1=1.0 indicates that the mode switching cost and the feasibility estimation value are equally important, and ω2=0.2 indicates that the navigation cost is relatively low. In one example, the mode switching weight coefficient ω1 and the navigation weight coefficient ω2 can follow the following priority settings: feasibility estimation weight coefficient> mode switching weight coefficient> navigation weight coefficient.

[0117] When the optimal motion planning (with the lowest cost) corresponding to the subtask is calculated, the key frame contained in the motion planning is selected as the key frame satisfying the preset condition.

[0118] S504 : For the first key frame to be assigned a propulsion mode among the selected key frames, determine whether the arc segment between the key frame and the next adjacent key frame intersects with an obstacle.

[0119] In one example, a geometric collision detection method, such as the GJK (Gilbert-Johnson-Keerthi) algorithm package, is used to detect whether the arc between a keyframe and its next adjacent keyframe intersects with an obstacle. The GJK algorithm is an efficient collision detection algorithm.

[0120] For example, the arc segment between a key frame and its adjacent next key frame is represented as First, the spatial area occupied by the transported object m as it moves along this arc is calculated. Then, based on this spatial area and the positional information of obstacles in the multi-robot collaborative task scenario, a geometric intersection check is performed to determine whether the arc between a keyframe and its next adjacent keyframe intersects with an obstacle. Since the transported object can be any polygonal shape, the pose changes of the object at each position along the arc, including changes in position and orientation, need to be considered during the spatial area calculation process.

[0121] If it is determined that the arc segment between the key frame and the next key frame adjacent thereto intersects with the obstacle, step S505 is executed and the process returns to step S502 ; if it is determined that the arc segment between the key frame and the next key frame adjacent thereto does not intersect with the obstacle, step S506 is executed. Figure 5 In the illustrated embodiment, the entire process is executed until all arc segments eventually do not intersect with obstacles.

[0122] S505: Insert an intermediate key frame between the key frame and the next adjacent key frame.

[0123] In one example, when it is determined that the arc segment between the key frame and the next adjacent key frame intersects with an obstacle, the intermediate posture of the arc segment between the key frame and the next adjacent key frame is determined as the intermediate key frame for insertion. After the intermediate key frame is inserted, the arc segment between the key frame and the next adjacent key frame is divided into two arc segments that are shorter than the original arc segment.

[0124] S506: Allocate a corresponding push mode to the key frame.

[0125] When it is determined that the arc segment between the key frame and the next adjacent key frame does not intersect with the obstacle, a corresponding propulsion mode is assigned to the key frame, and a motion plan corresponding to the robot performing the subtask is obtained. The obtained motion plan includes the key frames arranged in time sequence and their corresponding propulsion modes.

[0126] S507, end.

[0127] In an embodiment of the present invention, the subtask path segments are further decomposed into key frame sequences and intermediate arc segments, and the motion planning generation problem is converted into a joint optimization problem of key frames and propulsion modes to reduce the planning time of motion planning. In addition, when calculating the estimated cost, the direction feasibility, mode switching cost and navigation cost are comprehensively considered to minimize the control cost of the generated motion plan.

[0128] In a possible implementation, the step S506 assigning a corresponding push mode to the key frame includes:

[0129] Step 1: query a preset action set, and determine a corresponding predicted pushing mode for the key frame based on the correspondence between the key frames and the pushing modes included in the preset action set;

[0130] Step 2: Conduct a feasibility analysis on the forecast promotion model to determine whether the forecast promotion model is feasible;

[0131] Step 3: If the predicted pushing mode is feasible, the predicted pushing mode is determined as the pushing mode corresponding to the key frame;

[0132] Step 4: If the predicted push pattern is not feasible, a corresponding push pattern is generated for the key frame using a sparse optimization method.

[0133] In an embodiment of the present invention, for an object of a sample shape, the path of the object from the sample initial posture to the sample target posture is sampled in advance to obtain multiple sample key frames, and the pushing mode corresponding to each sample key frame is labeled. The sample shape of the object, a set number of robots, the sample initial posture and the sample target posture are used as training samples, and the sample key frames and their corresponding pushing modes are used as training labels to train a neural network model to obtain a diffusion model. The trained diffusion model is used to generate key frames and their corresponding pushing modes. In an embodiment of the present invention, there is no specific limitation on the set number. Then, for at least one object, a batch of basic shapes are generated by random deformation. The basic shapes generated by the random deformation of the object, a random number of robots, and the random initial posture and target posture of the object are used as sample data. Each sample data is input into the trained diffusion model to generate key frames and their corresponding pushing modes in batches. The generated key frames and their corresponding pushing modes are used as a preset action set, and the preset action set also includes the correspondence between key frames and pushing modes.

[0134] Furthermore, in the above-mentioned process of generating motion plans for the robot performing each subtask, when it is determined in step S506 that the arc segment between the key frame and the next adjacent key frame does not intersect with the obstacle, the preset action set is first queried to determine the corresponding predicted propulsion mode for the key frame from the preset action set based on the correspondence between the key frames and propulsion modes contained in the preset action set.

[0135] In one example, if the keyframe matches a keyframe in the preset action set (pose overlaps), the pushing mode corresponding to the keyframe in the preset action set that matches the keyframe is determined as the predicted pushing mode corresponding to the keyframe. If the keyframe does not match a keyframe in the preset action set (pose does not overlap), the pushing mode corresponding to the keyframe in the preset action set that is closest to the keyframe is determined as the predicted pushing mode corresponding to the keyframe.

[0136] Directly querying the preset action set and determining the corresponding predicted pushing mode for the key frame from the preset action set can significantly reduce the time of generating motion planning.

[0137] After determining the predicted push mode for the key frame, a feasibility analysis is performed on the predicted push mode corresponding to the key frame. For example, the feasibility value of the predicted push mode corresponding to the key frame can be calculated using the following expression:

[0138]

[0139] in, Indicates that the object to be transported m is in the pushing mode ξ l , arc segment is The feasibility estimate of the direction under the condition, D represents a set of base velocity directions used to cover the generalized velocity space of the object motion, ω d represents the weight coefficient along direction d, Indicates that the object to be transported m is in coordinate system B and the pushing mode is ξ l The combined generalized thrust, Indicates that the object to be transported m is in coordinate system B and the pushing mode is ξ l The combined generalized thrust set of Indicates that the object m to be transported is along the arc segment under the coordinate system B Generalized friction of motion, η m represents the coefficient of generalized friction, and || ||1 represents the L1 norm.

[0140] For evaluating the push mode l Under the condition of ξ, can the robot generate enough force to overcome friction and achieve the desired motion? lThe general thrust that the robot can provide Along the arc segment with the object m to be transported Generalized friction that needs to be overcome for motion The result of minimizing the difference between .

[0141] when When the value is large, it means that even after optimization, there is still a large gap between the thrust provided by the robot and the required friction force, indicating that this pushing mode is difficult to effectively push the object along the desired arc segment and is therefore not feasible. On the contrary, when A smaller value indicates that the robot's thrust can effectively balance or overcome friction, and the push mode is feasible. Therefore, if the calculated feasibility value is greater than the preset threshold, the predicted push mode corresponding to the keyframe is determined to be infeasible. Otherwise, the predicted push mode corresponding to the keyframe is determined to be feasible. The preset threshold can be set according to actual needs and is not specifically limited in this embodiment of the present invention.

[0142] When the predicted push mode corresponding to the key frame is feasible, the predicted push mode is directly determined as the push mode corresponding to the key frame; when the predicted push mode corresponding to the key frame is not feasible, the corresponding push mode is generated for the key frame through a sparse optimization method.

[0143] For example, the following expression is used as the objective function of the sparse optimization method to generate the corresponding pushing mode for the key frame:

[0144]

[0145] The constraints are:

[0146] Among them, F m,D Indicates that the object to be transported m is in the virtual pushing mode The thrust matrix applied on all possible contact points i is of dimension |I m |×2D,I m represents the set of all optional contact points of the object m to be transported, D represents a set of base velocity directions used to cover the generalized velocity space of the object's motion, Represents the thrust vector of the i-th contact point of the object to be transported m, specifically the i-th row of the thrust matrix, |||| ∞ represents the infinite norm, ||||1 represents the L1 norm, represents the generalized thrust of the transported object m along direction d in coordinate system B, represents the generalized friction force of the transported object m along direction d in coordinate system B, η m represents the coefficient of generalized friction, and J represents the Jacobian matrix used to convert the thrust of the contact point into Mapping to generalized thrust represents the object m to be transported in the virtual pushing mode the pushing force of the lower contact point along direction d, represents the object m to be transported in the virtual pushing mode the set of lower allowed pushing forces.

[0147] Sparse optimization refers to the optimization method that pursues most elements in the solution vector to be zero or close to zero in the optimization process. In the pushing mode generation, the goal of sparse optimization is to find a pushing force distribution scheme so that only a few contact points need to exert pushing force, while most contact points have zero pushing force. This is achieved by the L1 norm term in the objective function , because the L1 norm naturally has the property of promoting sparsity, which will punish the existence of non-zero elements, thus encouraging the optimization algorithm to produce more zero-element solutions.

[0148] By solving the above sparse optimization objective function, the optimal pushing force solution is obtained. The rows of are sorted by sparsity, and the first N rows are taken as the expected pushing mode, which contains contact point and pushing force information. Specifically, the contact points and their corresponding pushing force information contained in these N rows constitute a candidate pushing mode in the candidate pushing mode set, from which a set of candidate pushing modes is generated. The actual executable pushing mode ξ l is selected from the candidate pushing mode set corresponding to the lowest estimated cost of the above subtask, i.e. the pushing mode corresponding to the key frame is obtained.

[0149] where the sparsity sorting is the process of sorting each row of the optimization result . Each row represents the pushing force vector of the i-th contact point, and the sparsity sorting is arranged according to the "sparsity degree" of each row vector. Specifically, according to the number of non-zero elements in each row of the pushing force vector and the magnitude of these elements, the sorting is performed. The contact points with pushing force concentrated in a few directions and large magnitude are considered to be "more sparse" and "more effective", and are arranged in front, on the contrary, the contact points with pushing force dispersed in multiple directions or small magnitude are considered to contribute less, and are arranged in the back.

[0150] In the multi-robot cooperative pushing scenario, it is not necessary for all possible contact points to exert pushing force. Through sparse optimization, it can be identified which contact points are truly critical and which are redundant. Then the first N most important contact points are selected through sparse sorting, which not only guarantees the pushing effect, but also avoids resource waste, at the same time makes the pushing mode more concise and executable, especially suitable for practical application scenarios with limited number of robots.

[0151] In an example, for each key frame, the pushing speed and thrust of the robot on the arc segment between the key frame and the next adjacent key frame may be calculated, that is, the pushing speed and thrust in the pushing mode corresponding to the key frame.

[0152] For example, the optimal thrust can be obtained by calculating the first term in the objective function expression of the above sparse optimization method: For the pushing speed of the robot, we can use the coupled dynamics under the quasi-static assumption, assuming that the robot does not slide when in contact with the object (non-slip constraint), and assuming that the speed of the contact point must be equal when the robot pushes the object (rigid body kinematics). Based on this, the speed of the robot n is determined by the object speed and the contact constraint. Specifically, through the expression v n =v m +θ m ×(c n -x m ), calculate the speed of robot n, where v n represents the linear velocity of robot n, v m represents the linear velocity of the object m to be transported, θ m represents the angular velocity of the object m to be transported, c n represents the contact point position between robot n and object m to be transported, x m Indicates the location of the object m to be transported.

[0153] In this embodiment of the present invention, a pre-set action set is generated, which can then be directly queried to determine the corresponding predicted push pattern for a keyframe from the pre-set action set. This significantly reduces the time required to generate a motion plan. If the predicted push pattern obtained from the query is not feasible, a sparse optimization method is used to generate a corresponding push pattern for the keyframe, increasing the feasibility and security of the generated motion plan.

[0154] In one example, the preset action set can be maintained during the execution of the task, and during the processing of the actual multi-robot collaborative task, the preset action set can be dynamically modified through the pushing mode corresponding to the key frame, so that the preset action set can adapt to the multi-robot collaborative task processing in more scenarios.

[0155] In a possible implementation, the above motion plan includes a sequence of key frames and driving modes corresponding to the subtasks, that is, the key frames and their corresponding driving modes arranged in time sequence; Figure 6 As shown, the above step S105 sends the motion plan to the corresponding robot, so that the robot performs the transportation task of the corresponding subtask based on the motion plan, including:

[0156] S601: For each subtask, the motion plan corresponding to the subtask is sent to the corresponding robot.

[0157] S602: Determine a reference frame according to arc segment trajectories corresponding to the current frame and the next frame of the current frame in the subtask.

[0158] In the process of multi-robot collaborative execution of tasks, local reference trajectories can be generated in real time and task execution can be controlled. For example, in the control of each subtask for the object to be transported m, a link between the current frame s in the subtask is generated. m (t) The next frame from the current frame The arc segment trajectory is selected to satisfy ‖s r -s m The first pose where (t)‖>δ is taken as the reference frame Among them, s r is the position in the arc trajectory corresponding to the current frame and the next frame of the current frame, δ is a setting parameter, which is set according to the actual situation.

[0159] S603, based on the pushing mode from the current frame to the next frame of the current frame, calculate the contact point and contact direction of the robot on the object to be transported at the reference frame, and send the contact point and contact direction to the robot so that the robot transports the object to be transported according to the contact point and contact direction.

[0160] For example, the pushing pattern from the current frame to the next frame of the current frame includes the contact point and the thrust, based on which the contact point c between the current robot n and the object to be transported can be known. n and direction ψ n , using control expressions Calculate the contact point of robot n on the object to be transported in the reference frame and contact direction Among them, v represents the linear velocity control parameter, θ represents the angular velocity control parameter, K v and K ψ Represent the positive gain parameters of linear velocity and angular velocity, v, θ, K v and K ψ Set it according to the actual situation.

[0161] S604, detect whether the object to be transported deviates from the motion plan corresponding to the subtask. If it deviates, use the current frame as the starting posture of the subtask, and return to the step of generating a motion plan for the robot executing the subtask based on the starting posture and ending posture of the subtask.

[0162] The actual trajectory of the object to be transported is compared with the motion plan corresponding to the subtask. If the distance between the actual trajectory and the key frame in the motion plan is greater than a set value, it is determined that the object to be transported has deviated from the motion plan corresponding to the subtask. At this time, the current frame is used as the starting position of the subtask, and the process returns to the above step S104 to generate a motion plan for each subtask based on the starting and ending positions of the subtask for the robot performing the subtask. This can adapt to various uncertainties during task execution and thus ensure the continuity and stability of the task. If the distance between the actual trajectory and the key frame in the motion plan is not greater than a set value, it is determined that the object to be transported has not deviated from the motion plan corresponding to the subtask, and the robot continues to transport the object to be transported.

[0163] In one example, when a robot detects that it has transported an object to the end of an arc segment corresponding to the current frame and the frame immediately following the current one, the robot is controlled to switch from a push mode (from the current frame to the frame immediately following the current one) to a push mode (from the frame immediately following the current one) to the frame immediately following the current one. New contact points are assigned to the robots to prevent collisions. During the push mode switch, the robot's switching path is controlled to be no longer than half the circumference of the object to be transported, minimizing the robot's travel distance.

[0164] S605 , when it is detected that the subtask is completed, the status of the subtask is marked as task completed, and the status of the robot that completes the subtask is marked as idle.

[0165] In one example, when it is detected that the number of idle robots exceeds a certain value (such as more than half, etc.) or the time for the robot to perform the task exceeds the planned time (pre-set), the process returns to execute the above step S103 and assigns a corresponding robot to each subtask based on the status information of multiple robots and the partial order relationship between the subtasks, so as to reallocate tasks and increase the robustness of task execution.

[0166] In the embodiment of the present invention, during the execution of a task, arc segment trajectories corresponding to the current frame and the next frame of the current frame in the subtask are generated in real time, a reference frame is determined, and the contact point and contact direction of the robot on the object to be transported at the reference frame are calculated to control the robot to process the task in real time. When it is detected that the object to be transported deviates from the corresponding motion plan, the motion plan is regenerated to adapt to various uncertainties in the task execution process, thereby ensuring the continuity and stability of the task.

[0167] The method for processing multi-robot collaborative tasks in the embodiment of the present invention can be extended to various scenarios such as movable obstacle scenarios, heterogeneous robot groups, 6D pushing tasks and plane assembly, greatly expanding the application scope of collaborative pushing technology.

[0168] For example, Figure 7a-7c , assuming that four small ground robots (numbered as robot A, robot B, robot C, and robot D) are used to perform a collaborative pushing task in a controlled experimental environment. In this experimental environment, an L-shaped object (object 1) and a T-shaped object (object 2) are swapped on both sides of a narrow channel (see Figure 7a The traditional single-robot pushing method is inefficient and cannot meet the dynamic task requirements in complex environments.

[0169] Using the multi-robot collaborative task processing method of the present invention, at the start of the task, the robots knew the basic layout of the experimental field, including the locations of obstacles and the initial position of the objects. An OptiTrack motion capture system provided global positioning information. Each robot measured 0.2m x 0.3m (meters), had a maximum thrust of 10N (Newtons), and was equipped with an NVIDIA Jetson Nano running the Robot Operating System (ROS) for communication and control. The objects were made of cardboard, with a ground friction coefficient of 0.5 and a side friction coefficient of 0.3.

[0170] First, use the MAPF algorithm to generate the time series collision-free paths corresponding to object 1 and object 2. Assume that the path of the T-shaped object is planned first. Due to limited space, the L-shaped path should first avoid the T-shaped object. The path planned by the MAPF algorithm is as follows Figure 7a-7c As shown. Although the resulting path does not conflict in time and space, two collision points 1 and 2 can be found in the spatial dimension only. At collision point 1, the T-shaped object arrives before the L-shaped object, so the L-shaped object's path at collision point 1 is segmented, while the T-shaped path is not. At collision point 2, the T-shaped object arrives after the L-shaped object, so the L-shaped object's path at collision point 2 is not segmented, while the T-shaped path is segmented. Therefore, the trajectory of the T-shaped object at collision point 2 is decomposed into two subtasks (2,1) and (2,2). Because the first segment of the L-shaped object is too long, a split point is added, so the L-shaped object's path is decomposed into three subtasks (1,1), (1,2), and (1,3). Subtasks (1,1) and (2,1) have no predecessor tasks and can be assigned to execute simultaneously. Subtask (1,2) depends on the completion of (2,1), and subtask (2,2) depends on the completion of (1,1). The task allocation algorithm divides the four robots into two subgroups within 0.2 seconds. Subgroup 1 consists of robots A and B responsible for pushing the T-shaped object; subgroup 2 consists of robots C and D responsible for pushing the L-shaped object.

[0171] For each subtask path segment, if we want to search for the continuous pose strategy of the subtask, the entire search space is very large and the efficiency is very low. Therefore, we sample several key frames in the path segment to improve the search success rate and efficiency. Figure 7a-7c The hybrid optimization process of subtasks (1, 3) (i.e., the process of generating motion plans) is demonstrated in [1, 3]. First, a keyframe is sampled. Under this keyframe, the arc path from the split point to the keyframe and the arc path from the keyframe to the target position are guaranteed not to collide with obstacles. Otherwise, the keyframe needs to be resampled and the number of keyframes needs to be increased. Figure 7a-7c Subtasks (1, 3) in the example have different path parameters before and after the keyframe, so the forces applied should also be different. Two pushing modes are created before and after the keyframe, manifested by different robot contact points and applied forces. By pre-training a diffusion model to obtain an action set X (i.e., a preset action set), querying this preset action set X can accelerate keyframe and pushing mode generation. The processing of other subtasks is similar to that of (1, 3), with the keyframe count and processing time being the only issue. A dynamically updated primitive action library (i.e., action set X) can also be maintained, storing verified pushing primitives (i.e., pushing modes). For a given object type and robot subgroup, the primitive action library is first queried. If a corresponding pushing primitive does not exist in the action library, the diffusion model is invoked to batch generate candidate pushing strategies. After feasibility verification, these strategies are stored in the primitive action library. GPU parallel computing improves batch generation efficiency, achieving a positive feedback mechanism where "learning promotes planning, and planning enriches learning."

[0172] During execution, a local reference trajectory is generated in real time and the deviation of the object from the expected path is monitored. When the deviation exceeds a threshold or the robot cannot maintain the expected contact point for more than a predetermined time, a hybrid strategy replanning is triggered (i.e., returning to the execution step for each subtask, based on the starting and ending poses of the subtask, to generate a motion plan for the robot executing the subtask). A safe mode switching strategy is used to ensure that there are no collisions between robots. After each subtask is completed, resources are automatically released and the task status is marked. When the number of completed tasks changes significantly or the execution time exceeds a threshold, the tasks are reallocated. At the same time, the verified action set X can be continuously updated to record the successfully executed push modes to adapt to environmental changes and improve long-term execution efficiency.

[0173] The method for processing multi-robot collaborative tasks in the embodiment of the present invention can be generalized to a variety of application scenarios, such as mobile obstacles: treating them as pushable objects without pre-defining the target position; heterogeneous robots: considering the ability differences of different robots in task allocation and motion planning generation; 6D pushing tasks: replacing the arc transition in 2D with the spiral transition in 3D; plane assembly: first determine the target position, and then apply the method for processing multi-robot collaborative tasks in the embodiment of the present invention to complete the assembly task. At the same time, a verified action set X can be maintained, which is continuously updated during the execution process to record the successfully executed pushing modes and the corresponding state transitions. According to the execution results, the weight parameters in the cost estimation function are adjusted, and the frequently failed modes are marked to reduce their priority in future planning.

[0174] Corresponding to the above method embodiments, the embodiments of the present invention also provide corresponding system embodiments.

[0175] like Figure 8 As shown, the multi-robot collaborative task processing system provided by the embodiment of the present invention includes:

[0176] A path generation module 801 is configured to generate a time sequence path for each object to be transported from the initial position to the target position based on the initial positions and target positions of the multiple objects to be transported; the time sequence path for each object to be transported is collision-free in the spatiotemporal dimension;

[0177] A path partitioning module 802 is configured to partition the sequential path of each object to be transported into multiple path segments and determine a partial order relationship between the path segments of each object to be transported; wherein each path segment represents a subtask, and the partial order relationship is used to represent the execution order of the subtasks;

[0178] A task assignment module 803 is configured to assign a corresponding robot to each subtask based on the status information of the multiple robots and the partial order relationship between the subtasks;

[0179] A plan generation module 804 is configured to generate a motion plan for the robot performing each subtask based on the starting and ending poses of the subtask;

[0180] The task execution module 805 is used to send the motion plan to the corresponding robot, so that the robot executes the transportation task of the corresponding subtask based on the motion plan.

[0181] Optionally, the above-mentioned path division module 802 is specifically used to: determine whether there is an intersection between the time-series paths of different objects to be transported; based on the intersection, determine the segmentation points contained in the time-series paths of the objects to be transported where the intersection is located; based on the segmentation points, divide the time-series paths of each object to be transported into multiple path segments; based on the time-series relationship between each path segment, determine the partial order relationship between each path segment.

[0182] Optionally, the above-mentioned timing path contains the postures of the objects to be transported at different time stamps arranged in time sequence; the above-mentioned path division module 802 is specifically used to: for each timing path, compare the postures contained in the timing path with the postures contained in other timing paths one by one, and determine whether there is an intersection between the timing path and the other timing paths based on the comparison results; for each intersection of the timing path with other timing paths, determine whether the first time stamp corresponding to the intersection in the timing path is later than the second time stamp corresponding to the intersection in other timing paths; if so, determine the intersection as the segmentation point of the timing path; divide the timing path into path segments based on the determined segmentation point; and determine the partial order relationship between the path segments based on the timing relationship between the path segments of each object to be transported.

[0183] Optionally, the above-mentioned task assignment module 803 is specifically used to: initialize and generate a task assignment search tree; based on the partial order relationship between each subtask and the status of each subtask, select a subtask to be assigned as the root node of the task assignment search tree, and assign a robot in an idle state to the root node; construct a task assignment search tree based on the selection extension strategy; determine the target task assignment scheme from the constructed task assignment search tree; wherein the target task assignment scheme includes the identification of the subtask and the robot assigned to the subtask.

[0184] Optionally, the above-mentioned planning generation module 804 is specifically used to: for each subtask, use the starting posture and ending posture of the subtask as the starting keyframe and target keyframe respectively; the starting keyframe is the keyframe of the propulsion mode to be assigned; determine whether there is a keyframe of the propulsion mode to be assigned; if there is a keyframe of the propulsion mode to be assigned, select a keyframe whose estimated cost meets the preset conditions; for the first keyframe of the selected keyframes to be assigned a propulsion mode, determine whether the arc segment between the keyframe and the next adjacent keyframe intersects with the obstacle; if so, insert an intermediate keyframe between the keyframe and the next adjacent keyframe, and return to the step of determining whether there is a keyframe of the propulsion mode to be assigned; if not, assign the corresponding propulsion mode to the keyframe.

[0185] Optionally, the above-mentioned assigning a corresponding push mode to the key frame includes: querying a preset action set, and determining a corresponding predicted push mode for the key frame based on the correspondence between the key frames and the push modes contained in the preset action set; performing a feasibility analysis on the predicted push mode to determine whether the predicted push mode is feasible; if the predicted push mode is feasible, determining the predicted push mode as the push mode corresponding to the key frame; if the predicted push mode is not feasible, generating a corresponding push mode for the key frame through a sparse optimization method.

[0186] Optionally, the above-mentioned motion plan includes a key frame and a pushing pattern sequence corresponding to the subtask; the above-mentioned task execution module 805 is specifically used to: for each subtask, send the motion plan corresponding to the subtask to the corresponding robot; determine the reference frame according to the arc trajectory corresponding to the current frame and the next frame of the current frame in the subtask; calculate the contact point and contact direction of the robot on the object to be transported at the reference frame based on the pushing pattern from the current frame to the next frame of the current frame, and send the contact point and contact direction to the robot so that the robot transports the object to be transported according to the contact point and contact direction; detect whether the object to be transported deviates from the motion plan corresponding to the subtask; if deviated, take the current frame as the starting posture of the subtask, and return to execute the step of generating a motion plan for the robot executing the subtask based on the starting posture and ending posture of the subtask for each subtask; when it is detected that the subtask is completed, mark the status of the subtask as the task completed, and mark the status of the robot that completes the subtask as the idle state.

[0187] Optionally, the path generation module 801 is specifically configured to generate a temporal path for each object to be transported from its initial position to its target position using a multi-agent path planning algorithm based on the initial positions and target positions of multiple objects to be transported.

[0188] The embodiment of the present invention further provides an electronic device, such as Figure 9 As shown, it includes a processor 901 , a communication interface 902 , a memory 903 and a communication bus 904 , wherein the processor 901 , the communication interface 902 and the memory 903 communicate with each other via the communication bus 904 .

[0189] Memory 903, used for storing computer programs;

[0190] The processor 901 is configured to implement the steps of any of the above method embodiments when executing the program stored in the memory 903 to achieve the same technical effect.

[0191] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0192] The communication interface is used for communication between the above electronic device and other devices.

[0193] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0194] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0195] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above method embodiments to achieve the same technical effect.

[0196] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer executes the steps of any method embodiment in the above embodiments to achieve the same technical effect.

[0197] In the embodiments described above, all or some of the steps can be implemented by hardware, software, firmware or any combination thereof. When implemented in software, all or some of the steps can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, all or some of the steps generate the processes or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0198] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0199] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system / electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0200] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for processing multi-robot collaborative tasks, characterized in that: The method comprises: Based on the initial positions and target positions of multiple objects to be transported, a time sequence path is generated for each object to be transported from the initial position to the target position; the time sequence path of each object to be transported is collision-free in the time and space dimensions; Divide the sequential path of each object to be transported into multiple path segments, and determine a partial order relationship between the path segments of each object to be transported; wherein each path segment represents a subtask, and the partial order relationship is used to represent the execution order of the subtasks; Based on the status information of multiple robots and the partial order relationship between subtasks, a corresponding robot is assigned to each subtask; For each subtask, a motion plan is generated for the robot performing the subtask based on the starting and ending poses of the subtask. The motion plan is sent to a corresponding robot so that the robot performs the transportation task of the corresponding subtask based on the motion plan.

2. The method according to claim 1, characterized in that The step of dividing the time sequence path of each object to be transported into a plurality of path segments and determining the partial order relationship between the path segments of each object to be transported includes: Determine whether there is an intersection between the sequential paths of different objects to be transported; Based on the intersection, determining a segmentation point included in the temporal path of the object to be transported where the intersection is located; Based on the segmentation points, the temporal path of each object to be transported is divided into a plurality of path segments; Based on the temporal relationship between the path segments, a partial order relationship between the path segments is determined.

3. The method according to claim 1, characterized in that The temporal path includes the positions of the objects to be transported at different time stamps arranged in a temporal order; the temporal path of each object to be transported is divided into a plurality of path segments, and a partial order relationship between the path segments of each object to be transported is determined, including: For each timing path, compare the postures contained in the timing path with the postures contained in other timing paths one by one, and determine whether there is an intersection between the timing path and other timing paths based on the comparison results; For each intersection point in the timing path with other timing paths, determining whether a first timestamp corresponding to the intersection point in the timing path is later than a second timestamp corresponding to the intersection point in the other timing paths; If yes, the intersection point is determined as the split point of the timing path; Dividing the timing path into path segments based on the determined segmentation points; Based on the temporal relationship between the path segments of the objects to be transported, a partial order relationship between the path segments is determined.

4. The method according to claim 1, wherein The method of allocating a corresponding robot to each subtask based on the status information of the multiple robots and the partial order relationship between the subtasks includes: Initialize and generate the task allocation search tree; Based on the partial order relationship between the subtasks and the status of each subtask, a subtask to be assigned is selected as the root node of the task assignment search tree, and a robot in an idle state is assigned to the root node; Constructing the task allocation search tree based on the selected expansion strategy; A target task allocation scheme is determined from the constructed task allocation search tree; wherein the target task allocation scheme includes subtasks and identifiers of robots assigned to the subtasks.

5. The method according to claim 1, wherein For each subtask, generating a motion plan for the robot performing the subtask based on the starting pose and ending pose of the subtask includes: For each subtask, the starting posture and ending posture of the subtask are used as the starting keyframe and the target keyframe respectively; the starting keyframe is the keyframe to be assigned the pushing mode; Determine whether there is a keyframe to which a push mode is to be assigned; If there are key frames to be assigned a push mode, a key frame whose estimated cost satisfies a preset condition is selected; For a first key frame to be assigned a propulsion mode among the selected key frames, determining whether an arc segment between the key frame and a next adjacent key frame intersects with an obstacle; If they intersect, inserting an intermediate keyframe between the keyframe and the next keyframe adjacent thereto, and returning to the step of determining whether there is a keyframe to be assigned a push mode; If they do not intersect, the corresponding push mode is assigned to the keyframe.

6. The method according to claim 5, characterized in that The assigning of a corresponding push mode to the key frame includes: querying a preset action set, and determining a corresponding predicted pushing mode for the key frame based on a correspondence between the key frames and the pushing modes included in the preset action set; Conducting a feasibility analysis on the predicted promotion model to determine whether the predicted promotion model is feasible; If the predicted pushing mode is feasible, determining the predicted pushing mode as the pushing mode corresponding to the key frame; If the predicted push pattern is not feasible, a corresponding push pattern is generated for the key frame through a sparse optimization method.

7. The method according to claim 5, characterized in that The motion plan includes key frames and a pushing pattern sequence corresponding to the subtask; and sending the motion plan to the corresponding robot so that the robot performs the transportation task of the corresponding subtask based on the motion plan, including: For each subtask, the motion plan corresponding to the subtask is sent to the corresponding robot; Determine the reference frame based on the arc trajectory corresponding to the current frame and the next frame of the current frame in the subtask; calculating, based on the pushing pattern from the current frame to the next frame of the current frame, a contact point and a contact direction of the robot on the object to be transported at the reference frame, and sending the contact point and the contact direction to the robot so that the robot transports the object to be transported according to the contact point and the contact direction; Detecting whether the object to be transported deviates from the motion plan corresponding to the subtask; If there is a deviation, the current frame is used as the starting pose of the subtask, and the process returns to execute the step of generating a motion plan for the robot performing the subtask based on the starting pose and ending pose of the subtask; When it is detected that the subtask is completed, the state of the subtask is marked as task completed, and the state of the robot that completes the subtask is marked as idle.

8. The method according to any one of claims 1 to 7, characterized in that The method of generating a time sequence path from the initial position to the target position of each object to be transported based on the initial positions and target positions of the plurality of objects to be transported comprises: Based on the initial and target positions of multiple objects to be transported, a multi-agent path planning algorithm is used to generate a temporal path for each object to be transported from its initial position to its target position.

9. A multi-robot collaborative task processing system, characterized in that: The system comprises: A path generation module is used to generate a time sequence path for each object to be transported from the initial position to the target position based on the initial position and the target position of the multiple objects to be transported; the time sequence path of each object to be transported is collision-free in the time and space dimensions; A path partitioning module is used to divide the sequential path of each object to be transported into multiple path segments and determine a partial order relationship between the path segments of each object to be transported; wherein each path segment represents a subtask, and the partial order relationship is used to represent the execution order of the subtasks; A task assignment module is used to assign a corresponding robot to each subtask based on the status information of multiple robots and the partial order relationship between the subtasks; A plan generation module is used to generate a motion plan for the robot performing each subtask based on the starting and ending poses of the subtask; The task execution module is used to send the motion plan to the corresponding robot so that the robot executes the transportation task of the corresponding subtask based on the motion plan.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

Images