Multi-robot path planning method, control method, storage medium and electronic device

Abstract

The present application provides a multi-robot path planning method, control method, storage medium and electronic device. The method includes: obtaining multiple tasks to be executed, and determining at least the execution order of multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of multiple tasks; constructing a digital twin model of the assembly unit; identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located, and determining the first collision body type according to the distance between the dynamic obstacle and each robot arm, and determining the second collision body type according to the triangular face information of the static obstacle and its distance from each robot arm; establishing a dynamic obstacle model and a static obstacle model; determining the feasible motion path of each robot arm to perform multiple tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model and the static obstacle model; determining the optimal motion path of each robot arm from multiple feasible motion paths.

Description

Technical Field

[0001] The present application relates to the field of robot path planning, and more specifically, to a path planning method for multiple robot arms, a control method for multiple robot arms, a computer-readable storage medium, and an electronic device. Background Art

[0002] Existing robot arm path planning solutions focus on the simulation and analysis of the motion path of a single robot arm, and are mainly used to determine the joint motion and obstacle avoidance strategy of a single robot arm, thereby analyzing and optimizing a single performance indicator. However, in the actual process of the assembly unit, the realization of the task often requires the collaboration of multiple robot arm devices. The existing planning solutions cannot maximize the efficiency of each robot arm, resulting in low overall task completion efficiency. Summary of the invention

[0003] The main purpose of the present application is to provide a path planning method for multiple robotic arms, a control method for multiple robotic arms, a computer-readable storage medium and an electronic device, so as to at least solve the problem of low efficiency of tasks completed by multiple robotic arms in collaboration in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a path planning method for multiple robotic arms is provided, including: obtaining multiple tasks to be executed, and determining at least the execution order of the multiple tasks based on the safety indicators, urgency indicators, correlation indicators, data volume indicators and stability indicators of the multiple tasks, at least some of the tasks are tasks that are collaboratively executed by multiple robotic arms, the safety indicator is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the task, the urgency indicator is used to characterize the urgency of the task, the correlation indicator is used to characterize the degree of dependence between the task and any other task, the data volume indicator is used to characterize the amount of data required to execute the task, and the stability indicator is used to characterize the historical execution success rate of the task; constructing a digital twin model of an assembly unit, the assembly unit comprising multiple robotic arms; identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located. According to the method, a first collision body type is determined according to the distance between the dynamic obstacle and each of the robotic arms, and a second collision body type is determined according to the triangular patch information of the static obstacle and the distance between the static obstacle and each of the robotic arms, the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle; a dynamic obstacle model is established according to the first collision body type and the motion trajectory of the dynamic obstacle, and a static obstacle model is established according to the second collision body type; according to the execution order, the digital twin model, the dynamic obstacle model and the static obstacle model, a feasible motion path for each of the robotic arms to perform the multiple tasks in the execution order is determined; when there are multiple feasible motion paths, the optimal motion path for each of the robotic arms is determined from the multiple feasible motion paths with the shortest execution time of the multiple tasks as the optimal goal.

[0005] Optionally, based on the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least the execution order of the multiple tasks is determined, including: when the correlation index is greater than a predetermined threshold, predetermined processing is performed on the two tasks corresponding to the correlation index greater than the predetermined threshold, so that the correlation index after the predetermined processing is less than or equal to the predetermined threshold, and multiple new tasks are obtained, and the predetermined processing includes at least one of the following: splitting the task into multiple subtasks, reorganizing multiple tasks, and reorganizing some of the split subtasks; when the correlation index is less than or equal to the predetermined threshold, weighted summing up the safety index, the urgency index, the correlation index, the data volume index and the stability index of each task is performed to obtain the priority index of the task; and sorting the multiple tasks in descending order of the priority index to obtain the execution order.

[0006] Optionally, the type of the first collision body is determined according to the distance between the dynamic obstacle and each of the robotic arms, including: when the distance between the dynamic obstacle and each of the robotic arms is greater than or equal to a first predetermined multiple of a safety distance of the robotic arms, determining that the type of the first collision body is a capsule body; when the distance between the dynamic obstacle and each of the robotic arms is less than the first predetermined multiple of the safety distance of the robotic arms and greater than or equal to a second predetermined multiple of the safety distance of the robotic arms, determining that the type of the first collision body is a triangular mesh body, wherein both the first predetermined multiple and the second predetermined multiple are greater than or equal to 1.

[0007] Optionally, the triangular facet information includes the number of triangular facets constituting the static obstacle and the direction of the normal vector of each triangular facet, and the second collision body type is determined according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, including: determining the outer surface triangular facets of the static obstacle from a plurality of the triangular facets according to the direction of the normal vector, and obtaining the number of the outer surface triangular facets; when the number of the outer surface triangular facets is less than a quantity threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to a first predetermined multiple of a safety distance of the robotic arm, determining the second collision body type to be a sphere; when the number of the outer surface triangular facets is greater than or equal to the quantity threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to the first predetermined multiple of a safety distance of the robotic arm. When the number of triangular facets on the outer surface is less than the first predetermined multiple of the robot arm safety distance, the type of the second collision body is determined to be a box body; when the number of triangular facets on the outer surface is less than the number threshold and the distance between the static obstacle and each of the robot arms is less than the first predetermined multiple of the robot arm safety distance, and is greater than or equal to the second predetermined multiple of the robot arm safety distance, the type of the second collision body is determined to be a concave polyhedron; when the number of triangular facets on the outer surface is greater than or equal to the number threshold and the distance between the static obstacle and each of the robot arms is less than the first predetermined multiple of the robot arm safety distance, and is greater than or equal to the second predetermined multiple of the robot arm safety distance, the type of the second collision body is determined to be a grid body, wherein the first predetermined multiple and the second predetermined multiple are both greater than or equal to 1.

[0008] Optionally, according to the execution order, the digital twin model, the dynamic obstacle model and the static obstacle model, the feasible motion paths for each of the robotic arms to perform the multiple tasks in the execution order are determined, including: according to the execution order, a task list is assigned to each of the robotic arms in the digital twin model, and the starting position and the end position of each of the robotic arms are determined according to the task list; a kinematic model of each of the robotic arms is constructed; according to the starting position and the end position of the target robotic arm, the APF (Artificial Potential Field)-RRT (Rapidly-exploring Random Tree) algorithm is used to plan the feasible motion path of the target robotic arm with the goal of avoiding the dynamic obstacle model, the static obstacle model and the kinematic models of other robotic arms, so as to obtain the feasible motion path corresponding to each robotic arm, wherein the target robotic arm is one of the multiple robotic arms, and the other robotic arms are the robotic arms other than the target robotic arm.

[0009] Optionally, constructing a digital twin model of an assembly unit includes: acquiring image information and point cloud information of a physical entity in the assembly unit to obtain first information, wherein the physical entity includes a plurality of the robotic arms, materials to be grasped, and a material transportation track; based on the first information, using digital modeling technology to construct an initial twin model corresponding to the physical entity; periodically acquiring the image information and the point cloud information of the physical entity to obtain second information; comparing the first information with the second information, filtering out information in the second information that is identical to the first information, and the remaining second information is distinguishing information; based on the distinguishing information, adding a model component corresponding to the distinguishing information to the initial twin model to obtain the digital twin model.

[0010] Optionally, based on the first information, an initial twin model corresponding to the physical entity is constructed using digital modeling technology, including: determining the size information, position information and assembly relationship of the physical entity based on the first information; based on the size information, the position information and the assembly relationship, using the digital modeling technology to construct a 1:1 three-dimensional model corresponding to the physical entity in a virtual environment; adding attribute information of the physical entity to the three-dimensional model, the attribute information including material attributes, gripping force attributes and contact force attributes; obtaining the motion range of multiple joints in each of the robotic arms and the parent-child relationship between the multiple joints, the parent-child relationship representing the follow-up relationship between two connected joints; based on the motion range and the parent-child relationship, adding motion range restrictions and parent-child relationship restrictions to the three-dimensional model of the robotic arm to which the attribute information is added, to obtain the initial twin model.

[0011] Optionally, identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located includes: obtaining historical pixel features of the environment where the assembly unit is located, the historical pixel features being obtained by extracting pixel features from multiple historical environment images; obtaining an initial Gaussian mixture model, and training the initial Gaussian mixture model using an iterative learning algorithm based on the historical pixel features to obtain a Gaussian mixture model; obtaining multiple consecutive frames of images of the environment, and extracting pixel features of the multiple consecutive frames of images; and analyzing the pixel features using the Gaussian mixture model to determine the dynamic obstacles and the static obstacles in the environment.

[0012] According to another aspect of the present application, a method for controlling multiple robotic arms is provided, comprising: determining an optimal motion path of each of the robotic arms according to any of the methods described; and controlling the movement of each of the robotic arms according to the multiple optimal motion paths.

[0013] Optionally, the movement of each of the robotic arms is controlled according to the multiple optimal motion paths, including: converting the optimal motion path of each of the robotic arms to obtain an instruction code for the controller of the assembly unit; controlling the controller to execute the instruction code so that the controller controls the movement of each of the robotic arms; when the movement of any of the robotic arms does not meet predetermined requirements, adjusting the target parameters in the process of determining the optimal motion path, and redetermining the optimal motion path according to the adjusted target parameters, the target parameters including at least part of the following: the relative position relationship between any two of the robotic arms, the relative position relationship between the robotic arms and a static obstacle model, and the relative position relationship between the robotic arms and a dynamic obstacle model.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.

[0016] By applying the technical solution of the present application, the order of multiple tasks is determined by their safety indicators, urgency indicators, correlation indicators, data volume indicators and stability indicators, thereby realizing the priority sorting of multiple tasks; by identifying dynamic and static obstacles in the environment where the assembly unit is located and determining the types of collision bodies, the automatic construction of dynamic obstacle models and static obstacle models in the environment is realized; through digital twin technology, the task priority order and the obstacle model in the environment are integrated to plan the paths for multiple robotic arms to execute tasks, thereby obtaining the optimal motion path with the shortest execution time for multiple tasks, realizing collaborative path planning for multiple robotic arms, and ensuring that the collaborative work efficiency of multiple robotic arms is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for executing a path planning method for multiple robotic arms provided in an embodiment of the present application is shown;

[0019] Figure 2A schematic flow chart of a multi-robot arm path planning method provided according to an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of a flow chart for determining the execution order of multiple tasks provided according to an embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of a process for determining the type of a first collision body provided according to an embodiment of the present application is shown;

[0022] Figure 5 A schematic diagram of a process for determining the type of a second collision body provided according to an embodiment of the present application is shown;

[0023] Figure 6 A schematic diagram of a process for identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located according to an embodiment of the present application is shown;

[0024] Figure 7 A schematic diagram of a process for generating a feasible motion path of a mechanical arm for performing a grasping task provided in accordance with an embodiment of the present application is shown;

[0025] Figure 8 A schematic diagram of the overall process of path planning and control of a multi-manipulator system provided according to an embodiment of the present application is shown;

[0026] Fig. 9 A structural block diagram of a multi-robot arm path planning device provided according to an embodiment of the present application is shown.

[0027] The drawings include the following reference numerals:

[0028] 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, acquisition unit; 20, construction unit; 30, identification unit; 40, establishment unit; 50, first determination unit; 60, second determination unit. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] As introduced in the background technology, there is a technical problem in the prior art that the efficiency of tasks completed by multiple robotic arms in collaboration is low. In order to solve the above technical problem, the embodiments of the present application provide a multi-robotic arm path planning method, a multi-robotic arm control method, a computer-readable storage medium and an electronic device.

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a multi-robot path planning method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the path planning method of the multi-manipulator in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method described is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a multi-robot path planning method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 is a flow chart of a multi-robot path planning method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0038] Step S201, obtaining multiple tasks to be executed, and determining at least the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least part of the tasks are tasks performed collaboratively by multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks;

[0039] Specifically, according to the safety index, urgency index, correlation index, data volume index and stability index of each task, at least multiple tasks are prioritized to obtain the execution order. At least part of the tasks are tasks that are executed collaboratively by multiple robotic arms, which means that when there are m robotic arms, there is at least one task among the multiple tasks that requires n robotic arms to be executed collaboratively, m≥n. For multiple robotic arms, each of the robotic arms has limit requirements in all directions, and these limit requirements constitute the range of motion of the robotic arms. The correlation indicators include but are not limited to the degree of resource dependence between two tasks, the degree of dependence on the execution order, the number of interactions, and the amount of data interaction.

[0040] Step S202, constructing a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms;

[0041] Specifically, the digital twin model is a model obtained by mapping and simulating entities, equipment, systems, etc. in the physical world in a virtual space through digital means using digital twin technology.

[0042] Step S203, identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located, determining a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determining a second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle;

[0043] Specifically, the dynamic obstacle refers to an obstacle that causes the robot arm to move in the assembly unit; the static obstacle refers to an obstacle that causes the robot arm to not move in the assembly unit.

[0044] Step S204, establishing a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and establishing a static obstacle model according to the second collision body type;

[0045] Specifically, the dynamic obstacle model is a model obtained by mapping the dynamic obstacle into a virtual space, and the static obstacle model is a model obtained by mapping the static obstacle into a virtual space.

[0046] Step S205, determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model;

[0047] Specifically, the feasible motion path is a motion path that enables the plurality of robot arms to execute each task in the execution order. Each robot arm may have only one feasible motion path or may have multiple feasible motion paths.

[0048] Step S206, when there are multiple feasible motion paths, taking the shortest execution time of multiple tasks as the optimal goal, determine the optimal motion path of each of the robot arms from the multiple feasible motion paths.

[0049] Specifically, the execution time is the total time taken to execute multiple tasks in the execution order. When there are multiple feasible motion paths for the robot, the optimal motion path is the path among the multiple feasible motion paths that satisfies the goal of the shortest execution time. When there is only one feasible motion path for the robot, this one feasible motion path is the optimal motion path for the robot.

[0050] Through the described embodiment, firstly, multiple tasks to be performed by multiple robotic arms are obtained, and according to the safety index, urgency index, correlation index, data volume index and stability index of each task, at least the multiple tasks are sorted to obtain the execution order, wherein the safety index characterizes the degree of overlap of the motion ranges between multiple robotic arms during the execution of the task, the urgency index is used to characterize the urgency of the task, the correlation index characterizes the degree of dependence between two tasks, the data volume index characterizes the amount of data required to execute the task, and the stability index characterizes the historical execution success rate of the task; then, a digital twin model of an assembly unit including multiple robotic arms is constructed; and dynamic obstacles and static obstacles in the environment where the assembly unit is located are identified , according to the distance between the dynamic obstacle and each robotic arm, the collision body type of the dynamic obstacle is determined, and according to the triangular patch information of the static obstacle and the distance between the static obstacle and each robotic arm, the collision body type of the static obstacle is determined; then, a dynamic obstacle model and a static obstacle model are established according to the information of the collision body type and the motion trajectory; then, according to the obtained execution order, the digital twin model, the dynamic obstacle model and the static obstacle model, the feasible motion paths for each robotic arm to perform multiple tasks in the execution order are determined; finally, when there are multiple feasible motion paths for any robotic arm, the shortest execution time of multiple tasks is taken as the optimal goal, so as to determine the optimal motion path of the robotic arm from multiple feasible motion paths. The present application determines the order of multiple tasks through safety indicators, urgency indicators, correlation indicators, data volume indicators and stability indicators of multiple tasks, thereby realizing the priority sorting of multiple tasks; by identifying dynamic and static obstacles in the environment where the assembly unit is located and determining the types of collision bodies, it realizes the automatic construction of dynamic obstacle models and static obstacle models in the environment; through digital twin technology, the task priority order and the obstacle model in the environment are integrated to plan the paths for multiple robotic arms to execute tasks, thereby obtaining the optimal motion path with the shortest execution time for multiple tasks, realizing the collaborative path planning of multiple robotic arms, and ensuring the high efficiency of the collaborative work of multiple robotic arms.

[0051] It should be noted that executing the multiple tasks in the execution order means that in the execution order, the start execution time of a task A is earlier than the start execution time of the tasks arranged behind the task A. The end execution time of the task A may be later than the start execution time of the tasks arranged behind the task A, and the end execution time of the task A may also be earlier than or equal to the start execution time of the tasks arranged behind the task A, which is specifically determined according to the capabilities of the multiple robot arms and the task requirements.

[0052] Specifically, the assembly unit is an independent module or workstation in a production line or automation system that is responsible for completing specific assembly, connection, verification or processing tasks, including a variety of equipment and components, such as robotic arms, conveyor belts, feeding systems, fixtures, sensors, control systems (such as PLC, programmable logic controller) and other special equipment.

[0053] Those skilled in the art can flexibly select a suitable method to determine the motion trajectory of the dynamic obstacle according to actual needs, such as obtaining multiple consecutive frames of images including the dynamic obstacle, calculating the positions of the dynamic obstacle according to the multiple consecutive frames of images, and determining the duration of obtaining the multiple consecutive frames of images according to the interval between two consecutive frames of images; and establishing the motion trajectory equation according to each of the positions and the duration.

[0054] In an optional scheme, at least the execution order of the multiple tasks is determined based on the security indicators, urgency indicators, correlation indicators, data volume indicators and stability indicators of the multiple tasks, including: when the correlation indicator is greater than a predetermined threshold, predetermined processing is performed on the two tasks corresponding to the correlation indicators greater than the predetermined threshold, so that the correlation indicators after the predetermined processing are less than or equal to the predetermined threshold, and multiple new tasks are obtained, and the predetermined processing includes at least one of the following: splitting the task into multiple subtasks, reorganizing multiple tasks, and reorganizing some of the split subtasks; when the correlation index is less than or equal to the predetermined threshold, weighted summing up the security indicator, the urgency indicator, the correlation indicator, the data volume indicator and the stability indicator of each task is performed to obtain the priority indicator of the task; and sorting the multiple tasks in descending order of the priority indicators to obtain the execution order. In the present embodiment, when the correlation index of a task is greater than a predetermined threshold, it indicates that the task is highly dependent on other tasks. In order to reduce the difficulty of processing multiple complex tasks with a high degree of dependence, the present application first performs at least one of splitting, reorganizing, and splitting and then reorganizing on the tasks with a high degree of dependence, so that the correlation index of each task after processing is not greater than the predetermined threshold, which can further avoid conflicts between tasks. Then, the safety index, the urgency index, the correlation index, the data volume index and the stability index of the tasks that meet the correlation index requirements are weighted and summed to obtain the priority index of each task. Finally, multiple tasks are arranged in descending order of priority index to achieve priority sorting of multiple tasks. Subsequently, the robot resources can be reasonably allocated according to the priority order, ensuring that the overall allocation and processing efficiency of multiple tasks is high, thereby making the overall path planning efficiency of the robot high.

[0055] In the embodiment of the present application, each of the tasks includes multiple subtasks executed sequentially, and splitting the task into multiple subtasks is the process of splitting the task into multiple subtasks. Reorganizing multiple tasks is the process of sequentially reorganizing the subtasks included in the multiple tasks.

[0056] In a specific embodiment, the flowchart for determining the execution order of multiple tasks is as follows: Figure 3 As shown. During the execution of a task, there is overlap in the motion ranges of the two robotic arms executing the task, indicating that there is a risk of collision between the two robotic arms. In this case, the safety index of the task is low, and the higher the degree of overlap, the smaller the corresponding safety index. The urgency index may be a pre-set indicator characterizing the degree of urgency, and the higher the degree of urgency, the larger the corresponding urgency index. The higher the degree of dependence, the smaller the corresponding correlation index. The larger the amount of data, the larger the corresponding data volume index. The greater the historical execution success rate, the larger the corresponding stability index.

[0057] In order to further solve the technical problem of low efficiency of tasks completed by multiple robotic arms in collaboration, in some optional embodiments of the present application, the first collision body type is determined according to the distance between the dynamic obstacle and each of the robotic arms, including: Figure 4 As shown, when the distance between the dynamic obstacle and each of the robotic arms is greater than or equal to a first predetermined multiple of the robotic arm safety distance, the dynamic obstacle is determined to be a general obstacle, and the first collision body type is determined to be a capsule body; when the distance between the dynamic obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance, and greater than or equal to a second predetermined multiple of the robotic arm safety distance, the dynamic obstacle is determined to be a key obstacle, and the first collision body type is determined to be a triangular mesh body, wherein the first predetermined multiple and the second predetermined multiple are both greater than or equal to 1. The present application determines the complexity of the dynamic obstacle and its collision body type according to the distance between the dynamic obstacle and the robotic arm. When the distance between the dynamic obstacle type and the robotic arm is large, the dynamic obstacle is determined to be a capsule collision type. When the distance between the dynamic obstacle type and the robotic arm is small, the dynamic obstacle is determined to be a triangular mesh collision type. This realizes the automatic identification of the collision body type of the dynamic obstacle. Subsequently, a corresponding collision body type model can be established according to the identification result. There is no need to manually judge the obstacle type and establish an obstacle model, which improves the efficiency of establishing the obstacle model, so that the optimal motion path of the multiple robotic arms can be determined more quickly according to the model, so that the multiple robotic arms can operate according to the optimal motion path, thereby performing multiple tasks.

[0058] Among them, the robot arm safety distance, the first predetermined speed and the second predetermined multiple can be determined according to the actual movement accuracy of the robot arm. Since there are multiple robot arms, there are also multiple distances between the dynamic obstacle and each robot arm. As long as there is a distance greater than or equal to the first predetermined multiple of the robot arm safety distance, the first collision body type corresponding to the dynamic obstacle is a capsule body. In an exemplary embodiment, the first predetermined multiple is 1.5, and the second predetermined multiple is 1.

[0059] According to some further exemplary embodiments of the present application, the triangular face information includes the number of triangular facets constituting the static obstacle and the normal vector direction of each triangular facet, and the second collision body type is determined according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, including: Figure 5As shown, according to the direction of the normal vector, the outer surface triangular facets of the static obstacle are determined from the multiple triangular facets to obtain the number of the outer surface triangular facets; when the number of the outer surface triangular facets is less than the number threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to the first predetermined multiple of the robotic arm safety distance, the static obstacle is determined to be a simple static obstacle, and the second collision body type is determined to be a sphere; when the number of the outer surface triangular facets is greater than or equal to the number threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to the first predetermined multiple of the robotic arm safety distance, the static obstacle is determined to be a general static obstacle, and the second collision body type is determined to be a box; in the outer surface triangular facets, the number of the outer surface triangular facets is less than the number threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to the first predetermined multiple of the robotic arm safety distance, the static obstacle is determined to be a general static obstacle, and the second collision body type is determined to be a box. When the number of triangular facets on the outer surface is greater than or equal to the number threshold and the distance between the static obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance and is greater than or equal to the second predetermined multiple of the robotic arm safety distance, the static obstacle is determined to be a complex static obstacle, and the type of the second collision body is determined to be a concave polyhedron; when the number of triangular facets on the outer surface is greater than or equal to the number threshold and the distance between the static obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance and is greater than or equal to the second predetermined multiple of the robotic arm safety distance, the static obstacle is determined to be a key static obstacle, and the type of the second collision body is determined to be a grid body, wherein both the first predetermined multiple and the second predetermined multiple are greater than or equal to 1. The present application determines the complexity of the static obstacle and its collision body type according to the distance between the static obstacle and the robotic arm and the number of triangular facets on the outer surface, thereby realizing automatic identification of the collision body type of the static obstacle. Subsequently, a corresponding collision body type model can be established based on the identification result, without the need for manual judgment of the obstacle type and establishment of the obstacle model, thereby improving the efficiency of establishing the obstacle model, so that the optimal motion path of the multiple robotic arms can be determined more quickly according to the model, so that the multiple robotic arms can operate according to the optimal motion path, thereby performing multiple tasks.

[0060] The robot arm safety distance, the first predetermined speed, the second predetermined multiple and the quantity threshold can be determined according to the actual motion accuracy of the robot arm and the specific parameters of the static obstacle. In an exemplary embodiment, the first predetermined multiple is 1.5, the second predetermined multiple is 1, and the quantity threshold is 100,000 faces.

[0061] In addition, according to the execution order, the digital twin model, the dynamic obstacle model and the static obstacle model, the feasible motion paths for each of the robotic arms to perform the multiple tasks in the execution order are determined, including: according to the execution order, a task list is assigned to each of the robotic arms in the digital twin model, and the starting position and the end position of each of the robotic arms are determined according to the task list; a kinematic model of each of the robotic arms is constructed; according to the starting position and the end position of the target robotic arm, the APF-RRT algorithm is used to plan the feasible motion path of the target robotic arm with the goal of avoiding the dynamic obstacle model, the static obstacle model and the kinematic models of other robotic arms, so as to obtain the feasible motion path corresponding to each robotic arm, wherein the target robotic arm is one of the multiple robotic arms, and the other robotic arms are the robotic arms other than the target robotic arm. In this embodiment, in a twin environment, the APF-RRT algorithm is used to plan the feasible motion path of each robotic arm, so that the robotic arm can execute and complete the task list while avoiding obstacles, further realizing the collaborative work planning between multiple robotic arms. Since the APF-RRT algorithm combines the local obstacle avoidance capability of the APF algorithm and the global search capability of the RRT algorithm, it can not only effectively avoid obstacles in complex environments, but also quickly find the global optimal path, thereby obtaining a more accurate feasible path for multi-robotic arm planning.

[0062] Specifically, the task list is a list of subtasks and / or tasks that the robot needs to perform. In addition, the APF algorithm is a path planning technology based on physical concepts, which guides the robot movement by constructing a virtual potential field. In this potential field, the end point pose generates an attractive potential field, and obstacles generate a repulsive potential field; the robot moves according to the gradient direction of the current position in the comprehensive potential field, thereby achieving the purpose of both finding a path and avoiding obstacles. The RRT algorithm is a randomized path planning algorithm that randomly selects a spatial point and tries to connect it to the nearest tree node, gradually building a search tree until the tree touches the target area. The RRT algorithm can efficiently explore high-dimensional space and has a high probability of finding a path connecting the starting point and the end point; the APF-RRT algorithm combines APF with RRT, and can use APF to guide the growth direction of the RRT tree to improve planning efficiency and path quality. When generating each new node of the RRT tree, instead of randomly selecting a point in space, the position of the new node is determined based on the current tree structure and the guidance of the APF potential field, and obstacles are avoided more purposefully and tend to expand in the target direction. Combining the obstacle avoidance capability of APF and the efficient exploration characteristics of RRT, it can quickly find feasible paths in complex environments.

[0063] Optionally, constructing a digital twin model of an assembly unit includes: acquiring image information and point cloud information of a physical entity in the assembly unit to obtain first information, wherein the physical entity includes a plurality of the robotic arms, materials to be grasped, and a material transportation track; based on the first information, using digital modeling technology to construct an initial twin model corresponding to the physical entity; periodically acquiring the image information and the point cloud information of the physical entity to obtain second information; comparing the first information with the second information, filtering out information in the second information that is identical to the first information, and the remaining second information is distinguishing information; based on the distinguishing information, adding a model component corresponding to the distinguishing information to the initial twin model to obtain the digital twin model. In this embodiment, by scanning the physical entity, the first information such as image information and point cloud information is generated for the robot arm, material and track contained in the assembly unit, and the first information is used to construct a twin model. In order to ensure the accuracy of the twin model, the actual image information and point cloud information of the assembly unit are obtained for the second time to obtain the second information. According to the first information, the information in the second information that is repeated with the first information is filtered out, and the filtered information is used to improve the model components in the twin model, thereby realizing the effect of automatically filtering the information generated by the scan and only generating models for the ungenerated model components. This can not only ensure the accuracy of the digital twin model, but also reduce the amount of data processing during the twin model construction process.

[0064] Further, according to the first information, an initial twin model corresponding to the physical entity is constructed using digital modeling technology, including: according to the first information, determining the size information, position information and assembly relationship of the physical entity; according to the size information, the position information and the assembly relationship, using the digital modeling technology to construct a 1:1 three-dimensional model corresponding to the physical entity in a virtual environment; adding attribute information of the physical entity to the three-dimensional model, the attribute information including material attributes, gripping force attributes and contact force attributes; obtaining the range of motion of multiple joints in each of the mechanical arms and the parent-child relationship between the multiple joints, the parent-child relationship representing the follow-up relationship between the two connected joints; according to the range of motion and the parent-child relationship, adding range of motion restrictions and parent-child relationship restrictions to the three-dimensional model of the mechanical arm to which the attribute information is added, to obtain the initial twin model. This application performs a 1:1 restoration of the assembly unit of the physical world based on digital twin technology, builds a digital twin scene, can accurately map and simulate the actual structure and relative position relationship of the assembly unit of the physical world, and provide a more accurate model support for the subsequent path planning of the mechanical arm.

[0065] Specifically, in the created twin environment, each movable joint of each robot arm is assigned a specific number, and a local coordinate system is defined for each joint and end effector of the robot arm. The origin of the joint coordinate system is generally located at the center of the joint, and the direction of the axis depends on the type of joint, such as the Z axis of the rotary joint along the direction of the rotation axis. Set the range of motion limit for each joint, that is, the minimum and maximum rotation angles of the joint (if it is a mobile joint, it is a linear displacement limit). Connect the joints in the correct order to form a complete kinematic chain. This step includes defining the parent-child relationship between joints to ensure that when a joint moves, its downstream links and joints can follow correctly. Through the end effector of the robot arm, each robot arm in the twin environment is task-bound, and the improved APF-RRT algorithm is used to plan a feasible path from the starting position to the end position for each robot arm. By clarifying the workspace of the robot arm and the position and size of the obstacles and target grasping objects, a comprehensive potential field is established, and an RRT tree from the starting node to the target node is generated until a collision-free moving path from the starting position to the end position is obtained.

[0066] Exemplarily, a flowchart for identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located is as follows: Figure 6 As shown, by scanning the environment where the assembly unit is located, it is determined whether there is a motion path for the obstacle, thereby distinguishing and identifying dynamic and static obstacles.

[0067] In some other exemplary embodiments, identifying the dynamic obstacles and static obstacles in the environment where the assembly unit is located specifically includes: obtaining the historical pixel features of the environment where the assembly unit is located, the historical pixel features are obtained by extracting pixel features from multiple historical environment images; obtaining an initial Gaussian mixture model, and training the initial Gaussian mixture model using an iterative learning algorithm based on the historical pixel features to obtain a Gaussian mixture model; obtaining continuous multi-frame images of the environment, and extracting pixel features of the continuous multi-frame images; using the Gaussian mixture model to analyze the pixel features to determine the dynamic obstacles and static obstacles in the environment. By distinguishing the obstacles in the environment where the assembly unit is located using the Gaussian mixture model, complex image pixel feature data sets can be better processed to obtain dynamic obstacles and static obstacles, thereby achieving automatic and accurate identification of dynamic and static obstacles, and facilitating the subsequent automatic determination of the collision body type of the obstacle and the establishment of an obstacle model.

[0068] In addition, for a robot that executes a task list, other robot arms in motion are also dynamic obstacles to the robot arm. According to the priority of the task, the robot arm's position information and the task operation space range are used to perform task binding assignments, and the subordinate relationship of the robot arm's obstacle avoidance path planning is assigned. The execution robot arm with a high task level is given priority for path planning, and the multiple robot arms involved in its path are updated frame by frame. On this basis, the data is accessed through the interface for grasping, and the path preview is used for collision detection to determine whether there is collision interference between the operation of multiple robot arms, as well as uneven resource allocation. Monitor the distance between each part of the robot arm itself and other objects in the twin environment. When this distance is less than the preset safety threshold, it is considered that a collision has occurred. Trigger the robot arm collision detection mechanism for emergency stop, record path data, adjust parameters in time, perform multiple collision detections, and obtain the optimal path through deep learning of multiple robot arms.

[0069] A specific implementation of generating a feasible motion path of a robot arm for performing a grasping task is as follows: Figure 7 As shown, Figure 7 It can be seen that the grasping target point, that is, the end point position, is first determined; then the grasping posture is generated; when the grasping signal is obtained, the grasping path planning of the robot arm is carried out to determine the nodes and paths; then a collision detection is performed, and if no collision occurs, it is determined whether the target point has been reached. If so, the planning of a grasping path is completed and a feasible motion path is obtained; in the event of a collision, the nodes and paths are re-determined, and the subsequent steps are continued.

[0070] An embodiment of the present application also provides a method for controlling multiple robotic arms, comprising: determining an optimal motion path of each of the robotic arms according to any of the methods described; and controlling the movement of each of the robotic arms according to the multiple optimal motion paths.

[0071] In the control method of multiple robotic arms, the optimal motion path of the multiple robotic arms is determined according to the described method. The method determines the order of multiple tasks through the safety indicators, urgency indicators, correlation indicators, data volume indicators and stability indicators of multiple tasks, thereby realizing the priority sorting of multiple tasks; by identifying the dynamic and static obstacles in the environment where the assembly unit is located and determining the collision body type, the dynamic obstacle model and the static obstacle model in the environment are automatically constructed; through the digital twin technology, the task priority order and the obstacle model in the environment are integrated to plan the paths for executing tasks for multiple robotic arms, thereby obtaining the optimal motion path with the shortest execution time for multiple tasks, realizing the collaborative path planning of multiple robotic arms, and then controlling each robotic arm to move according to its corresponding optimal motion path, thereby realizing the execution of multiple tasks, ensuring that the collaborative work efficiency of multiple robotic arms is high, and solving the technical problem of low efficiency of tasks completed by collaborative multiple robotic arms in the prior art.

[0072] In an optional scheme, the movement of each of the robotic arms is controlled according to the multiple optimal motion paths, including: converting the optimal motion path of each of the robotic arms to obtain an instruction code for the controller of the assembly unit; controlling the controller to execute the instruction code so that the controller controls the movement of each of the robotic arms; when the movement of any of the robotic arms does not meet predetermined requirements, adjusting the target parameters in the process of determining the optimal motion path, and re-determining the optimal motion path according to the adjusted target parameters, the target parameters including at least part of the following: the relative position relationship between any two of the robotic arms, the relative position relationship between the robotic arms and a static obstacle model, and the relative position relationship between the robotic arms and a dynamic obstacle model. In this embodiment, the optimal motion path of each robotic arm is converted into an instruction code of the controller of the assembly unit, and the controller executes the instruction code to control each robotic arm in the assembly unit to move according to the corresponding optimal path. When the predetermined requirements are not met during the actual operation of the robotic arm, the relative position relationship of the components of the model in the path planning method of the multi-robotic arm is adjusted, and then the optimal motion path of each robotic arm to perform multiple tasks in the execution order is planned again according to the execution order of the tasks, the adjusted digital twin model, the dynamic obstacle model and the static obstacle model, thereby further ensuring that the final optimal motion path is more consistent with the actual working condition of the assembly unit and more accurate.

[0073] Among them, the predetermined requirements are requirements for the actual movement of the robotic arm, and the requirements can be flexibly set according to actual needs, such as no collision during the movement of the robotic arm, the closest distance between the robotic arm and the obstacle meets the safety distance of the robotic arm, multiple tasks are successfully executed, etc.

[0074] Specifically, after completing the path planning according to the path planning method of multiple robotic arms, the motion trajectory in the twin environment is converted into an instruction code that can be understood by the physical entity controller of the robotic arm. Since the robotic arm in the physical world may have installation errors, joint clearances, load changes and other factors, it is necessary to fine-tune and calibrate the path and parameters exported from the virtual environment. Upload the generated instruction code to the controller of the physical entity of the robotic arm and perform preliminary tests in a safe environment. Observe whether the actual movement of the robotic arm is as expected, check for collision risks, and adjust the speed and acceleration parameters to ensure the smoothness and safety of the movement. Depending on the test results, it may be necessary to return to the twin's collision detection stage to adjust the path or parameters, and then import the physical entity of the robotic arm for testing again. This process may require multiple iterations until the best effect is achieved.

[0075] The overall process of the multi-robot path planning and control method of the present application is as follows: Figure 8 As shown, it includes: step S1, determining tasks, assigning priorities, assigning corresponding tasks to each robotic arm through distributed coordination, and planning paths by APF-RRT algorithm; step S2, acquiring the current status of multiple robotic arms through data collection; step S3, determining the optimal path according to the current status and target status of multiple robotic arms through collision detection in the twin scene; step S4, feeding back the optimal path result to the robotic arm in the physical world and monitoring the operating status.

[0076] Through the method, each robotic arm in the assembly unit can move and operate independently, cooperate with the obstacle avoidance function, perform precise operations and collaboration, reduce the risk of collision, complete the work with less human intervention, and create a safer working environment for the industrial site.

[0077] The solution of the present application can allocate and manage the tasks of the assembly unit, and collect, model and store image information and point cloud information, and filter the parts of the secondary collected information that are the same as the stored information to ensure the smoothness of the modeling process and avoid problems such as redundant data and repeated processes; make dynamic and static obstacle judgments for the model, and make facet quantity and safety distance condition judgments based on the obstacles, and automatically classify and generate corresponding grid bodies through two mandatory condition judgments, and simulate the paths of multiple robotic arms in the assembly unit during grasping through the twin environment, which can ensure the optimal layout and task allocation when multiple robotic arms work together, and avoid idleness and waste of resources. In the twin environment preview, collision problems can be discovered and solved before actual operation, reducing maintenance costs, and engineers can perform multiple iterations under risk-free conditions to shorten the development cycle.

[0078] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0079] The embodiment of the present application also provides a path planning device for multiple robotic arms. It should be noted that the path planning device for multiple robotic arms in the embodiment of the present application can be used to execute the path planning method for multiple robotic arms provided in the embodiment of the present application. The device is used to implement the embodiments and preferred implementations, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0080] The following is an introduction to the multi-robot path planning device provided in an embodiment of the present application.

[0081] Fig. 9 Schematic diagram of a multi-robot path planning device according to an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0082] An acquisition unit 10 is used to acquire multiple tasks to be executed, and at least determine the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least part of the tasks are tasks performed in collaboration with multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks;

[0083] Specifically, according to the safety index, urgency index, correlation index, data volume index and stability index of each task, at least multiple tasks are prioritized to obtain the execution order. At least part of the tasks are tasks that are executed collaboratively by multiple robotic arms, which means that when there are m robotic arms, there is at least one task among the multiple tasks that requires n robotic arms to be executed collaboratively, m≥n. For multiple robotic arms, each of the robotic arms has limit requirements in all directions, and these limit requirements constitute the range of motion of the robotic arms. The correlation indicators include but are not limited to the degree of resource dependence between two tasks, the degree of dependence on the execution order, the number of interactions, and the amount of data interaction.

[0084] A construction unit 20, used to construct a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms;

[0085] Specifically, the digital twin model is a model obtained by mapping and simulating entities, equipment, systems, etc. in the physical world in a virtual space through digital means using digital twin technology.

[0086] an identification unit 30, configured to identify dynamic obstacles and static obstacles in the environment where the assembly unit is located, determine a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determine a second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle;

[0087] Specifically, the dynamic obstacle refers to an obstacle that causes the robot arm to move in the assembly unit; the static obstacle refers to an obstacle that causes the robot arm to not move in the assembly unit.

[0088] An establishing unit 40, configured to establish a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and to establish a static obstacle model according to the second collision body type;

[0089] Specifically, the dynamic obstacle model is a model obtained by mapping the dynamic obstacle into a virtual space, and the static obstacle model is a model obtained by mapping the static obstacle into a virtual space.

[0090] A first determination unit 50 is used to determine a feasible motion path for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model;

[0091] Specifically, the feasible motion path is a motion path that enables the plurality of robot arms to execute each task in the execution order. Each robot arm may have only one feasible motion path or may have multiple feasible motion paths.

[0092] The second determination unit 60 is used to determine the optimal motion path of each of the robot arms from the multiple feasible motion paths when there are multiple feasible motion paths, taking the shortest execution time of the multiple tasks as the optimal goal.

[0093] Specifically, the execution time is the total time taken to execute multiple tasks in the execution order. When there are multiple feasible motion paths for the robot, the optimal motion path is the path among the multiple feasible motion paths that satisfies the goal of the shortest execution time. When there is only one feasible motion path for the robot, this one feasible motion path is the optimal motion path for the robot.

[0094] The path planning device for the multi-robot arm includes a processor and a memory, wherein the acquisition unit, the construction unit, the identification unit, the establishment unit, the first determination unit, and the second determination unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0095] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of low efficiency of tasks completed by multi-robot collaboration in the prior art can be at least solved by adjusting kernel parameters.

[0096] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0097] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the multi-robotic arm path planning method.

[0098] Specifically, the path planning method of multiple robotic arms includes:

[0099] Step S201, obtaining multiple tasks to be executed, and determining at least the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least part of the tasks are tasks performed collaboratively by multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks;

[0100] Specifically, according to the safety index, urgency index, correlation index, data volume index and stability index of each task, at least multiple tasks are prioritized to obtain the execution order. At least part of the tasks are tasks that are executed collaboratively by multiple robotic arms, which means that when there are m robotic arms, there is at least one task among the multiple tasks that requires n robotic arms to be executed collaboratively, m≥n. For multiple robotic arms, each of the robotic arms has limit requirements in all directions, and these limit requirements constitute the range of motion of the robotic arms. The correlation indicators include but are not limited to the degree of resource dependence between two tasks, the degree of dependence on the execution order, the number of interactions, and the amount of data interaction.

[0101] Step S202, constructing a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms;

[0102] Specifically, the digital twin model is a model obtained by mapping and simulating entities, equipment, systems, etc. in the physical world in a virtual space through digital means using digital twin technology.

[0103] Step S203, identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located, determining a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determining a second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle;

[0104] Specifically, the dynamic obstacle refers to an obstacle that causes the robot arm to move in the assembly unit; the static obstacle refers to an obstacle that causes the robot arm to not move in the assembly unit.

[0105] Step S204, establishing a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and establishing a static obstacle model according to the second collision body type;

[0106] Specifically, the dynamic obstacle model is a model obtained by mapping the dynamic obstacle into a virtual space, and the static obstacle model is a model obtained by mapping the static obstacle into a virtual space.

[0107] Step S205, determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model;

[0108] Specifically, the feasible motion path is a motion path that enables the plurality of robot arms to execute each task in the execution order. Each robot arm may have only one feasible motion path or may have multiple feasible motion paths.

[0109] Step S206, when there are multiple feasible motion paths, taking the shortest execution time of multiple tasks as the optimal goal, determine the optimal motion path of each of the robot arms from the multiple feasible motion paths.

[0110] Specifically, the execution time is the total time taken to execute multiple tasks in the execution order. When there are multiple feasible motion paths for the robot, the optimal motion path is the path among the multiple feasible motion paths that satisfies the goal of the shortest execution time. When there is only one feasible motion path for the robot, this one feasible motion path is the optimal motion path for the robot.

[0111] According to another aspect of the present application, a method for controlling multiple robotic arms is provided, comprising: determining an optimal motion path of each of the robotic arms according to any of the methods described; and controlling the movement of each of the robotic arms according to the multiple optimal motion paths.

[0112] Optionally, the movement of each of the robotic arms is controlled according to the multiple optimal motion paths, including: converting the optimal motion path of each of the robotic arms to obtain an instruction code for the controller of the assembly unit; controlling the controller to execute the instruction code so that the controller controls the movement of each of the robotic arms; when the movement of any of the robotic arms does not meet predetermined requirements, adjusting the target parameters in the process of determining the optimal motion path, and redetermining the optimal motion path according to the adjusted target parameters, the target parameters including at least part of the following: the relative position relationship between any two of the robotic arms, the relative position relationship between the robotic arms and a static obstacle model, and the relative position relationship between the robotic arms and a dynamic obstacle model.

[0113] An embodiment of the present invention provides a processor, which is used to run a program, wherein the path planning method of the multi-robotic arm is executed when the program is run.

[0114] An embodiment of the present invention provides a device, the device including a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0115] Step S201, obtaining multiple tasks to be executed, and determining at least the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least part of the tasks are tasks performed collaboratively by multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks;

[0116] Step S202, constructing a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms;

[0117] Step S203, identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located, determining a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determining a second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle;

[0118] Step S204, establishing a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and establishing a static obstacle model according to the second collision body type;

[0119] Step S205, determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model;

[0120] Step S206, when there are multiple feasible motion paths, taking the shortest execution time of multiple tasks as the optimal goal, determine the optimal motion path of each of the robot arms from the multiple feasible motion paths.

[0121] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0122] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement at least the following method steps:

[0123] Step S201, obtaining multiple tasks to be executed, and determining at least the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least part of the tasks are tasks performed collaboratively by multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks;

[0124] Step S202, constructing a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms;

[0125] Step S203, identifying dynamic obstacles and static obstacles in the environment where the assembly unit is located, determining a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determining a second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle;

[0126] Step S204, establishing a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and establishing a static obstacle model according to the second collision body type;

[0127] Step S205, determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model;

[0128] Step S206, when there are multiple feasible motion paths, taking the shortest execution time of multiple tasks as the optimal goal, determine the optimal motion path of each of the robot arms from the multiple feasible motion paths.

[0129] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0136] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0137] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0138] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A path planning method for multiple robotic arms, characterized in that: include: Acquire multiple tasks to be executed, and determine at least the execution order of the multiple tasks according to the safety index, urgency index, correlation index, data volume index and stability index of the multiple tasks, at least some of the tasks are tasks performed in collaboration with multiple robotic arms, the safety index is used to characterize the degree of overlap of the motion ranges of the multiple robotic arms during the execution of the tasks, the urgency index is used to characterize the urgency of the tasks, the correlation index is used to characterize the degree of dependence between the tasks and any other tasks, the data volume index is used to characterize the amount of data required to execute the tasks, and the stability index is used to characterize the historical execution success rate of the tasks; Constructing a digital twin model of an assembly unit, wherein the assembly unit includes a plurality of the robotic arms; Identify dynamic obstacles and static obstacles in the environment where the assembly unit is located, determine a first collision body type according to the distance between the dynamic obstacle and each of the robotic arms, and determine a second collision body type according to the triangular patch information of the static obstacle and the distance between the static obstacle and each of the robotic arms, wherein the first collision body type is the collision body type of the dynamic obstacle, and the second collision body type is the collision body type of the static obstacle; Establishing a dynamic obstacle model according to the first collision body type and the motion trajectory of the dynamic obstacle, and establishing a static obstacle model according to the second collision body type; Determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model; In the case where there are multiple feasible motion paths, the optimal motion path of each of the robot arms is determined from the multiple feasible motion paths with the shortest execution time of the multiple tasks as the optimal goal. Determining the type of a first collision body according to the distance between the dynamic obstacle and each of the mechanical arms includes: When the distance between the dynamic obstacle and each of the robotic arms is greater than or equal to a first predetermined multiple of a safety distance of the robotic arms, determining that the first collision body type is a capsule body; When the distance between the dynamic obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance and greater than or equal to the second predetermined multiple of the robotic arm safety distance, the first collision body type is determined to be a triangular mesh body, wherein the first predetermined multiple and the second predetermined multiple are both greater than or equal to 1.

2. The method according to claim 1, characterized in that: According to the safety index, urgency index, relevance index, data volume index and stability index of the plurality of tasks, at least determining the execution order of the plurality of tasks comprises: In the case where the correlation index is greater than a predetermined threshold, performing predetermined processing on the two tasks corresponding to the correlation index greater than the predetermined threshold, so that the correlation index after the predetermined processing is less than or equal to the predetermined threshold, and obtaining multiple new tasks, wherein the predetermined processing includes at least one of the following: splitting the task into multiple subtasks, reorganizing the multiple tasks, and reorganizing some of the split subtasks; When the relevance index is less than or equal to the predetermined threshold, weighted summing the safety index, the urgency index, the relevance index, the data volume index and the stability index of each task is performed to obtain a priority index of the task; The plurality of tasks are sorted in descending order according to the priority indexes to obtain the execution order.

3. The method according to claim 1, characterized in that The triangular face information includes the number of triangular facets constituting the static obstacle and the direction of the normal vector of each triangular facet, and determining the second collision body type according to the triangular facet information of the static obstacle and the distance between the static obstacle and each of the robotic arms, including: According to the direction of the normal vector, determine the outer surface triangular facets of the static obstacle from the plurality of triangular facets, and obtain the number of the outer surface triangular facets; When the number of triangular facets on the outer surface is less than a number threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to a first predetermined multiple of a robotic arm safety distance, determining that the type of the second collision body is a sphere; When the number of triangular facets on the outer surface is greater than or equal to the number threshold and the distance between the static obstacle and each of the robotic arms is greater than or equal to the first predetermined multiple of the robotic arm safety distance, determining that the second collision body type is a box body; When the number of triangular facets on the outer surface is less than the number threshold and the distance between the static obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance and greater than or equal to the second predetermined multiple of the robotic arm safety distance, determining that the type of the second collision body is a concave polyhedron; When the number of triangular facets on the outer surface is greater than or equal to the number threshold, and the distance between the static obstacle and each of the robotic arms is less than the first predetermined multiple of the robotic arm safety distance, and greater than or equal to the second predetermined multiple of the robotic arm safety distance, the second collision body type is determined to be a grid body, wherein the first predetermined multiple and the second predetermined multiple are both greater than or equal to 1.

4. The method according to claim 1, characterized in that: Determining feasible motion paths for each of the robotic arms to perform the plurality of tasks in the execution order according to the execution order, the digital twin model, the dynamic obstacle model, and the static obstacle model, including: According to the execution order, a task list is assigned to each of the robotic arms in the digital twin model, and a starting position and an end position of each of the robotic arms are determined according to the task list; Constructing a kinematic model of each of the robotic arms; According to the starting position and the end position of the target robotic arm, with the goal of avoiding the dynamic obstacle model, the static obstacle model and the kinematic models of other robotic arms, the APF-RRT algorithm is used to plan the feasible motion path of the target robotic arm, and the feasible motion path corresponding to each robotic arm is obtained. The target robotic arm is one of the multiple robotic arms, and the other robotic arms are the robotic arms other than the target robotic arm.

5. The method according to claim 1, characterized in that Build a digital twin model of the assembly unit, including: Acquire image information and point cloud information of a physical entity in the assembly unit to obtain first information, wherein the physical entity includes a plurality of the robotic arms, materials to be grasped, and a material transportation track; Based on the first information, construct an initial twin model corresponding to the physical entity using digital modeling technology; Periodically acquiring the image information and the point cloud information of the physical entity to obtain second information; comparing the first information and the second information, filtering out information in the second information that is the same as the first information, and using the remaining second information as distinguishing information; According to the distinguishing information, model components corresponding to the distinguishing information are added to the initial twin model to obtain the digital twin model.

6. The method according to claim 5, characterized in that According to the first information, an initial twin model corresponding to the physical entity is constructed using a digital modeling technology, including: Determine the size information, position information and assembly relationship of the physical entity according to the first information; According to the size information, the position information and the assembly relationship, the digital modeling technology is used to construct a 1:1 three-dimensional model corresponding to the physical entity in a virtual environment; Adding attribute information of the physical entity to the three-dimensional model, the attribute information including material attributes, gripping force attributes, and contact force attributes; Acquire the motion range of multiple joints in each of the robotic arms and the parent-child relationship between the multiple joints, wherein the parent-child relationship represents the follow-up relationship between two connected joints; According to the motion range and the parent-child relationship, motion range restrictions and parent-child relationship restrictions are added to the three-dimensional model of the robotic arm to which the attribute information is added, so as to obtain the initial twin model.

7. The method according to claim 1, characterized in that Identifying dynamic obstacles and static obstacles in the environment in which the assembly unit is located, including: Acquire historical pixel features of the environment where the assembly unit is located, where the historical pixel features are obtained by extracting pixel features from a plurality of historical environment images; Obtaining an initial Gaussian mixture model, and training the initial Gaussian mixture model using an iterative learning algorithm according to the historical pixel features to obtain a Gaussian mixture model; Acquire a plurality of continuous frames of images of the environment, and extract pixel features of the plurality of continuous frames of images; The Gaussian mixture model is used to analyze the pixel features to determine the dynamic obstacles and the static obstacles in the environment.

8. A method for controlling a multi-robot arm, characterized in that: include: Determine the optimal motion path of each of the robotic arms according to the method of any one of claims 1 to 7; The movement of each of the robot arms is controlled according to the multiple optimal motion paths.

9. The method according to claim 8, characterized in that According to the plurality of optimal motion paths, controlling the movement of each of the robotic arms comprises: Converting the optimal motion path of each of the robotic arms to obtain an instruction code for a controller of an assembly unit; Controlling the controller to execute the instruction code so that the controller controls the movement of each of the mechanical arms; When the movement of any of the robotic arms does not meet the predetermined requirements, the target parameters in the process of determining the optimal motion path are adjusted, and the optimal motion path is re-determined based on the adjusted target parameters, and the target parameters include at least part of the following: the relative position relationship between any two of the robotic arms, the relative position relationship between the robotic arms and the static obstacle model, and the relative position relationship between the robotic arms and the dynamic obstacle model.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

11. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 7.

Images