Multi-robot visual-haptic fusion control and fault-tolerant system for flexible assembly

Abstract

The present application relates to a kind of flexible assembly-oriented multi-arm vision-haptics fusion control and fault-tolerant system and method, it is related to manipulator control technical field, system includes: data construction module, for constructing heterogeneous scene graph model;Task allocation module, for generating collaborative representation vector, and based on collaborative representation vector output task priority ranking result and initial candidate task allocation set;Trajectory generation module, for generating multi-arm collaborative schedule and corresponding time parameterized motion trajectory;Instruction generation module, for generating used to drive each manipulator actuator compliant assembly control instruction;Signal output module, for making digital twin simulation module output corrected impedance parameter set and for triggering trajectory re-planning re-planning trigger signal;Fault-tolerant control module, for starting fault-tolerant switching mechanism based on reinforcement learning strategy, the dynamic redistribution of multi-arm task or anti-interference control is executed.The control precision of system can improve manipulator.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, and in particular to a vision-force fusion control and fault-tolerant system, method, electronic device and non-transitory computer-readable storage medium for multi-robotic arms in flexible assembly. Background Technology

[0002] Currently, multi-robotic arm collaborative operations are widely used in electronic manufacturing, precision assembly, and automated production in flexible assembly lines. Roboarms typically rely on two-dimensional or three-dimensional vision systems to locate and identify workpieces, guiding them to perform gripping, mounting, or welding operations based on pre-calibrated pose information. In some high-precision assembly scenarios, force sensors are also introduced to monitor contact forces, enabling basic force control or force limiting protection.

[0003] However, existing multi-robotic arm flexible assembly systems still rely mainly on simple superposition at the level of visual and force sensing coordination, lacking a unified control mechanism with deep integration. This results in insufficient adaptability of the system to the assembly process when there are assembly tolerances, slight workpiece offsets, or changes in contact stiffness, which can easily lead to problems such as excessive assembly stress, misalignment of mounting, or unstable welding quality. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a multi-manipulator vision-force fusion control and fault-tolerant system, method, electronic device, and non-transitory computer-readable storage medium that can improve the control accuracy of manipulators.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0006] This invention provides a vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly, the system comprising:

[0007] Furthermore, the data construction module is also used for:

[0008] Based on the three-dimensional point cloud data, the surface of the workpiece is segmented and features are extracted to obtain a set of spatial geometric features of the workpiece;

[0009] Based on the set of spatial geometric features of the workpiece and the working environment state information, an environmental topology structure containing spatial constraint relationships is constructed.

[0010] Based on the environmental topology and the contact force information at the end of the robotic arm, multiple types of nodes and the constraint relationships between nodes are established to obtain the heterogeneous scene graph model.

[0011] Furthermore, the task allocation module is also used for:

[0012] Based on the heterogeneous scene graph model, initial feature encoding is performed on various types of nodes to obtain a set of node feature vectors;

[0013] Based on the set of node feature vectors, the collaborative relationship features of multiple robotic arms are extracted through cross-node information propagation and aggregation to obtain the intermediate collaborative feature representation.

[0014] Relationship reasoning is performed based on the intermediate collaborative feature representation to generate the collaborative representation vector.

[0015] Furthermore, the task allocation module is also used for:

[0016] Based on the collaborative representation vector, the executability of each assembly task is evaluated to obtain the task feasibility evaluation result;

[0017] Based on the task feasibility assessment results, the assembly tasks are prioritized to obtain the task priority ranking results.

[0018] Based on the task priority ranking results and the current load status of each robotic arm, the initial candidate task allocation set is generated.

[0019] Furthermore, the trajectory generation module is also used for:

[0020] Based on the cooperative representation vector, the coupling constraints between multiple robotic arms are modeled to obtain the cooperative constraint state of the multiple robotic arms;

[0021] Based on the multi-robotic arm collaborative constraint state, the initial candidate task allocation set is coordinated and corrected to obtain an intermediate task allocation scheme;

[0022] Based on the intermediate task allocation scheme, the three-sub-problem collaborative optimizer is input to perform joint optimization of task allocation and task scheduling, and the joint optimized task allocation and scheduling results are obtained.

[0023] Furthermore, the trajectory generation module is also used for:

[0024] Based on the jointly optimized task allocation and scheduling results, a multi-robotic arm motion constraint space is constructed to obtain a set of motion constraints.

[0025] Based on the set of motion constraints and the assembly accuracy requirements, the motion path of the multiple robotic arms is initially planned to obtain the initial motion path.

[0026] Based on the initial motion path, time parameterization is performed to generate the time parameterized motion trajectory.

[0027] Furthermore, the instruction generation module is also used for:

[0028] Based on the time-parameterized motion trajectory, the desired pose information of each robotic arm at the current moment is obtained, and the desired pose sequence is obtained.

[0029] Based on the real-time collected visual pose confidence information, the expected pose sequence is corrected to obtain the visually corrected pose.

[0030] The fused pose control quantity is obtained by fusing the visually corrected pose and the force contact estimation information.

[0031] Furthermore, the instruction generation module is also used for:

[0032] Based on the fused pose control quantity, the current assembly contact state is calculated, and the assembly contact state parameters are obtained;

[0033] Based on the assembly contact state parameters, the impedance control parameters are adaptively adjusted to obtain the corrected impedance parameters;

[0034] The compliant assembly control command is generated based on the modified impedance parameter.

[0035] Furthermore, the signal output module is also used for:

[0036] Based on the execution status data of the compliant assembly control command, a virtual execution status model is constructed to obtain virtual execution feedback;

[0037] Based on the virtual execution feedback and the real-time scene change data, a consistency comparison is performed in the digital twin environment to obtain the deviation detection result;

[0038] Based on the deviation detection results, the control parameters are corrected in reverse, and the corrected impedance parameter set and the replanning trigger signal are output.

[0039] Furthermore, the fault-tolerant control module is also used for:

[0040] Based on the replanning trigger signal or fault status information, the fault type is determined to obtain fault characteristic information;

[0041] Based on the fault characteristic information, a risk assessment is performed on the current multi-robotic arm task execution status to obtain the task risk level;

[0042] Based on the task risk level, a corresponding fault-tolerant control action is selected through a reinforcement learning strategy to complete the dynamic redistribution or anti-interference control of multi-robotic arm tasks.

[0043] This invention also provides a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly, the method comprising:

[0044] The system collects 3D point cloud data of the workpiece, contact force information of the robotic arm end effector, and operational environment status information. It then normalizes the collected data to obtain normalized multi-source information and constructs a heterogeneous scene graph model based on the normalized multi-source information.

[0045] The heterogeneous scene graph model is input into a graph neural network for cross-node feature aggregation and relation reasoning to generate a collaborative representation vector. Based on the collaborative representation vector, the task priority ranking result and the initial candidate task allocation set are output. The collaborative representation vector represents the cross-arm interference risk, path reachability and load capacity between multiple robotic arms.

[0046] The collaborative representation vector and the initial candidate task allocation set are input into a three-sub-problem collaborative optimizer based on reinforcement learning. Joint optimization is performed in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. A dynamic cost weight adaptive mechanism is used to make a real-time trade-off between assembly accuracy and collision avoidance safety, generating a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet high-dimensional freedom constraints.

[0047] Using the time-parameterized motion trajectory as a reference input, the visual pose confidence information and force contact estimation information collected in real time are fused together to perform visual-force deep fusion control. By adaptively adjusting the pose weight and impedance control parameters, compliant assembly control commands for driving each robotic arm actuator are generated.

[0048] The execution status data of the compliant assembly control command and the real-time scene change data are input into the digital twin simulation module. Online dynamic scene adaptation and control parameter reverse optimization are performed in the digital twin environment. The corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning are output.

[0049] When the replanning trigger signal is received or a fault condition is detected in any robotic arm actuator, a fault-tolerant switching mechanism is initiated based on a reinforcement learning strategy to perform dynamic redistribution of tasks among multiple robotic arms or anti-interference control.

[0050] Furthermore, to achieve the above objectives, the present invention also proposes an electronic device, comprising: a memory for storing computer software programs; and a processor for reading and executing the computer software programs, thereby realizing the vision-force fusion control and fault-tolerant method for multi-manipulators in flexible assembly as described above.

[0051] Furthermore, to achieve the above objectives, the present invention also proposes a non-transitory computer-readable storage medium storing a computer software program, which, when executed by a processor, implements the vision-force fusion control and fault-tolerant method for multi-manipulators in flexible assembly as described above.

[0052] The beneficial effects of this invention are:

[0053] (1) Through the three-dimensional point cloud refinement and workpiece spatial geometric feature extraction, this invention can accurately characterize the workpiece shape, key assembly points and surface constraint characteristics, and combine force information for dynamic fusion, so that the robotic arm can achieve high-precision pose control when performing operations such as mounting, welding or dispensing, and significantly reduce assembly errors and collision risks.

[0054] (2) By constructing a heterogeneous scene graph model, the present invention integrates workpieces, robotic arms and environmental obstacles into a graph structure expression, and establishes various relationships between nodes such as spatial adjacency, reachability and contact constraints, providing a unified data foundation for collaborative strategies based on graph neural networks, realizing multi-robotic arm task allocation, path planning and dynamic interference prediction, thereby improving production line throughput and operational stability.

[0055] (3) Through the dynamic updating of the environmental topology and the real-time correction of force feedback information, this invention can sense the movement of obstacles, belt speed fluctuations and workpiece offset in the work area, and adjust the trajectory, impedance parameters and task allocation in real time to achieve adaptive control of dynamic work scenarios, ensuring the continuity and safety of operation. Attached Figure Description

[0056] Figure 1 A scene diagram illustrating a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly, as provided by this invention.

[0057] Figure 2 A schematic diagram of a vision-force fusion control and fault-tolerant system for a multi-robotic arm oriented to flexible assembly provided by the present invention;

[0058] Figure 3 A flowchart of a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly provided by the present invention;

[0059] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0060] Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Please see Figure 1 , Figure 1 This is a scene diagram illustrating a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly, as provided by the present invention. Figure 1 As shown, the terminal and server are connected via a network, such as a wired or wireless network. The terminal can include, but is not limited to, portable devices such as mobile phones and tablets with various network platform applications installed, as well as fixed terminals such as computers, kiosks, and advertising machines. The server provides users with various business services, including service push servers and user recommendation servers.

[0063] It should be noted that, Figure 1 The scenario diagram shown is merely an example of a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly. The terminals, servers, and application scenarios described in the embodiments of the present invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0064] The terminal can be used for:

[0065] The system collects 3D point cloud data of the workpiece, contact force information of the robotic arm end effector, and status information of the working environment. The collected data is normalized to obtain normalized multi-source information, and a heterogeneous scene graph model is constructed based on the normalized multi-source information.

[0066] The heterogeneous scene graph model is input into the graph neural network for cross-node feature aggregation and relation reasoning to generate a collaborative representation vector. Based on the collaborative representation vector, the task priority ranking result and the initial candidate task allocation set are output. Among them, the collaborative representation vector represents the cross-arm interference risk, path reachability and load capacity between multiple robotic arms.

[0067] The collaborative representation vector and the initial candidate task allocation set are input into a reinforcement learning-based three-sub-problem collaborative optimizer. Joint optimization is performed in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. A dynamic cost weight adaptive mechanism is used to make a real-time trade-off between assembly accuracy and collision avoidance safety, generating a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet high-dimensional freedom constraints.

[0068] Using time-parameterized motion trajectory as reference input, the system integrates real-time acquired visual pose confidence information and force contact estimation information to perform visual-force deep fusion control. By adaptively adjusting pose weights and impedance control parameters, it generates compliant assembly control commands to drive each robotic arm actuator.

[0069] The execution status data of the compliant assembly control command and the real-time scene change data are input into the digital twin simulation module. Online dynamic scene adaptation and control parameter reverse optimization are performed in the digital twin environment. The corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning are output.

[0070] When a replanning trigger signal is received or a fault condition is detected in any robotic arm actuator, a fault-tolerant switching mechanism is initiated based on a reinforcement learning strategy to perform dynamic redistribution of tasks among multiple robotic arms or anti-interference control.

[0071] Please see Figure 2 , Figure 2 This invention provides a structural schematic diagram of a multi-robotic arm vision-force fusion control and fault-tolerant system for flexible assembly.

[0072] like Figure 2 As shown in the figure, the multi-robotic arm vision-force fusion control and fault-tolerant system for flexible assembly proposed in this embodiment of the invention includes:

[0073] The data construction module 201 is used to collect the three-dimensional point cloud data of the workpiece, the contact force information of the robotic arm end effector, and the status information of the working environment. It performs normalization processing on the collected data to obtain normalized multi-source information, and constructs a heterogeneous scene graph model based on the normalized multi-source information.

[0074] In some embodiments, the data construction module 201 can acquire 3D point cloud data of the workpiece through a 3D vision sensor installed at the work site, collect end-effector contact force information through a force sensor installed at the end of the robotic arm, and simultaneously collect environmental status information such as obstacle distribution, equipment status, and operating parameters within the work area through an environmental sensing unit. The above-mentioned multi-source heterogeneous data is normalized by scaling, amplitude constraints, and outlier removal to obtain normalized multi-source information. Based on this, a heterogeneous scene graph model representing the coupling relationship between multiple robotic arms, the robotic arm, environmental targets, and their spatial relationships is constructed using the workpiece, the robotic arm, environmental targets, and their spatial relationships as nodes and edges, providing a unified data representation basis for subsequent task allocation and path planning.

[0075] In some embodiments, the data construction module 201 is further configured to:

[0076] Based on the 3D point cloud data, the surface of the workpiece is segmented and features are extracted to obtain a set of spatial geometric features of the workpiece;

[0077] Based on the set of spatial geometric features of the workpiece and combined with the status information of the working environment, an environmental topology structure containing spatial constraint relationships is constructed.

[0078] Based on the environmental topology and the contact force information at the end of the robotic arm, a heterogeneous scene graph model is obtained by establishing multiple types of nodes and the constraint relationships between nodes.

[0079] First, based on the acquired 3D point cloud data of the workpiece, point cloud preprocessing is performed on the workpiece surface to ensure data integrity and stability. Specific processing includes: statistical filtering and radius filtering of the point cloud to remove random noise and isolated points; reducing the number of point clouds using voxel mesh downsampling or adaptive sampling methods based on surface features while preserving key geometric features; and ensuring the preservation of edges and key assembly feature points by using neighborhood density analysis or distance threshold detection when removing outliers. After preprocessing, surface region segmentation is performed on the point cloud to distinguish continuous curved surfaces, edge regions, and assembly feature regions. Region segmentation can be achieved through normal vector consistency clustering, curvature change analysis, or feature point-based region growing methods. Subsequently, spatial feature information reflecting the geometric attributes of the workpiece is extracted from each segmented region, including curvature features, normal vector distribution, boundary contour features, local surface curvature histograms, and key assembly pose features, such as hole centers, boss edges, and fixture contact surface directions. Through these processes, a set of spatial geometric features of the workpiece is formed. This set can accurately describe the shape, geometric constraints, and assembly key point positions of the workpiece in three-dimensional space, providing basic data for subsequent path planning and operational constraints.

[0080] Secondly, the spatial relationship between the workpiece and the working environment can be modeled based on the workpiece's spatial geometric feature set and the working environment status information. The working environment status information includes the spatial location, size, fixed or movable attributes of obstacles within the working area, as well as the location of each robotic arm base, the range of the workspace, and the accessibility of the robotic arm joints. The module constructs a spatial mapping matrix by aligning the workpiece feature coordinates with the environment coordinate system and uses adjacency analysis, shortest path search, or voxel occupancy grid methods to calculate the adjacency relationships, accessibility, and potential interference relationships between the workpiece, obstacles, and the robotic arm's workspace. Simultaneously, dynamic information such as conveyor belt speed, moving obstacle trajectories, and robotic arm motion states are incorporated into the modeling to achieve dynamic updates of the environmental topology. The final constructed environmental topology reflects the spatial constraints and interference relationships between objects in the working scene, providing a structured constraint basis for path planning, collision detection, and obstacle avoidance strategies.

[0081] Furthermore, based on the environmental topology and the contact force information of the robotic arm's end effector, node-based modeling is performed on various objects in the work scenario. The module integrates workpiece nodes, robotic arm nodes, and environmental obstacle nodes into a unified graph structure representation, and establishes various constraint relationships between nodes reflecting spatial adjacency, motion accessibility, contact constraints, and potential collision risks. The contact force information of the robotic arm's end effector is collected through a six-axis force / torque sensor or a skin-like tactile array, characterizing the actual contact state, force magnitude, and distribution between the workpiece and the tool. This information is used to dynamically adjust the constraint weights between nodes; for example, automatically reducing the trajectory speed or adjusting the contact point distribution when the contact force abnormally increases, thereby ensuring the safety and compliance of the assembly process. By updating and correcting the constraint relationships in real time, a heterogeneous scene graph model that can simultaneously describe geometric structural relationships, motion constraint relationships, and contact mechanical relationships is finally formed. This model not only comprehensively depicts the complex coupling relationships between multiple robotic arms, workpieces, and the environment, but also provides a unified data foundation and inference carrier for subsequent task allocation, path planning, and collaborative control based on graph neural networks.

[0082] Through the above methods The constructed heterogeneous scene graph model can not only comprehensively depict the complex coupling relationship among multiple manipulators, workpieces and environment, but also provide a unified data basis and reasoning carrier for subsequent task allocation, path planning and collaborative control based on graph neural network Figure 3 Figure 4 .

[0083] The task allocation module 202 is used to input the heterogeneous scene graph model into the graph neural network for cross-node feature aggregation and relation reasoning, generate collaborative representation vectors, and output the task priority ranking results and the initial candidate task allocation set based on the collaborative representation vectors; wherein, the collaborative representation vectors represent the cross-arm interference risk, path reachability and load capacity between multiple robotic arms.

[0084] In some embodiments, the task allocation module 202 can input the constructed heterogeneous scene graph model into the graph neural network, and extract the correlation features between multiple robotic arms in terms of spatial position, motion constraints, and environmental interference through cross-node feature aggregation and relational reasoning, generating a collaborative representation vector that reflects the overall collaborative state of the system. Based on the collaborative representation vector, the urgency, feasibility, and risk level of each assembly task are comprehensively evaluated, the task priority ranking results are output, and an initial candidate task allocation set is further selected. The collaborative representation vector is used to represent key constraint information such as cross-arm interference risk, path reachability, and load capacity among multiple robotic arms.

[0085] In some embodiments, the task allocation module 202 can also be used for:

[0086] Based on the heterogeneous scene graph model, initial feature encoding is performed on various types of nodes to obtain a set of node feature vectors;

[0087] Based on the set of node feature vectors, the collaborative relationship features of multiple robotic arms are extracted through cross-node information propagation and aggregation to obtain the intermediate collaborative feature representation;

[0088] Relational reasoning is performed based on intermediate collaborative feature representations to generate collaborative representation vectors.

[0089] First, the task allocation module 202 receives the heterogeneous scene graph model generated by the data construction module 201 and performs initial feature encoding on various nodes in the scene graph. Nodes include, but are not limited to, robotic arm nodes, task nodes, workpiece nodes, and obstacle nodes. During the initial feature encoding process, the system maps the geometric attributes, spatial position, motion capabilities, contact constraints, and historical operation data of each node into a node feature vector of a unified dimension for subsequent information transmission and aggregation processing. During the encoding process, the module can also normalize the node features, mapping different physical quantities or numerical quantities to the same dimension, thereby ensuring the stability and numerical consistency of subsequent feature calculations.

[0090] Secondly, the module performs cross-node information propagation and aggregation based on the node feature vector set and the edge connection information within the graph structure. Specifically, for each node, the system iteratively collects the feature information of its neighboring nodes and performs weighted aggregation according to the edge type and weight (including spatial adjacency, task dependency, and potential collision risk weights). An attention mechanism can also be introduced to highlight the influence of key neighboring nodes. After several rounds of information propagation and update processing, each node obtains an updated feature representation. The system then aggregates the updated feature vectors of all nodes to form an intermediate collaborative feature representation, thereby capturing high-dimensional features such as collaborative relationships between multiple robotic arms, task interference risks, and path reachability.

[0091] Furthermore, the task allocation module 202 utilizes intermediate collaborative feature representations for relational reasoning to generate a collaborative representation vector. Specifically, the system can perform nonlinear transformations and information fusion on intermediate features through multi-layer neural networks or graph attention networks to extract key collaborative patterns and potential conflict features, forming a collaborative representation vector suitable for reinforcement learning strategies. This collaborative representation vector can guide subsequent task allocation decisions, trajectory planning, and multi-robotic arm collaborative control, enabling task priority ranking, action sequence optimization, and conflict avoidance, providing a unified reasoning basis and intelligent decision support for flexible assembly operations.

[0092] In this way, the task allocation module 202 can fully explore the collaborative relationships and potential constraint characteristics between nodes based on the heterogeneous scene graph model, and generate a high-dimensional collaborative representation that can directly drive reinforcement learning strategies, enabling multiple robotic arms to achieve efficient, accurate, and schedulable flexible collaboration in complex working environments.

[0093] In some embodiments, the task allocation module 202 can also be used for:

[0094] Based on the collaborative representation vector, the executability of each assembly task is evaluated to obtain the task feasibility evaluation results;

[0095] Based on the task feasibility assessment results, the assembly tasks are prioritized to obtain the task priority ranking results;

[0096] Based on the task priority ranking results and the current load status of each robotic arm, an initial candidate task allocation set is generated.

[0097] First, the system evaluates the executability of each assembly task based on the collaborative representation vector generated by the task allocation module 202. Specifically, for each assembly task, the system comprehensively considers the required motion space, workpiece spatial geometric constraints, interference from surrounding obstacles, the robotic arm's motion and load capacity, as well as the task completion time and accuracy requirements, and scores the task's executability in the current scenario. During the executability evaluation process, the system can also introduce normalization processing to map indicators of different dimensions (such as distance, force, time, and accuracy) to the same numerical range, so as to calculate the task feasibility score by weighted comprehensive calculation. After calculation, each assembly task obtains a normalized feasibility evaluation result, reflecting its performance level in the current multi-robotic arm collaborative scenario.

[0098] Secondly, based on the task feasibility assessment results, the system prioritizes each assembly task. During the prioritization process, the system considers not only the executability of individual tasks but also the dependencies between tasks, task completion timing constraints, and potential risks of cross-interference between robotic arms, performing multi-dimensional sorting of the task list. By iteratively adjusting the sorting weights and employing a multi-condition judgment mechanism, the system obtains the task priority ranking results, ensuring that critical, urgent, or constrained tasks are preferentially assigned to suitable robotic arms, thereby improving overall assembly line efficiency and assembly safety.

[0099] Next, the system combines the current load status of each robotic arm, the real-time available joint capacity, and the remaining motion space to generate an initial candidate task allocation set based on the priority ranking results. The generation process includes: selecting the robotic arm capable of completing the task and with the most balanced load for each high-priority task; for tasks with multiple robotic arms, the system can simultaneously generate multiple candidate allocation schemes and record the feasibility score and potential interference risk of each candidate scheme for further optimization by the subsequent three-sub-problem collaborative optimization module. The generated initial candidate task allocation set ensures both task coverage and balances multi-robotic arm collaboration and safety constraints, providing structured starting point data for subsequent system scheduling and path planning.

[0100] Through the above processing steps, the system can comprehensively evaluate the executability of assembly tasks based on collaborative representation vectors, scientifically determine task priorities, and generate reasonable initial candidate task allocation schemes in combination with the real-time status of the robotic arm, thereby achieving efficient, safe, and balanced task scheduling in complex multi-arm flexible assembly environments.

[0101] The trajectory generation module 203 is used to input the collaborative representation vector and the initial candidate task allocation set into the reinforcement learning-based three-sub-problem collaborative optimizer, and to perform joint optimization in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. It also uses a dynamic cost weight adaptive mechanism to make real-time trade-offs between assembly accuracy and collision avoidance safety, and generates a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet the high-dimensional degree of freedom constraints.

[0102] In some embodiments, the trajectory generation module 203 can take the collaborative representation vector and the initial candidate task allocation set as input and feed them into a reinforcement learning-based three-sub-problem collaborative optimizer to jointly solve the problems at three levels: task allocation optimization, task scheduling optimization, and multi-manipulator motion planning optimization. Simultaneously, through a dynamic cost weight adaptive mechanism, it adjusts the balance between assembly accuracy and collision avoidance safety in real time according to the current operational risk and assembly accuracy requirements, ultimately generating a multi-manipulator collaborative timetable and corresponding time-parameterized motion trajectories that satisfy the high-degree-of-freedom motion constraints and temporal constraints of the multi-manipulator.

[0103] In some embodiments, the trajectory generation module 203 can also be used for:

[0104] Based on the collaborative representation vector, the coupling constraints between multiple robotic arms are modeled to obtain the collaborative constraint state of the multiple robotic arms;

[0105] Based on the multi-robotic arm collaborative constraint state, the initial candidate task allocation set is coordinated and corrected to obtain an intermediate task allocation scheme.

[0106] Based on the intermediate task allocation scheme, input the three subproblem co-optimizer to perform joint optimization of task allocation and task scheduling, and obtain the jointly optimized task allocation and scheduling results.

[0107] First, the system models the coupling constraints between multiple robotic arms based on the collaborative representation vector generated by the task allocation module 202, thereby obtaining the collaborative constraint state of the multiple robotic arms. Specifically, the system comprehensively considers the spatial interference relationships between robotic arms, the time dependency constraints between tasks, the load distribution limitations, and the kinematic redundancy constraints, and quantifies the collaborative state through the weights of the edges between nodes and the constraint function. During the modeling process, the system can normalize the joint reachability range of each robotic arm, the end effector posture limitations, and the potential collision risks to a unified dimension, so as to perform weighted calculations and dynamic evaluations in multi-robotic arm collaboration. The output of the collaborative constraint state can accurately reflect the possible interference, force conflicts, and motion limitations between the robotic arms in the current assembly scenario, providing a constraint basis for task allocation.

[0108] Secondly, the system utilizes the collaborative constraint state of multiple robotic arms to coordinate and correct the initial candidate task allocation set, generating intermediate task allocation schemes. During the correction process, the system performs constraint verification on each candidate task allocation scheme, including spatial reachability checks, robotic arm load balancing assessments, potential collision probability calculations, and task priority matching, and performs local adjustments for conflicting allocation schemes. Adjustment strategies include task swapping, task fragmentation, time delays, or redundant alternative allocations to ensure that the intermediate task allocation schemes retain the original priority order and task coverage as much as possible while satisfying the robotic arm collaborative constraints.

[0109] Next, the system inputs the generated intermediate task allocation scheme into a three-sub-problem collaborative optimizer for joint optimization of task allocation and scheduling. Based on a multi-objective cost function, the optimizer incorporates task completion time, collision risk, trajectory smoothness, load balancing, and force control reachability into a unified evaluation index, dynamically adjusting the weights of each cost to adapt to different scenarios and task requirements. During the optimization process, the system employs a hierarchical strategy: the high-level layer performs task allocation optimization, the middle layer generates and smooths time-parameterized trajectories, and the low-level layer performs force simulation and constraint verification. Through iterative optimization and candidate scheme selection, the final jointly optimized task allocation and scheduling results are obtained, achieving efficient, balanced, and safe assembly task execution sequence and time-parameterized trajectory arrangement among multiple robotic arms.

[0110] Through the above processing steps, the system can comprehensively characterize the collaborative constraints of multiple robotic arms based on the collaborative representation vector, reasonably correct the candidate task allocation scheme, and obtain efficient, executable and safe multi-arm task allocation and scheduling results through joint optimization, thereby significantly improving the efficiency of the production line and the assembly quality in complex flexible assembly environments.

[0111] The task allocation cost function can be expressed as:

[0112]

[0113] in, It is the comprehensive cost function value for multi-robotic arm task allocation. It is the normalized value of the expected delay in task completion. It is the normalized value of the collision risk assessment. It is the normalized value of the trajectory jitter energy. It is the normalized value of the load balancing difference. These are the first dynamic cost weight, the second dynamic cost weight, the third dynamic cost weight, and the fourth dynamic cost weight, respectively, and satisfy the following conditions: .

[0114] Specifically, This is a normalized value representing the expected delay in task completion, used to reflect the merits of candidate solutions in terms of time efficiency. The normalized value for collision risk assessment is calculated by analyzing the trajectory of each robotic arm, the position of obstacles, and the possibility of potential interference. This is a normalized value for trajectory jitter energy, used to measure the trajectory smoothness and dynamic stability during the execution of the robotic arm; This is a normalized value for the load balancing difference, used to describe the deviation between the current load state of each robotic arm and the ideal balanced load.

[0115] Secondly, the system dynamically adjusts the weights of each cost item based on different scenarios, task requirements, and risk assessment results. Each weight represents the importance of task timeliness, collision safety, trajectory smoothing, and load balancing in the current environment, and satisfies normalization constraints. The above weights can be adaptively adjusted through offline experience setting, online learning, or reinforcement learning strategies, so that the cost function can balance efficiency, accuracy, and safety requirements in different assembly tasks and multi-robotic arm collaboration scenarios.

[0116] Furthermore, during the cost function calculation, each normalized value is normalized using upper and lower bounds obtained from historical statistics or digital twin simulations to ensure that different physical quantities have the same dimensions and maintain numerical stability when participating in the weighted summation operation. The system can iteratively calculate for each candidate task allocation scheme. This serves as the key basis for the three-subproblem collaborative optimizer to select and rank task allocation and scheduling schemes.

[0117] In this way, the task allocation cost function can not only comprehensively and quantitatively evaluate the candidate task schemes of multiple robotic arms, but also adaptively adjust in real time according to the dynamic environment, task requirements and robotic arm status, thereby effectively improving the executability, collaborative efficiency and assembly safety of task allocation.

[0118] In some embodiments, the trajectory generation module 203 can also be used for:

[0119] Based on the jointly optimized task allocation and scheduling results, a multi-robotic arm motion constraint space is constructed to obtain a set of motion constraints.

[0120] Based on the set of motion constraints and the assembly accuracy requirements, the initial motion path of the multi-robotic arm is initially planned to obtain the initial motion path;

[0121] Based on the initial motion path, time parameterization is performed to generate a time parameterized motion trajectory.

[0122] First, based on the jointly optimized task allocation and scheduling results, the system performs constraint analysis on the motion capabilities, joint degrees of freedom, and workspace of each robotic arm, constructing a multi-robotic arm motion constraint space. This motion constraint space includes, but is not limited to, the joint angle range of the robotic arms, joint velocity and acceleration limits, end-effector pose accessibility, spatial interference constraints between robotic arms, and collision avoidance constraints with environmental obstacles. By uniformly modeling these constraints, the system obtains a complete set of motion constraints to limit the possible motion range of the multiple robotic arms during task execution, ensuring that the planning results both meet task execution requirements and avoid potential collisions or interference.

[0123] Secondly, based on the set of motion constraints and considering assembly accuracy and force feedback requirements, the system performs initial planning of the motion paths for multiple robotic arms. The initial planning employs segmented path generation, spline interpolation, and local optimization strategies to check the continuity, smoothness, and trajectory reachability of each robotic arm path, while also considering the pose accuracy constraints of key assembly points. The initial motion paths not only reflect the spatial trajectory of each robotic arm from its starting position to the target task point but also embed information on obstacle avoidance, redundant degree-of-freedom optimization, and end-effector pose correction, providing a foundation for subsequent time parameterization and control law execution.

[0124] Next, the system performs time parameterization processing on the initial motion path to generate a time parameterized motion trajectory. This time parameterization includes velocity and acceleration allocation at path points, trajectory smoothing filtering, and time synchronization of key nodes. This ensures that the robot arm joint speed, acceleration, and force control response meet design constraints during execution, while also guaranteeing the synchronization and accuracy of multi-arm collaboration. The time parameterized motion trajectory serves as the input to the execution layer controller and can be directly sent to the vision-force fusion control module, enabling high-precision, multi-robot collaborative motion control during flexible assembly.

[0125] Through the above methods, the system can transform the jointly optimized task allocation and scheduling results into executable motion constraints, initial paths, and time-parameterized trajectories, enabling high-precision, collaborative, and safe execution of multiple robotic arms in complex working environments.

[0126] The instruction generation module 204 is used to use the time-parameterized motion trajectory as a reference input, integrate the real-time acquired visual pose confidence information and force contact estimation information, perform visual-force deep fusion control, and generate compliant assembly control instructions for driving each robotic arm actuator by adaptively adjusting the pose weight and impedance control parameters.

[0127] In some embodiments, the instruction generation module 204 uses the generated time-parameterized motion trajectory as a reference input, and simultaneously integrates real-time acquired visual pose confidence information and force contact estimation information to perform visual-force deep fusion control. Based on the confidence levels of different sensor information, the pose fusion weights and impedance control parameters are adaptively adjusted to ensure the robotic arm exhibits compliance when approaching and contacting the workpiece, and high trajectory tracking accuracy during free-space movement. Finally, compliant assembly control instructions are generated to drive each robotic arm actuator.

[0128] In some embodiments, the instruction generation module 204 can be used to:

[0129] Based on the time-parameterized motion trajectory, the desired pose information of each robotic arm at the current moment is obtained, and the desired pose sequence is obtained.

[0130] Based on the real-time collected visual pose confidence information, the expected pose sequence is corrected to obtain the visually corrected pose.

[0131] The fused pose control quantity is obtained by fusing visually corrected pose and force contact estimation information.

[0132] First, the system acquires the desired pose information of each robotic arm in real time based on the time-parameterized motion trajectory, including the position vector, attitude angle vector, and velocity information of the end effector. The system then generates a desired pose sequence from these desired poses in chronological order, which describes the target spatial trajectory and motion requirements of the multiple robotic arms at various moments throughout the assembly task. This desired pose sequence not only considers the joint constraints of the robotic arms and the multi-arm collaboration constraints, but also includes the accuracy requirements of key assembly nodes, providing a reference benchmark for subsequent vision and force fusion.

[0133] Secondly, the system performs confidence correction on the desired pose sequence based on real-time acquired visual information. Visual pose confidence information comes from multimodal visual sensors, such as structured light, stereo cameras, or Time-of-Flight (TOF) sensors, and is generated by combining feature matching scoring, point cloud consistency detection, and occlusion analysis. The system performs weighted correction of the desired pose according to visual confidence: when visual confidence is high, the visually measured pose is used as a reference; when visual confidence decreases or occlusion exists, the visual weight is appropriately reduced, and compensation is made by combining historical pose and force information, thereby obtaining a visually corrected pose that ensures the target pose remains reasonable and continuous even when visual information is limited.

[0134] Next, the system fuses the visually corrected pose and force-contact estimation information to obtain the fused pose control quantity. The force-contact estimation information is acquired through force / torque sensors or tactile arrays at the robotic arm's end effector, including contact point distribution, contact force magnitude, and stiffness estimation. The system employs an adaptive weighted fusion strategy, dynamically weighting the displacement information from the visually corrected pose and force-contact estimation to form the final fused pose control quantity. This fused pose control quantity can be directly used as input to a compliant / impedance controller to adjust the robotic arm's motion trajectory and end-effector impedance parameters, achieving deep vision-force fusion control and ensuring assembly accuracy and safety under micro-offsets, contact force variations, or environmental disturbances.

[0135] In this way, the system can closely integrate time-parameterized trajectories with real-time sensing data, enabling high-precision and compliant assembly operations of multiple robotic arms in complex working environments, while providing reliable control inputs for subsequent dynamic collision avoidance, task adjustment, and collaborative control.

[0136] In some embodiments, the instruction generation module 204 can be used to:

[0137] Based on the fused pose control variables, the current assembly contact state is calculated, and the assembly contact state parameters are obtained.

[0138] Based on the assembly contact state parameters, the impedance control parameters are adaptively adjusted to obtain the corrected impedance parameters;

[0139] Based on the corrected impedance parameters, compliant assembly control commands are generated.

[0140] First, the system calculates the contact state between the robotic arm's end effector and the workpiece in real time based on the fused pose control parameters, obtaining assembly contact state parameters. These parameters include, but are not limited to, the magnitude of the end effector force / torque, the distribution of contact points, contact stiffness, and relative displacement error. The system then filters, integrates, and compares the data collected by the force sensors with a reference pose to achieve a high-precision estimate of the contact state, thereby determining whether the contact is stable, whether the force is uniform, and the possible direction of deviation during the current assembly process.

[0141] Secondly, the system adaptively adjusts the impedance control parameters of the robotic arm based on the assembly contact state parameters to cope with mechanical disturbances and uncertainties during the assembly process. Specifically, the system dynamically adjusts the end effector stiffness, damping, and inertia parameters based on the deviation between the contact force and the desired force value, the uniformity of the contact point distribution, and displacement errors, achieving rigid-flexible coupling control. For example, when excessive local force or deviation from the desired direction is detected, the system reduces the stiffness parameter in that direction and increases damping to buffer the impact; when the contact force is insufficient or there is a risk of loosening, the system appropriately increases the stiffness to ensure assembly accuracy and stability, thereby obtaining corrected impedance parameters.

[0142] Furthermore, the system generates compliant assembly control commands based on the corrected impedance parameters to drive the robotic arm to perform assembly actions. These compliant assembly control commands include multi-dimensional control quantities, such as end-effector position adjustment, attitude correction, speed control, and force / torque compensation, enabling continuous, smooth, and controllable adjustment of the robotic arm's movements. By executing these control commands, the robotic arm can adaptively respond to external disturbances, mechanical changes, and environmental uncertainties during the assembly process, achieving high-precision, low-impact compliant assembly operations.

[0143] Through the above methods, this solution can sense the contact state between the robotic arm and the workpiece in real time and dynamically adjust the impedance control parameters, so that the compliant assembly process takes into account accuracy, safety and efficiency, and provides a stable and reliable execution guarantee for complex multi-robotic arm collaborative assembly tasks.

[0144] Among them, the confidence score of visual-mechanical fusion and the fusion pose weight can be expressed as:

[0145]

[0146] in, It is the normalized value of visual pose confidence. It is the normalized value of the confidence level of force contact. These are the first priority coefficient and the second priority coefficient, respectively. It is a normalized representation of the visual pose estimate. It is the visual pose weight. It is force-sensory pose weight.

[0147] Specifically, visual-force fusion is achieved using a weighted confidence strategy. Let the normalized value of the visual pose confidence be... The normalized value of the force contact confidence score is The first priority coefficient is The second priority coefficient is The system follows the formula. The fusion weights are calculated, exponentially smoothed, and then subjected to amplitude processing with preset upper and lower limits to obtain the final weights. The fusion pose calculation is performed according to... Implementation, in which , These are the normalized representations of visual pose estimation and force-sensory contact displacement, respectively. The fused pose, after inverse normalization, serves as the pose control component of the compliant assembly control command, which, together with real-time impedance parameters, is used in an impedance control law to achieve compliant assembly. If the confidence level of either the visual or force perception drops abnormally, the system switches the control channel according to pre-set fault-tolerant logic and triggers trajectory replanning or operation deceleration to ensure safety.

[0148] The adaptive impedance parameter scheduling can be expressed as:

[0149]

[0150] in, It is a real-time impedance-controlled stiffness parameter. It is the normalized value of the reference impedance stiffness. It is the normalized value of the system risk index (combining collision risk, EMI, belt speed fluctuation, etc.). It is the normalized value of the workpiece assembly tolerance (smaller tolerance → larger value, requiring higher rigidity for precise positioning). These are the risk suppression weighting coefficient and the assembly accuracy enhancement weighting coefficient, respectively.

[0151] Specifically, the system comprehensive risk index is obtained by weighting and normalizing sub-items such as collision risk, EMI index and belt speed fluctuation; assembly tolerance is mapped to normalized value from process specifications or CAD data (the smaller the tolerance, the larger the normalized value). After calculation, the data is limited and low-pass filtered, then mapped to the actual stiffness scale of the actuator, and finally sent to the impedance controller to form compliant assembly control commands together with the fused pose reference. Weighting coefficients In a digital twin environment, reinforcement learning strategies can be used to adaptively optimize the system to balance security and accuracy.

[0152] The signal output module 205 is used to input the execution status data of the compliant assembly control command and the real-time scene change data into the digital twin simulation module, perform online dynamic scene adaptation and control parameter reverse optimization in the digital twin environment, and output the corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning.

[0153] In some embodiments, the signal output module 205 synchronously inputs the execution status data of the compliant assembly control commands and real-time scene change data to the digital twin simulation module, performing online dynamic mapping and scene adaptation of the actual assembly process in the constructed digital twin environment. Based on the simulation feedback results, the current impedance control parameters are reverse-corrected and optimized, and the corrected impedance parameter set is output. Simultaneously, when a significant deviation between the virtual and real execution results is detected, a replanning trigger signal is generated to trigger trajectory replanning, ensuring the safety and stability of the assembly process.

[0154] In some embodiments, the signal output module 205 can also be used for:

[0155] Based on the execution status data of the compliant assembly control commands, a virtual execution status model is constructed to obtain virtual execution feedback;

[0156] Based on virtual execution feedback and real-time scene change data, a consistency comparison is performed in the digital twin environment to obtain deviation detection results;

[0157] Based on the deviation detection results, the control parameters are corrected in reverse, and the corrected impedance parameter set and reprogramming trigger signal are output.

[0158] First, the system constructs a virtual execution state model based on the execution state data of the compliant assembly control commands. This model simulates the robot arm's motion trajectory, end-effector force / torque response, and contact state changes under the current control commands, thereby obtaining virtual execution feedback. The virtual execution state model includes the angle, velocity, and acceleration information of each joint of the robot arm, as well as the pose, force response, and assembly contact state parameters of the end-effector, generated through numerical simulation or physics-based modeling methods. The virtual execution feedback reflects the expected performance of the robot arm's actions under ideal execution conditions, providing a reference benchmark for subsequent comparative analysis.

[0159] Secondly, the system compares the virtual execution feedback with the real-time scene change data. Real-time scene change data includes changes in workpiece position and posture, changes in environmental obstacles, the motion states of other robotic arms, and the latest data collected by vision / force sensors. The system calculates deviation indices, including positional deviation, posture error, contact force differences, and potential collision risks, by comparing the virtual model output with the actual feedback data from multiple dimensions, thus obtaining deviation detection results. These deviation detection results are used to quantify the degree of consistency between the actual and ideal execution states of the robotic arm.

[0160] Furthermore, based on the deviation detection results, the system performs reverse correction on the compliant assembly control parameters to optimize the dynamic response performance of the assembly process. Specifically, when the deviation detection results exceed a preset threshold, the system adjusts the end effector stiffness, damping, and force / torque compensation parameters, outputting a corrected set of impedance parameters. Simultaneously, if a potential infeasibility risk is detected in the assembly path or task execution, the system triggers a replanning signal to initiate the replanning of the multi-robotic arm motion trajectory or the adjustment of task scheduling, achieving closed-loop optimization.

[0161] Through the above methods, this solution can realize virtual-actual closed-loop monitoring of the compliant assembly process of robotic arms, and perform real-time detection and dynamic correction of deviations. This not only ensures assembly accuracy and safety, but also enhances the robustness and adaptability of multi-robotic arm collaborative operations, providing reliable intelligent execution support for complex assembly tasks.

[0162] The fault-tolerant control module 206 is used to initiate a fault-tolerant switching mechanism based on a reinforcement learning strategy when a replanning trigger signal is received or a fault state of any robotic arm actuator is detected, and to perform dynamic redistribution or anti-interference control of multi-robotic arm tasks.

[0163] In some embodiments, when the fault-tolerant control module 206 receives a replanning trigger signal from the signal output module, or detects a fault state such as failure, performance degradation, or communication anomaly in any robotic arm actuator, it automatically activates the fault-tolerant switching mechanism based on a pre-trained reinforcement learning strategy. By assessing the capabilities of the remaining robotic arms and analyzing task feasibility, it performs dynamic reallocation of tasks across multiple robotic arms, or enters an anti-interference control mode to correct the current motion, thereby effectively preventing the propagation of system failures while ensuring operational continuity.

[0164] In some embodiments, the fault-tolerant control module 206 can also be used for:

[0165] Based on the replanning trigger signal or fault status information, the fault type is determined and fault characteristic information is obtained;

[0166] Based on the fault characteristic information, a risk assessment is performed on the current multi-robotic arm task execution status to obtain the task risk level;

[0167] Based on the task risk level, appropriate fault-tolerant control actions are selected through reinforcement learning strategies to achieve dynamic redistribution or anti-interference control of multi-robotic arm tasks.

[0168] First, the system identifies the abnormal events or fault types based on the replanning trigger signal or real-time acquired fault status information, obtaining fault characteristic information. Fault characteristic information includes, but is not limited to, robotic arm joint drive failure, end effector failure, excessive path deviation, increased collision risk, or sensor malfunction. Through comprehensive analysis of sensor data, control command execution results, and virtual execution feedback, the system identifies abnormal patterns and extracts key fault parameters, such as fault occurrence time, location, impact range, and duration, thereby forming structured fault characteristic information that can be used for subsequent processing.

[0169] Secondly, based on fault characteristic information, the system performs a risk assessment on the current multi-robotic arm task execution status to obtain the task risk level. The risk assessment includes evaluating the impact of faults on the assembly task completion time, path accuracy, collision probability, and multi-robotic arm collaborative stability, and quantifies the assessment results to form a classification of low risk, medium risk, and high risk. By determining the task risk level, the system can identify potential hazards that may lead to assembly failure or safety accidents, providing a basis for dynamic control strategies.

[0170] Furthermore, based on the task risk level, the system selects appropriate fault-tolerant control actions through a reinforcement learning strategy to achieve dynamic task redistribution or anti-interference control for multiple robotic arms. The reinforcement learning strategy, based on historical task execution experience, current task status, and risk assessment results, outputs adaptive control decisions, including adjusting task allocation priorities, replanning motion trajectories, modifying impedance control parameters, or initiating local collaborative interventions. Through fault-tolerant control actions, multiple robotic arms can maintain task continuity and operational safety under abnormal or faulty conditions, achieving intelligent, real-time dynamic task scheduling and collaborative control.

[0171] Through the above methods, this solution can achieve adaptive fault identification, task risk assessment, and dynamic fault-tolerant control in the multi-robotic arm assembly process. This not only improves the reliability and safety of the assembly system, but also enhances the robustness and flexibility of multi-robotic arm collaborative operation, providing stable intelligent execution support for complex assembly tasks.

[0172] Please see Figure 4 The present invention provides a flowchart of a vision-force fusion control and fault-tolerant method for multi-robotic arms in flexible assembly, comprising the following steps:

[0173] Step 301: Collect the 3D point cloud data of the workpiece, the contact force information of the robotic arm end effector, and the status information of the working environment. Normalize the collected data to obtain normalized multi-source information, and construct a heterogeneous scene graph model based on the normalized multi-source information.

[0174] Step 302: The task allocation module is used to input the heterogeneous scene graph model into the graph neural network for cross-node feature aggregation and relation reasoning, generate collaborative representation vectors, and output the task priority ranking results and the initial candidate task allocation set based on the collaborative representation vectors.

[0175] Step 303: The trajectory generation module is used to input the collaborative representation vector and the initial candidate task allocation set into the reinforcement learning-based three-sub-problem collaborative optimizer, and to perform joint optimization in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. Through a dynamic cost weight adaptive mechanism, it makes a real-time trade-off between assembly accuracy and collision avoidance safety, and generates a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet the high-dimensional degree of freedom constraints.

[0176] Step 304: The instruction generation module is used to use the time-parameterized motion trajectory as a reference input, integrate the real-time acquired visual pose confidence information and force contact estimation information, perform visual-force deep fusion control, and generate compliant assembly control instructions for driving each robotic arm actuator by adaptively adjusting the pose weight and impedance control parameters.

[0177] Step 305: The signal output module is used to input the execution status data of the compliant assembly control command and the real-time scene change data into the digital twin simulation module, perform online dynamic scene adaptation and control parameter reverse optimization in the digital twin environment, and output the corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning.

[0178] Step 306: Fault-tolerant control module, used to activate fault-tolerant switching mechanism based on reinforcement learning strategy when a replanning trigger signal is received or a fault state of any robotic arm actuator is detected, to perform dynamic redistribution or anti-interference control of multi-robotic arm tasks.

[0179] Please see Figure 4 , Figure 5 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 5 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it performs the following steps:

[0180] The system collects 3D point cloud data of the workpiece, contact force information of the robotic arm end effector, and status information of the working environment. The collected data is normalized to obtain normalized multi-source information, and a heterogeneous scene graph model is constructed based on the normalized multi-source information.

[0181] The heterogeneous scene graph model is input into the graph neural network for cross-node feature aggregation and relation reasoning to generate a collaborative representation vector. Based on the collaborative representation vector, the task priority ranking result and the initial candidate task allocation set are output. Among them, the collaborative representation vector represents the cross-arm interference risk, path reachability and load capacity between multiple robotic arms.

[0182] The collaborative representation vector and the initial candidate task allocation set are input into a reinforcement learning-based three-sub-problem collaborative optimizer. Joint optimization is performed in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. A dynamic cost weight adaptive mechanism is used to make a real-time trade-off between assembly accuracy and collision avoidance safety, generating a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet high-dimensional freedom constraints.

[0183] Using time-parameterized motion trajectory as reference input, the system integrates real-time acquired visual pose confidence information and force contact estimation information to perform visual-force deep fusion control. By adaptively adjusting pose weights and impedance control parameters, it generates compliant assembly control commands to drive each robotic arm actuator.

[0184] The execution status data of the compliant assembly control command and the real-time scene change data are input into the digital twin simulation module. Online dynamic scene adaptation and control parameter reverse optimization are performed in the digital twin environment. The corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning are output.

[0185] When a replanning trigger signal is received or a fault condition is detected in any robotic arm actuator, a fault-tolerant switching mechanism is initiated based on a reinforcement learning strategy to perform dynamic redistribution of tasks among multiple robotic arms or anti-interference control.

[0186] Please see Figure 5 , Figure 1 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. For example... Figure 1 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 411 is stored. When the computer program 411 is executed by a processor, it performs the following steps:

[0187] The system collects 3D point cloud data of the workpiece, contact force information of the robotic arm end effector, and status information of the working environment. The collected data is normalized to obtain normalized multi-source information, and a heterogeneous scene graph model is constructed based on the normalized multi-source information.

[0188] The task allocation module is used to input heterogeneous scene graph models into graph neural networks for cross-node feature aggregation and relation reasoning, generate collaborative representation vectors, and output task priority ranking results and initial candidate task allocation sets based on collaborative representation vectors; wherein, the collaborative representation vectors represent the cross-arm interference risk, path reachability and load capacity between multiple robotic arms.

[0189] The trajectory generation module is used to input the collaborative representation vector and the initial candidate task allocation set into the reinforcement learning-based three-sub-problem collaborative optimizer, and perform joint optimization in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. It also uses a dynamic cost weight adaptive mechanism to make real-time trade-offs between assembly accuracy and collision avoidance safety, and generates a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet the high-dimensional degree of freedom constraints.

[0190] The instruction generation module is used to take the time-parameterized motion trajectory as a reference input, integrate the real-time acquired visual pose confidence information and force contact estimation information, execute visual-force deep fusion control, and generate compliant assembly control instructions to drive each robotic arm actuator by adaptively adjusting the pose weight and impedance control parameters.

[0191] The signal output module is used to input the execution status data of the compliant assembly control command and the real-time scene change data into the digital twin simulation module, perform online dynamic scene adaptation and control parameter reverse optimization in the digital twin environment, and output the corrected impedance parameter set and the replanning trigger signal used to trigger trajectory replanning.

[0192] The fault-tolerant control module is used to initiate a fault-tolerant switching mechanism based on a reinforcement learning strategy when a replanning trigger signal is received or a fault state of any robotic arm actuator is detected, and to perform dynamic redistribution or anti-interference control of multi-robotic arm tasks.

[0193] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0194] Those skilled in the art will understand that embodiments of the present invention can be provided as systems, methods, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0196] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0198] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0199] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly, characterized in that, The system includes: The data construction module is used to collect three-dimensional point cloud data of the workpiece, contact force information of the robotic arm end effector, and working environment status information. The collected data is normalized to obtain normalized multi-source information, and a heterogeneous scene graph model is constructed based on the normalized multi-source information. The task allocation module is used to input the heterogeneous scene graph model into the graph neural network for cross-node feature aggregation and relation reasoning, generate a collaborative representation vector, and output the task priority ranking result and the initial candidate task allocation set based on the collaborative representation vector; wherein, the collaborative representation vector represents the cross-arm interference risk, path reachability and load capacity between multiple robotic arms; The trajectory generation module is used to input the collaborative representation vector and the initial candidate task allocation set into a reinforcement learning-based three-sub-problem collaborative optimizer, and to perform joint optimization in three dimensions: task allocation, task scheduling and multi-manipulator motion planning. It also uses a dynamic cost weight adaptive mechanism to make real-time trade-offs between assembly accuracy and collision avoidance safety, and to generate a multi-manipulator collaborative timetable and corresponding time parameterized motion trajectory that meet high-dimensional freedom constraints. The instruction generation module is used to take the time-parameterized motion trajectory as a reference input, fuse the real-time acquired visual pose confidence information and force contact estimation information, perform visual-force deep fusion control, and generate compliant assembly control instructions for driving each robotic arm actuator by adaptively adjusting the pose weight and impedance control parameters. The signal output module is used to input the execution status data of the compliant assembly control command and the real-time scene change data into the digital twin simulation module, perform online dynamic scene adaptation and control parameter reverse optimization in the digital twin environment, and output the corrected impedance parameter set and the replanning trigger signal for triggering trajectory replanning. The fault-tolerant control module is used to initiate a fault-tolerant switching mechanism based on a reinforcement learning strategy when it receives the replanning trigger signal or detects a fault state in any robotic arm actuator, and to perform dynamic redistribution or anti-interference control of multi-robotic arm tasks.

2. The multi-robotic arm vision-force fusion control and fault-tolerant system for flexible assembly according to claim 1, characterized in that, The data construction module is also used for: Based on the three-dimensional point cloud data, the surface of the workpiece is segmented and features are extracted to obtain a set of spatial geometric features of the workpiece; Based on the set of spatial geometric features of the workpiece and the working environment state information, an environmental topology structure containing spatial constraint relationships is constructed. Based on the environmental topology and the contact force information at the end of the robotic arm, multiple types of nodes and the constraint relationships between nodes are established to obtain the heterogeneous scene graph model.

3. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly according to claim 2, characterized in that, The task allocation module is also used for: Based on the heterogeneous scene graph model, initial feature encoding is performed on various types of nodes to obtain a set of node feature vectors; Based on the set of node feature vectors, the collaborative relationship features of multiple robotic arms are extracted through cross-node information propagation and aggregation to obtain the intermediate collaborative feature representation. Relationship reasoning is performed based on the intermediate collaborative feature representation to generate the collaborative representation vector.

4. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly according to claim 3, characterized in that, The task allocation module is also used for: Based on the collaborative representation vector, the executability of each assembly task is evaluated to obtain the task feasibility evaluation result; Based on the task feasibility assessment results, the assembly tasks are prioritized to obtain the task priority ranking results. Based on the task priority ranking results and the current load status of each robotic arm, the initial candidate task allocation set is generated.

5. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly according to claim 4, characterized in that, The trajectory generation module is also used for: Based on the cooperative representation vector, the coupling constraints between multiple robotic arms are modeled to obtain the cooperative constraint state of the multiple robotic arms; Based on the multi-robotic arm collaborative constraint state, the initial candidate task allocation set is coordinated and corrected to obtain an intermediate task allocation scheme; Based on the intermediate task allocation scheme, the three-sub-problem collaborative optimizer is input to perform joint optimization of task allocation and task scheduling, and the joint optimized task allocation and scheduling results are obtained.

6. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly according to claim 5, characterized in that, The trajectory generation module is also used for: Based on the jointly optimized task allocation and scheduling results, a multi-robotic arm motion constraint space is constructed to obtain a set of motion constraints. Based on the set of motion constraints and the assembly accuracy requirements, the motion path of the multiple robotic arms is initially planned to obtain the initial motion path. Based on the initial motion path, time parameterization is performed to generate the time parameterized motion trajectory.

7. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly according to claim 6, characterized in that, The instruction generation module is also used for: Based on the time-parameterized motion trajectory, the desired pose information of each robotic arm at the current moment is obtained, and the desired pose sequence is obtained. Based on the real-time collected visual pose confidence information, the expected pose sequence is corrected to obtain the visually corrected pose. The fused pose control quantity is obtained by fusing the visually corrected pose and the force contact estimation information.

8. The multi-robotic arm vision-force fusion control and fault-tolerant system for flexible assembly according to claim 7, characterized in that, The instruction generation module is also used for: Based on the fused pose control quantity, the current assembly contact state is calculated, and the assembly contact state parameters are obtained; Based on the assembly contact state parameters, the impedance control parameters are adaptively adjusted to obtain the corrected impedance parameters; The compliant assembly control command is generated based on the modified impedance parameter.

9. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly as described in claim 8, characterized in that, The signal output module is also used for: Based on the execution status data of the compliant assembly control command, a virtual execution status model is constructed to obtain virtual execution feedback; Based on the virtual execution feedback and the real-time scene change data, a consistency comparison is performed in the digital twin environment to obtain the deviation detection result; Based on the deviation detection results, the control parameters are corrected in reverse, and the corrected impedance parameter set and the replanning trigger signal are output.

10. The vision-force fusion control and fault-tolerant system for multi-robotic arms in flexible assembly as described in claim 9, characterized in that, The fault-tolerant control module is also used for: Based on the replanning trigger signal or fault status information, the fault type is determined to obtain fault characteristic information; Based on the fault characteristic information, a risk assessment is performed on the current multi-robotic arm task execution status to obtain the task risk level; Based on the task risk level, a corresponding fault-tolerant control action is selected through a reinforcement learning strategy to complete the dynamic redistribution or anti-interference control of multi-robotic arm tasks.

Images