A Collaborative Control Method and System for Coal and Gangue Sorting Using Multi-Robot Arms Based on VLM Prior Constraint Reinforcement Learning

Abstract

This invention discloses a collaborative control method and system for multiple robotic arms in coal gangue sorting based on reinforcement learning with prior constraints using Virtual Model (VLM). In a virtual simulation environment, the system automatically constructs a multimodal cognitive dataset containing multi-view images, motion states, and collision feedback using the sweep volume collision detection function of a physics engine. This dataset is then serialized into CoT text, guiding VLM to extract spatial geometric constraints based on CoT logical reasoning, transforming them into a task allocation topology matrix and a set of high-risk interference regions. During the reinforcement learning training phase, the system uses the task allocation topology matrix as an action masking mechanism to block illegally assigned actions and transforms the set of high-risk interference regions into a dynamic repulsive potential field penalty term integrated into the comprehensive reward function, guiding the agent to learn collaborative obstacle avoidance. Finally, a lightweight policy network with fixed semantic priors is deployed on an edge controller, outputting task allocation and motion control commands in real time based on on-site perception information, reducing ineffective exploration and potential conflicts.

Description

Technical Field

[0001] This invention belongs to the field of robot control, and in particular relates to a collaborative control method and system for coal gangue sorting using multiple robotic arms based on VLM prior constraint reinforcement learning. Background Technology

[0002] Existing intelligent coal and gangue sorting solutions typically include target identification and localization, target allocation, robotic arm trajectory planning, and sorting execution. Most publicly available technologies first acquire the location information of coal and gangue on the conveyor belt, then allocate targets to different robotic arms based on rules, distance costs, or scheduling strategies, with each arm performing the corresponding sorting action. While these solutions can improve sorting automation to some extent, they usually treat task allocation and cooperative collision avoidance as independent problems. This leads to situations where, although the static allocation result may satisfy local optima, problems such as trajectory intersections, end-effector interference, and waiting for yielding can still easily arise during dynamic execution.

[0003] For example, existing patent CN111993418B discloses a multi-target, multi-robotic arm collaborative sorting system and sorting strategy for coal gangue sorting robots. This solution mainly focuses on coal gangue identification and positioning, multi-robotic arm task scheduling, and collaborative sorting processes, which can improve processing capabilities in multi-target scenarios. However, the technical focus of this type of solution is still mainly on target allocation and execution process optimization. It has limited utilization of high-level spatial relationships under conditions such as complex occlusion, stacking and contact, and dynamic interference, making it difficult to further serve the unified generation of multi-robotic arm collaborative control strategies.

[0004] However, in actual coal and gangue sorting operations, coal and gangue on conveyor belts typically exhibit characteristics such as random distribution, stacking and obstruction, dense multi-target activity, and continuous dynamic changes. Furthermore, the working areas of different robotic arms often overlap and intersect. While task allocation based solely on target location or local cost may yield better results in a static sense, actual execution may still result in trajectory intersections, end-effector interference, and repeated yielding, thus reducing overall sorting efficiency and increasing collision risk. On the other hand, directly employing pure deep reinforcement learning for joint learning of target allocation and motion control for multiple robotic arms faces challenges such as a large state-action space, numerous ineffective exploration branches, and convergence difficulties due to sparse rewards. While existing large-scale models possess strong scene understanding capabilities, they often struggle to directly meet the real-time control requirements of industrial edge computing. Moreover, existing solutions generally lack an effective mechanism for automatically generating labels based on collision detection results from virtual simulation environments and physics engines, and for further constructing a high-quality multimodal cognitive dataset that can drive model training.

[0005] In summary, existing technologies for collaborative control of multiple robotic arms in coal gangue sorting have the following main shortcomings:

[0006] 1. It is difficult to perform unified modeling and collaborative optimization of multi-target task allocation and dynamic collision avoidance of multiple robotic arms;

[0007] 2. It is difficult to fully utilize unstructured scene semantic information such as occlusion, stacking, and dynamic interference to guide the generation of control strategies;

[0008] 3. Pure reinforcement learning methods suffer from problems such as large state-action space, many ineffective exploration branches, and difficulty in training convergence in multi-robotic arm coal and gangue sorting scenarios;

[0009] 4. While existing large-scale model methods have strong scene understanding capabilities, they are difficult to directly meet the requirements of real-time deployment and online control at the industrial edge. Summary of the Invention

[0010] Purpose of the invention: This invention provides a collaborative control method and system for multiple robotic arms in coal gangue sorting based on VLM prior constraint reinforcement learning, in order to solve the problems in the prior art, such as the difficulty in unified modeling and collaborative optimization of multi-objective task allocation and dynamic collision avoidance of multiple robotic arms, the difficulty in directly using semantic information of complex scenes to serve the generation of control strategies, the large number of invalid exploration branches and the difficulty in convergence during pure reinforcement learning training, and the difficulty in directly meeting the real-time deployment requirements of industrial edge.

[0011] Technical solution: The present invention provides a collaborative control method for multiple robotic arms in coal and gangue sorting based on VLM prior constraint reinforcement learning, comprising:

[0012] S1. Build a virtual simulation environment corresponding to the real scene, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels, and generate the optimal topology label based on the comprehensive cost.

[0013] S2. Fuse multimodal data, collision detection results, risk labels, and optimal topology labels. Generate CoT text from the fused data according to a preset template and construct a state vector.

[0014] S3. Input multimodal data and CoT text into the pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target allocation topology matrix and a set of high-risk interference regions;

[0015] S4. Construct a multi-robotic arm collaborative sorting reinforcement learning model, using the state vector as the state space, and output a joint action space containing discrete task allocation actions and continuous motion control actions based on the Actor network.

[0016] S5. Input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct a risk penalty term, and update and calculate the comprehensive reward based on the risk penalty term, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model.

[0017] S6. Use the constrained multi-robotic arm collaborative sorting reinforcement learning model to perform constrained reinforcement learning training to obtain the multi-robotic arm collaborative control strategy network.

[0018] S7. Deploy the multi-robotic arm collaborative control strategy network on the edge controller.

[0019] Furthermore, the acquisition and processing of multimodal data includes: acquiring the speed and running status of the conveyor belt, top view images and side view images of candidate targets, joint angles of multiple robotic arms, joint velocities of multiple robotic arms, and end effector poses of multiple robotic arms; inputting the top view images and side view images into a preset target detection model respectively, and extracting and recording the three-dimensional position, pose, category, and upper limit of the time window of the candidate targets.

[0020] Furthermore, the collision detection includes: within the execution time range of the candidate grasping scheme, acquiring the motion sweep space of each robotic arm end effector, link sampling point, candidate target and adjacent robotic arm, and calculating the minimum safe distance between each robotic arm and obstacles, non-candidate targets and adjacent robotic arms; when it is detected that there is volume overlap in the motion sweep space, the minimum safe distance is lower than the safe distance threshold or the motion trajectory intersects, the collision risk area is determined to be the volume overlap area or the spatial area with the point corresponding to the minimum safe distance as the center and a preset radius.

[0021] Furthermore, the extraction of risk labels includes: extracting the geometric center of the collision risk area, determining the scope of influence based on the spatial distribution of the collision risk area, and using the minimum bounding sphere radius corresponding to the geometric center of the collision risk area and the scope of influence as the risk label;

[0022] The process of generating optimal topology tags based on comprehensive cost includes: calculating the comprehensive cost of candidate crawling schemes, using the following formula:

[0023] ;

[0024] in, As candidate crawling solutions, The estimated execution time for the candidate crawling solutions. The collision risk cost of the candidate grasping scheme is calculated as the sum of the products of the risk level weight of each collision risk area and the time taken for the robotic arm to traverse that area. The timeout cost or waiting cost for candidate crawling solutions is calculated as the product of the timeout duration and the unit time penalty coefficient, or as the product of the waiting duration and the unit time penalty coefficient. The sorting efficiency benefit of the candidate grasping scheme is the target number of items successfully sorted per unit time, or the total number of items sorted within a fixed time window. , , , These are the weighting coefficients;

[0025] The candidate grasping scheme with the lowest overall cost, which satisfies the following conditions: the robotic arm can reach the workspace, the posture meets the grasping requirements, the grasping is completed before the upper limit of the grasping time window, the motion sweep space has no volume overlap, and the minimum safe distance is greater than the safety threshold, is selected as the optimal topology label.

[0026] Furthermore, the fusion includes performing unified coordinate system, unified timestamp, and unified data format operations on multimodal data, collision detection results, risk labels, and optimal topology labels in sequence to obtain fused data; the construction of the state vector includes using the three-dimensional position, attitude, category, upper limit of the time window, state of multiple robotic arms, and running state of the conveyor belt as the state vector.

[0027] Furthermore, the extraction of high-level spatial semantic information includes: based on the reasoning logic of CoT text, parsing and extracting the occlusion relationship, stacking relationship, spatial adjacency relationship, robotic arm operation area overlap relationship and potential trajectory interference relationship of candidate targets from the multimodal data, and forming a standardized high-level spatial semantic information set after structured extraction and verification.

[0028] Furthermore, the conversion into a target allocation topology matrix and a high-risk interference region set includes: transforming the standardized high-level spatial semantic information set in two ways according to semantic type: the first way generates a target allocation topology matrix based on the occlusion relationship, stacking relationship, spatial adjacency relationship, time window upper limit, robotic arm reachability, and risk assessment value of candidate targets. The risk assessment value The risk is calculated by weighting path collision risk, minimum safe distance risk, occlusion stacking risk, and time constraint risk. The formula is as follows:

[0029] ;

[0030] in, The collision detection risk is represented by the average risk level parameter of all collision risk areas involved in the pairing of the robotic arm with the candidate target. The minimum safe distance risk is represented by the quotient of the safe distance threshold divided by the actual minimum distance, obtained by normalization after taking the logarithm mapping. This represents the target occlusion or stacking risk, obtained by weighting the sum of the number of occlusion sources and the number of stacking layers, and then normalizing by dividing by the sum of the maximum number of observed occlusion sources and the maximum number of stacking layers. This indicates a tight time window and is the ratio of the estimated execution time to the upper limit of the capture time window. These are the weighting coefficients;

[0031] The target allocation topology matrix Recorded as:

[0032] ;

[0033] in, Indicates the number of robotic arms. Indicates the number of candidate targets. Indicates the first The robotic arm and the first Whether a valid pairing is allowed between candidate targets satisfies the following formula:

[0034] ;

[0035] Indicates the first One candidate target, Indicates the first The reachable workspace of a robotic arm Indicates the first The robotic arm reached the first... The estimated execution time for each candidate target Indicates the first The upper limit of the capture time window for each candidate target. Indicates the first Risk assessment value for each robotic arm performing a corresponding grasping action. Indicates the risk threshold;

[0036] The second approach identifies areas with difficult detours based on the overlapping relationships of the robotic arm's operating areas and potential trajectory interference. It then generates a set of high-risk interference areas, considering collision risk areas, areas with difficult detours, historical risk samples, and areas with high incidence of cross-movement. The historical risk samples and high-incidence areas of cross-movement are the spatial range in which the frequency of multiple robotic arm trajectories crossing or passing close to each other exceeds a preset statistical threshold in a preset round of simulation.

[0037] The set of high-risk interference areas Recorded as:

[0038] ;

[0039] in, Indicates the first The three-dimensional center coordinates of a high-risk interference region Indicates the first Parameters regarding the impact range of high-risk interference areas. Indicates the first Risk level parameters for high-risk interference areas This indicates the number of high-risk intervention areas.

[0040] Furthermore, the action mask constraint for the discrete task allocation action includes: assuming the preference value vector of the discrete task allocation action is... The formula is:

[0041] ;

[0042] in, This represents the task assignment branch in the Actor network used to output discrete task assignment action preference values. For network parameters, This is the current state;

[0043] Assign a topology matrix according to the target. Constructing a mask penalty term The formula is:

[0044] ;

[0045] in, It is a pre-set large positive number;

[0046] The probability of assigning actions to discrete tasks satisfies the following formula:

[0047] ;

[0048] in, For the first Assign actions to discrete tasks.

[0049] Furthermore, the construction of risk penalty items, and the updating and calculation of the comprehensive reward based on the risk penalty items, includes:

[0050] The key points of the robotic arm are determined, and a risk penalty term is constructed based on the spatial distance between the key points of the robotic arm and the centers of each risk area in the high-risk interference area cluster. The formula is as follows:

[0051] ;

[0052] in, Indicates the first The key points of the robotic arm are relative to the first The penalty value for each risk area, Indicates the first Risk level parameters for each risk area Indicates the first Attenuation parameters for each risk region Indicates the first The impact range parameter of each risk area This is a preset proportional coefficient. Indicates the safe buffer distance;

[0053] The current state is formed by all the key points and all the risk areas of the robotic arm. The risk penalty for each moment is calculated using the following formula:

[0054] ;

[0055] Calculate the current The comprehensive reward function at time point is given by the formula:

[0056] ;

[0057] in, This indicates the successful capture reward, representing the total number of targets successfully captured by each robotic arm; Weighting for successfully capturing rewards; This represents a sorting efficiency bonus, calculated based on the number of items effectively sorted per unit time or the reciprocal of the completion time. The weighting of sorting efficiency rewards This represents the actual collision penalty, calculated using a collision indicator function or the number of collisions. The weight of the real collision penalty term, Indicates risk penalty items, The weight of the risk penalty item, This represents the delay penalty, calculated based on the target missing the grasping time window, the robotic arm's waiting time, or the idle time. This represents the weight of the delay penalty term.

[0058] The present invention discloses a multi-robotic arm collaborative control system for coal and gangue sorting based on VLM prior constraint reinforcement learning, comprising:

[0059] The simulation modeling module is used to build a virtual simulation environment corresponding to the real scene, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels, and generate the optimal topology label based on the comprehensive cost.

[0060] The data generation module is used to fuse multimodal data, collision detection results, risk labels, and optimal topology labels, generate CoT text from the fused data according to a preset template, and construct a state vector.

[0061] The visual language constraint generation module is used to input multimodal data and CoT text into a pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target assignment topology matrix and a set of high-risk interference regions.

[0062] The reinforcement learning model building module is used to build a multi-robotic arm collaborative sorting reinforcement learning model. The state vector is used as the state space, and the joint action space containing discrete task allocation actions and continuous motion control actions is output based on the Actor network.

[0063] The collaborative obstacle avoidance module is used to input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct risk penalty terms, and update and calculate the comprehensive reward based on the risk penalty terms, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model.

[0064] The reinforcement learning training module is used to perform constrained reinforcement learning training on the constrained multi-robotic arm collaborative sorting reinforcement learning model to obtain a multi-robotic arm collaborative control strategy network.

[0065] An edge control deployment module is used to deploy the multi-robotic arm collaborative control strategy network on an edge controller.

[0066] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0067] 1. By introducing a target allocation topology matrix and high-risk intervention areas It can transform the occlusion relationship, stacking and contact relationship, overlapping relationship of working area and potential interference relationship in the coal gangue sorting site into structured constraint information that can be directly involved in the generation of control strategy, thereby improving the rationality of multi-robotic arm task allocation, the pertinence of collaborative control and spatial decision-making ability in complex scenarios.

[0068] 2. Assign topology matrix to the target Introducing action constraints into the reinforcement learning model and setting high-risk intervention regions. Introducing a reward calculation module can reduce illegal action exploration and high-conflict action output, enhance continuous risk feedback during training, thereby improving policy training efficiency, enhancing convergence stability, and improving the executability and robustness of collaborative control strategies.

[0069] 3. By integrating multi-target task allocation, motion control, and spatial collision avoidance into a unified decision-making framework, it is possible to achieve multi-robot collaborative optimization under conditions of dense target distribution, overlapping robot arm operation areas, and continuous conveyor belt operation. This reduces target competition, trajectory intersections, and repeated yielding phenomena, thereby improving the overall sorting cycle time, operation continuity, and collaborative sorting efficiency.

[0070] 4. By adopting a layered architecture that combines offline VLM prior generation with online lightweight strategy network execution, it can reduce the computational burden and response latency in the online control link while retaining the semantic understanding capability of complex scenarios. This meets the requirements of industrial edge for real-time performance, continuity and stability, and has good engineering feasibility and field deployment value. Attached Figure Description

[0071] Figure 1 This is a flowchart of the method of the present invention.

[0072] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0073] like Figure 1 and Figure 2 As shown, the multi-robotic arm collaborative control method for coal gangue sorting based on VLM prior constraint reinforcement learning described in this invention includes: an offline prior construction stage, a constrained reinforcement learning training stage, and an online edge deployment stage;

[0074] Offline prior construction phase: S1, build a virtual simulation environment corresponding to the real site, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels and generate the optimal topology label based on comprehensive cost;

[0075] S1 specifically includes:

[0076] S1.1 Build a virtual simulation environment, including a conveyor belt model, a coal and gangue target model, a multi-robotic arm model, an end effector model, a camera model, a coordinate transformation model, and a collision detection model, etc.; the multi-robotic arm model, camera model, and conveyor belt model correspond to the hardware layout of the actual coal and gangue sorting site; the virtual simulation environment is a digital twin model, which can be deployed on a robot simulation platform with rigid body dynamics simulation, collision detection, and sensor simulation capabilities, such as Gazebo, Isaac Sim, PyBullet, CoppeliaSim, or other similar platforms;

[0077] The conveyor belt model is used to simulate the continuous operation of the conveyor belt. The coal and gangue target model is used to simulate the random distribution of coal and gangue blocks. The camera model is used to simulate the top-view image captured by the top-side camera and the side-view image captured by the side-view camera. The multi-robotic arm model is used to simulate multiple robotic arms, such as robotic arm A, robotic arm B, and robotic arm C. The end effector model is used to simulate the collaborative grasping of the multiple robotic arms. The coordinate transformation model is used to simulate the transformation relationship between the top-view camera coordinate system, the side-view camera coordinate system, the conveyor belt coordinate system, the world coordinate system, the multi-robotic arm base coordinate system, and the end effector coordinate system. The collision detection model is used to simulate the collision detection of the multi-robotic arm collaborative grasping scheme.

[0078] S1.2 At each simulation moment, the speed and running status of the conveyor belt, the image of the candidate target, and the status of the multiple robotic arms are acquired; the image of the candidate target includes a top view image and a side view image; the status of the multiple robotic arms includes the joint angles of the multiple robotic arms, the joint velocities of the multiple robotic arms, and the pose of the end effector of the multiple robotic arms.

[0079] The top-view and side-view images are input into a preset target detection model (such as YOLO, Faster R-CNN, RT-DETR, DETR, etc.). The top-view image is used to obtain the 2D position, contour, orientation, and category of the candidate target in the conveyor belt plane, while the side-view image is used to supplement the target's height, stacking relationship, occlusion relationship, and spatial pose information. Through camera calibration parameters and coordinate transformation models, the detection results from both perspectives are uniformly transformed to the conveyor belt coordinate system or the world coordinate system: the position is calculated from the image detection box, depth information, camera calibration parameters, and coordinate transformation relationship; the pose is determined based on the principal axis direction of the target contour, depth point cloud distribution, or multi-view matching results; and the category is output by the target detection model. This yields the 3D position, pose, and category of the candidate target, and records the upper limit of the capture time window for the candidate target, forming the candidate target image features (3D position, pose, category, and upper limit of the time window for the candidate target).

[0080] S1.3. Based on the three-dimensional position of the candidate target, the state of the multiple robotic arms, and the conveyor belt speed at the current simulation moment, call the trajectory planner or heuristic allocator to generate multiple sets of candidate grasping schemes; the trajectory planner or heuristic allocator runs as a functional module in the virtual simulation environment or as an external algorithm module that communicates with the simulation environment. Its function is to generate candidate grasping schemes based on the target's three-dimensional position, robotic arm state, and conveyor belt speed provided by the simulation environment, and then hand them over to the simulation environment for collision detection and execution evaluation;

[0081] S1.4 Collision detection is performed on the candidate grasping schemes, specifically: within the execution time range of the candidate grasping schemes, that is, from the start of the execution of the candidate grasping scheme to the time period when all participating robotic arms in the scheme complete the grasping, withdrawal or the target leaves the grasping window, the motion sweep space between the end effectors of each robotic arm, the motion sweep space between the sampling points of each robotic arm link, the motion sweep space of the candidate target, and the motion sweep space of adjacent robotic arms are obtained. The minimum safe distance between each robotic arm and obstacles (such as conveyor belt supports, frame structures, etc.) in the virtual simulation environment, the minimum safe distance between each robotic arm and non-candidate targets, and the minimum safe distance between adjacent robotic arms are calculated. The minimum safe distance between adjacent robotic arms is the smaller value between the minimum safe distance between the sampling points of adjacent robotic arm links and the minimum safe distance between the end effectors of adjacent robotic arms.

[0082] When any two or more motion sweep spaces are detected to have volume overlap, or any minimum safe distance is lower than the safe distance threshold, or the motion trajectories of any two or more robotic arms intersect, a collision risk area is determined: if the motion sweep spaces have volume overlap, the collision risk area is the volume overlap area of ​​the motion sweep spaces; if the motion sweep spaces do not have volume overlap, but the minimum safe distance is lower than the safe distance threshold, the collision risk area is the spatial area with a preset radius centered on the nearest point that reaches the minimum safe distance (hereinafter referred to as the area near the nearest point).

[0083] Extract the geometric center of the collision risk area and calculate the impact range of the collision risk area: Based on the spatial distribution of risk sample points in the collision risk area, extract the boundary points or internal voxels of the volume overlapping area as risk sample points, and use the coordinates of the sampling points near the nearest point or the midpoint of the line connecting them as risk sample points. Calculate the geometric center of all risk sample points, and use the maximum distance or statistical quantile distance from the risk sample point to the geometric center as the impact range. Determine the risk level of the collision risk area according to the preset risk level classification table.

[0084] The minimum enclosing sphere radius corresponding to the geometric center of the collision risk area and the range of influence is used as the risk label;

[0085] S1.5 Calculate the overall cost of candidate crawling solutions, using the following formula:

[0086] ;

[0087] in, As candidate crawling solutions, is the estimated execution time of the candidate grasping scheme, which is the sum of the robot arm movement, grasping, waiting time, and conveyor belt compensation time. The collision risk cost of the candidate grasping scheme is calculated as the sum of the products of the risk level weight of each collision risk area and the time taken for the robotic arm to traverse that area. The timeout cost or waiting cost for candidate grasping schemes is calculated as follows: when the candidate target exceeds the upper limit of the grasping time window, it is calculated by multiplying the excess time by the unit time penalty coefficient; when the robotic arm is idle due to cooperative avoidance, it is calculated as the product of the waiting time and the unit time penalty coefficient, and the larger value is taken as the timeout cost or waiting cost. The sorting efficiency benefit of the candidate grasping scheme is the target number of items successfully sorted per unit time, or the total number of items sorted within a fixed time window. , , , These are the weighting coefficients;

[0088] The candidate grasping scheme with the lowest overall cost and that satisfies the basic safety constraints (robotic arm reachability constraint, target time window constraint, collision volume non-overlap constraint, minimum safe distance constraint) is selected as the optimal topology label;

[0089] Robotic arm accessibility constraint: The target position must be within the reachable workspace of the robotic arm, and the posture must meet the grasping requirements of the end effector.

[0090] Target time window constraint: Candidate targets must be captured before the upper limit of the capture time window, otherwise the target has left the operation area.

[0091] Collision volume no overlap constraint: There must be no volume overlap between the sweep space of each robotic arm movement and between the robotic arm and obstacles and non-candidate targets.

[0092] Minimum safe distance constraint: The distance between the ends of adjacent robotic arms and links, and between robotic arms and obstacles, must be greater than the safe distance threshold.

[0093] S1.6 Feedback on collision detection results, including whether volume overlap occurred, overlapping objects, time of overlap, overlapping area or nearest point location, minimum safe distance, and risk level;

[0094] S2. Fuse multimodal data, collision detection results, risk labels, and optimal topology labels. Generate CoT text from the fused data according to a preset template and construct a state vector.

[0095] S2 specifically includes:

[0096] S2.1. Fuse multimodal samples such as top view image, side view image, multi-robotic arm status information, candidate target 3D position information, collision detection results, risk labels, and optimal topology labels. The fusion first unifies the coordinate system, then unifies the timestamp, and finally unifies the data format, transforming the data to the conveyor belt coordinate system or world coordinate system, and organizing them into a sample according to the same simulation time, i.e., the fused multimodal sample. Based on the fused multimodal sample, generate interpretable natural language reasoning text (hereinafter referred to as CoT text) according to a preset template. This text is essentially a structured chain of thought (CoT), which guides the visual language model (VLM) to learn spatial semantic understanding and logical reasoning ability by explicitly presenting the complete reasoning basis from scene perception to decision output, rather than directly exposing the implicit reasoning process inside the model. The template includes scene description, target list, robotic arm status, collision detection results, risk judgment, topology judgment, output labels, etc.

[0097] S2.2 Unify the candidate target image features, the state of the multiple robotic arms, and the operating state of the conveyor belt into multimodal data, which will serve as the state vector. , is represented as:

[0098] ;

[0099] in, Indicates the current The position, attitude, category, and upper limit of the time window for candidate targets at any given time; Indicates the first The joint angle vectors of a robotic arm; Indicates the first The joint velocity vector of a robotic arm; Indicates the first The pose of the end effector of the robotic arm; Indicates the operating status of the conveyor belt; This indicates the number of robotic arms.

[0100] The aforementioned state vectors can unify the coal and gangue target, the conveyor belt operating state, and the state of the multiple robotic arms into the state space of the reinforcement learning model, providing input for subsequent target allocation and joint motion control decision-making.

[0101] S3. Input multimodal data and CoT text into the pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target allocation topology matrix and a set of high-risk interference regions;

[0102] S3 specifically includes:

[0103] S3.1 Input the top view image, side view image, multi-robotic arm status, and CoT text into the pre-trained VLM model, and based on the reasoning logic in the CoT text, parse and extract high-level spatial semantic information from the top view image, side view image, and multi-robotic arm status.

[0104] The process is implemented by combining prompt templates, constrained output formats and post-processing verification: First, a prompt template is constructed, which combines top view image, side view image, multiple robot arm status and CoT text to form a multimodal input context. The CoT text serves as reasoning guidance, describing the complete logical chain from scene perception to risk judgment and then to target allocation judgment, prompting VLM to pay attention to occlusion, stacking, and adjacency relationships between targets, as well as overlapping work areas and potential trajectory interference between robot arms.

[0105] Subsequently, the output format is constrained, requiring VLM to output a semantic relationship description in a fixed format, including target occlusion relationship (occluded target number and occlusion source), stacking relationship (stacked target group and support order), spatial adjacency relationship (adjacent target pairs and spacing), robotic arm operation area overlap relationship (overlapping robotic arm pairs and overlapping area range), and potential trajectory interference relationship (interfering robotic arm pairs and interference time period), presented in the form of structured natural language or tokenized text.

[0106] Finally, post-processing verification is performed. The semantic relationships output by the VLM are extracted and verified in a structured manner through a rule parser. Natural language descriptions are converted into semantic relation tuples with confidence using regular expression matching, keyword extraction, or lightweight syntax parsing. Abnormal outputs with confidence below the threshold or that violate physical consistency (such as conflicting target locations or robot reachability) are removed, forming a standardized high-level spatial semantic information set.

[0107] S3.2. The standardized high-level spatial semantic information set is converted into structured constraints that can be directly invoked for reinforcement learning training. This conversion process is divided into two paths based on the type of semantic information: The first path generates a target assignment topology matrix based on the occlusion relationship, stacking relationship, spatial adjacency relationship, time window upper limit, robotic arm reachability, and risk assessment value of candidate targets. The second path, based on the overlapping relationships of the robotic arm's operating areas and potential trajectory interference relationships, identifies areas where detours are difficult. It then generates a set of high-risk interference areas, considering collision risk areas, areas with difficult detours, historical risk samples, and areas with high incidence of cross-movement. The historical risk samples and high-incidence areas of cross-movement are the spatial range in which the frequency of multiple robotic arm trajectories crossing or passing close to each other exceeds a preset statistical threshold in a preset round of simulation.

[0108] The target allocation topology matrix Recorded as:

[0109] ;

[0110] in, Indicates the number of robotic arms. Indicates the number of candidate targets. Indicates the first The robotic arm and the first Are legal pairings allowed among the candidate targets? A value of 1 indicates that a valid pairing is allowed; otherwise, pairing is not allowed. The rules for determining the value of satisfy the following formula:

[0111] ;

[0112] in, Indicates the first One candidate target, Indicates the first The reachable workspace of a robotic arm Indicates the first The robotic arm reached the first... The estimated execution time for each candidate target Indicates the first The upper limit of the capture time window for each candidate target. Indicates the first Risk assessment value for each robotic arm performing a corresponding grasping action. This represents the risk threshold. The risk assessment value is calculated by weighting path collision risk, minimum safe distance risk, occlusion stacking risk, and time constraint risk, using the following formula:

[0113] ;

[0114] in, This represents the collision detection risk, calculated as the average risk level parameter for all collision risk areas involved in the pairing of the robotic arm and the target. This represents the risk at the minimum safe distance, calculated as the quotient of the safe distance threshold divided by the actual minimum distance. After logarithmic mapping and normalization, the risk is higher the closer the actual distance is to the threshold, and zero when the actual distance is greater than or equal to twice the threshold. This represents the risk of target occlusion or stacking. It is a weighted sum of the number of occlusion sources and the number of stacking layers, then normalized by dividing by the sum of the maximum observed number of occlusion sources and the maximum number of stacking layers. It is zero when there is no occlusion and no stacking. This indicates the urgency and risk within the time window. It is the ratio of the estimated execution time to the upper limit of the capture time window. The closer it is to 1, the higher the risk. When it exceeds or equals 1, the maximum value is taken and marked as infeasible. These are the weighting coefficients.

[0115] The set of high-risk interference areas Recorded as:

[0116] ;

[0117] in, Indicates the first The three-dimensional center coordinates of a high-risk interference region Indicates the first Parameters regarding the impact range of high-risk interference areas. Indicates the first Risk level parameters for high-risk interference areas Indicates the number of high-risk intervention areas;

[0118] High-risk interference areas specifically include: volume overlap areas, i.e., the spatial range where the volume overlaps between the end effector and link of the robotic arm and the candidate target or adjacent robotic arms; near-closest point areas, i.e., the spatial range where the minimum distance between the end effector or link and other objects in the historical trajectories of multiple robotic arms is lower than the safe distance threshold; trajectory intersection areas, i.e. the spatial range where the motion trajectories of multiple robotic arms intersect or converge in space; high-incidence areas of cross motion, i.e. the spatial range where the frequency of cross motion or close-range passage of multiple robotic arm trajectories exceeds the preset statistical threshold in multiple rounds of simulation execution; and difficult detour areas, i.e. the spatial range where the reachable path of the robotic arm is significantly limited due to the occlusion or stacking relationship between targets.

[0119] In this way, VLM transforms high-level semantic relationships such as occlusion, stacking, adjacency, and interference in the coal and gangue sorting site into a target assignment topology matrix. and high-risk intervention areas This is used for constructing subsequent action mask constraints and risk rewards;

[0120] Constrained reinforcement learning training phase: S4, Construct a multi-robotic arm collaborative sorting reinforcement learning model, using the state vector as the state space, and output a joint action space containing discrete task allocation actions and continuous motion control actions based on the Actor network;

[0121] S4 specifically includes:

[0122] Construct a multi-robotic arm collaborative sorting reinforcement learning model (e.g., PPO, SAC, TD3, or other reinforcement learning algorithms suitable for continuous control and joint decision-making). Use the state vector as the state space of the multi-robotic arm collaborative sorting reinforcement learning model, and obtain the action space based on the Actor network of the multi-robotic arm collaborative sorting reinforcement learning model. This includes discrete task allocation actions and continuous motion control actions, and the formula is:

[0123] ;

[0124] in, This indicates the action assignment for discrete tasks, used to determine the grasping target or no-load action corresponding to different robotic arms at the current moment; It represents continuous motion control actions, used to generate joint control quantities, end effector fine-tuning quantities, or grasping execution parameters for each robotic arm;

[0125] By using the reinforcement learning model construction method described above, target allocation and continuous motion control of multiple robotic arms can be unified into the same decision framework, thereby providing a model foundation for subsequent action mask constraints, risk reward construction, and constrained reinforcement learning training.

[0126] S5. Input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct a risk penalty term, and update and calculate the comprehensive reward based on the risk penalty term, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model.

[0127] S5 specifically includes:

[0128] Assign topology matrix to target The action constraint endpoint of the multi-robotic arm collaborative sorting reinforcement learning model is input to apply action mask constraints to the action allocation for discrete tasks. Let the preference value vector of the action allocation for the discrete tasks be... , where is the unnormalized logits output by the Actor network, and the formula is:

[0129] ;

[0130] This represents the task assignment branch in the Actor policy network used to output discrete task assignment action preference values. For network parameters, This is the current state;

[0131] Assign a topology matrix according to the target. Constructing a mask penalty term The number of legal actions is 0, and the number of illegal actions is 0. The formula is:

[0132] ;

[0133] in, It is a preset large positive number; when the actions in the discrete task allocation are determined by the target allocation topology matrix. When an action is determined to be illegal, the mask penalty term takes a preset large negative value to block the illegal pairing action at the output; when the action in the discrete task allocation action is assigned by the target allocation topology matrix When the action is deemed valid, the mask penalty term is set to zero, thus preserving the possibility of the action participating in strategy optimization.

[0134] The probability of assigning actions to discrete tasks satisfies the following formula:

[0135] ;

[0136] in, For the first Assign actions to discrete tasks.

[0137] Through the aforementioned action mask constraints, reinforcement learning models can directly exclude unreachable, time-consuming, or high-risk robotic arm-target pairing actions during training, reducing invalid exploration branches and improving training efficiency and convergence stability.

[0138] High-risk intervention areas Input the reward calculation terminal of the multi-robotic arm collaborative sorting reinforcement learning model. Construct a risk penalty term based on the spatial distance between the key points of the robotic arms (including joint centers, link sampling points, end effector centers, tool center points, etc.) and the centers of each risk region in the high-risk interference region set. For the ... The key points of the robotic arm and the first The three-dimensional Euclidean distance between the centers of the risk areas is calculated in real time. When the three-dimensional Euclidean distance is less than the sum of the influence range of the corresponding risk area and the safety buffer distance, a continuous negative reward is generated; when the distance is greater than the sum of the influence range of the corresponding risk area and the safety buffer distance, no penalty is generated for that risk area. The formula for the risk penalty term is:

[0139] ;

[0140] in, Indicates the first The key points of the robotic arm are relative to the first The penalty value for each risk area, Indicates the first Risk level parameters for each risk area Indicates the first The attenuation parameter for each risk area is calculated using the following formula:

[0141] ;

[0142] Indicates the first The impact range parameter of each risk area This is a preset proportional coefficient. Indicates the safe buffer distance.

[0143] The current state is formed by all the key points and all the risk areas of the robotic arm. The risk penalty for each moment is calculated using the following formula:

[0144] ;

[0145] Calculate the current The comprehensive reward function at time point is given by the formula:

[0146] ;

[0147] in, The formula for successfully claiming a reward is:

[0148] ;

[0149] Indicates the first A robotic arm at any time The value is 0 if the target was successfully captured within the current control period, otherwise it is 0. Weighting for successfully capturing rewards,

[0150] The sorting efficiency bonus is calculated based on the number of sorted items effectively per unit time or the reciprocal of the completion time. The formula is:

[0151] ;

[0152] This indicates the number of target items successfully sorted within the current time window. Indicates the length of the time window. To prevent constants with a denominator of zero, The weighting of sorting efficiency rewards

[0153] The actual collision penalty is represented by a collision indicator function or the number of collisions, calculated using the collision indicator function formula:

[0154] ;

[0155] If a real collision occurs at the current moment, then Otherwise, it is 0;

[0156] The formula for calculating the number of collisions is:

[0157] ;

[0158] This indicates the number of collisions detected within the current time step or the current time window. The weight of the real collision penalty term,

[0159] Indicates risk penalty items, The weight of the risk penalty item,

[0160] This represents the delay penalty, calculated based on the target missing the grasping time window, the robotic arm's waiting time, or the idle time. The formula is:

[0161] ;

[0162] Indicates the first If a candidate target misses the capture window, the value is 0; otherwise, the value is 0. Indicates the waiting time of the robotic arm. The waiting time weighting coefficient, This represents the weight of the delay penalty term.

[0163] Using the above method, the target is assigned a topology matrix. Acting on the action output end, it is used to shield illegal task allocation actions; high-risk interference area set It operates on the reward calculation end to guide the policy network in learning proactive collision avoidance and collaborative yielding actions. Based on action mask constraints and a comprehensive reward function, a multi-robotic arm collaborative sorting reinforcement learning model with constraints is obtained.

[0164] S6. Use the constrained multi-robotic arm collaborative sorting reinforcement learning model to perform constrained reinforcement learning training, and obtain a multi-robotic arm collaborative control strategy network with a fixed semantic prior anti-collision strategy.

[0165] S6 specifically includes:

[0166] By using a reinforcement learning model constrained by action masking and risk reward, a constrained reinforcement learning training method is used to obtain a multi-manipulator collaborative control strategy network with a fixed semantic prior collision avoidance strategy.

[0167] During training, the policy network adjusts its behavior according to the current state. Output joint action The simulation environment returns reward signals based on the robotic arm's execution results, grasping success, collision status, latency, and proximity to risk areas. The reinforcement learning training module calculates the cumulative reward and advantage function based on the sampled trajectory and updates the policy network parameters based on the policy gradient or Actor-Critic type algorithm.

[0168] Specifically, the policy network parameter updates follow the following basic form:

[0169] ;

[0170] in, Indicates policy network parameters, Indicates the learning rate. This indicates the goal of reinforcement learning optimization.

[0171] If the PPO algorithm is used, its objective function is:

[0172] ;

[0173] in, This represents the ratio of the probabilities of the new and old strategies. This represents the estimated value of the advantage function. Indicates the cutting factor;

[0174] As training iterates, the target is assigned a topology matrix. and high-risk intervention areas The expressed spatial semantic priors are gradually internalized into the multi-manipulator collaborative control strategy network.

[0175] After training, the multi-robotic arm collaborative control strategy network is output. This strategy network can be exported as ONNX, TensorRT Engine, TorchScript, or other model formats suitable for edge inference deployment.

[0176] Online edge deployment phase: S7, Deploy the multi-robotic arm collaborative control strategy network on the edge controller;

[0177] S7 specifically includes:

[0178] The computationally expensive Visual Language Model (VLM) is removed, and only the trained multi-robotic arm collaborative control strategy network is deployed on the edge controller. The edge controller can be an NVIDIA Jetson Nano, Jetson Xavier, Jetson Orin, an industrial control computer, an embedded GPU platform, or other edge computing devices with neural network inference capabilities.

[0179] During online operation, it receives data from a top-view camera, a side-view camera, a conveyor belt encoder, and a robotic arm joint encoder. The top-view and side-view cameras are used to capture images of coal and gangue blocks on the conveyor belt; the conveyor belt encoder is used to obtain the conveyor belt speed and operating status; and the robotic arm joint encoder is used to obtain the joint angles, joint speeds, and end effector poses of each robotic arm.

[0180] YOLO, Faster R-CNN, RT-DETR, DETR or other target detection algorithms are used to detect coal blocks and gangue blocks on the conveyor belt. The three-dimensional spatial position and pose information of the candidate targets are obtained by combining binocular vision, structured light, depth camera, camera calibration parameters or multi-view geometric reconstruction methods.

[0181] The collected data is processed to output target detection results, 3D spatial positioning results, multi-arm state estimation results, and conveyor belt speed information. The edge controller loads the trained multi-arm cooperative control strategy network and performs real-time inference according to a preset control cycle. Based on the current state information, it outputs the task allocation results and motion control commands for each arm and sends them to the arm actuators, thereby realizing real-time cooperative control of multiple arms.

[0182] Specifically, the edge controller calculates the safe velocity boundary of the end effector in real time based on the kinematic model of the robotic arm and the Jacobian matrix, and maps the safe velocity boundary of the end effector to dynamic safety upper and lower limits in joint space; subsequently, it performs hard truncation processing on the continuous motion control commands output by the multi-robotic arm cooperative control strategy network based on the dynamic safety upper and lower limits. The output control values ​​of each joint are safely pruned, as follows:

[0183] ;

[0184] in, Indicates the first Each joint The original control quantity at any given time. and These represent the current state of the robotic arm. The dynamic safety lower limit and dynamic safety upper limit are calculated using the Jacobian matrix, end-effector velocity constraint, joint velocity constraint, and safety distance constraint. This indicates the control quantity after safety trimming.

[0185] The aforementioned safety trimming mechanism ensures that the underlying kinematic commands do not exceed the physical safety boundaries of the mechanical system, thereby improving the security and executability during the online deployment phase.

[0186] For example, in a sorting cycle, the top-view camera and the side-view camera first acquire target images of coal and gangue on the current conveyor belt. The field perception module obtains the target bounding boxes, category information, and 3D spatial positions of multiple coal and gangue blocks, and simultaneously obtains the state estimation results of robotic arms A, B, and C. The edge control deployment module calls the multi-robotic arm cooperative control strategy network based on the current state information, and outputs the robotic arm-target task allocation results and corresponding motion control commands for this cycle. Subsequently, each robotic arm executes the grasping action according to the corresponding control commands, and based on the cooperative collision avoidance strategy internalized during the training phase, actively detours, yields, or adjusts the grasping order when approaching potentially high-risk areas to complete continuous coal and gangue sorting.

[0187] Through the above-mentioned online deployment method, the present invention does not require running VLM in the field control stage. Instead, it can achieve real-time task allocation and motion control by relying solely on a lightweight multi-robotic arm collaborative control strategy network, thereby taking into account both the semantic understanding capability of complex scenarios and the real-time control capability of the industrial edge.

[0188] The present invention discloses a multi-robotic arm collaborative control system for coal and gangue sorting based on VLM prior constraint reinforcement learning, comprising: an offline training platform, a top-view camera, a side-view camera, a conveyor belt, robotic arms (in this embodiment, robotic arm A, robotic arm B, and robotic arm C), an edge controller, and software modules.

[0189] The software module includes: a simulation modeling module, used to build a virtual simulation environment corresponding to the real scene, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels, and generate the optimal topology label based on the comprehensive cost;

[0190] The data generation module is used to fuse multimodal data, collision detection results, risk labels, and optimal topology labels, generate CoT text from the fused data according to a preset template, and construct a state vector.

[0191] The visual language constraint generation module is used to input multimodal data and CoT text into a pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target assignment topology matrix and a set of high-risk interference regions.

[0192] The reinforcement learning model building module is used to build a multi-robotic arm collaborative sorting reinforcement learning model. The state vector is used as the state space, and the joint action space containing discrete task allocation actions and continuous motion control actions is output based on the Actor network.

[0193] The collaborative obstacle avoidance module is used to input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct risk penalty terms, and update and calculate the comprehensive reward based on the risk penalty terms, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model.

[0194] The reinforcement learning training module is used to perform constrained reinforcement learning training on the constrained multi-robotic arm collaborative sorting reinforcement learning model to obtain a multi-robotic arm collaborative control strategy network.

[0195] The field perception module is used to receive raw data collected online by the field camera equipment, conveyor belt encoder and joint encoder, and output target detection results, three-dimensional spatial positioning results, multi-robotic arm state estimation results and conveyor belt speed information;

[0196] An edge control deployment module is used to deploy the multi-robotic arm collaborative control strategy network on an edge controller.

Claims

1. A coal and gangue sorting multi-robot arm cooperative control method based on VLM prior constraint reinforcement learning, characterized in that, include: S1. Build a virtual simulation environment corresponding to the real scene, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels, and generate the optimal topology label based on the comprehensive cost. S2. Fuse multimodal data, collision detection results, risk labels, and optimal topology labels. Generate CoT text from the fused data according to a preset template and construct a state vector. S3. Input multimodal data and CoT text into the pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target allocation topology matrix and a set of high-risk interference regions; S4. Construct a multi-robotic arm collaborative sorting reinforcement learning model, using the state vector as the state space, and output a joint action space containing discrete task allocation actions and continuous motion control actions based on the Actor network. S5. Input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct a risk penalty term, and update and calculate the comprehensive reward based on the risk penalty term, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model. S6. Use the constrained multi-robotic arm collaborative sorting reinforcement learning model to perform constrained reinforcement learning training to obtain the multi-robotic arm collaborative control strategy network. S7. Deploy the multi-robotic arm collaborative control strategy network on the edge controller.

2. The coal and gangue sorting multi-robot collaborative control method based on VLM prior constraint reinforcement learning according to claim 1, characterized in that, The process of collecting and processing multimodal data includes: collecting the speed and running status of the conveyor belt, top view images and side view images of candidate targets, joint angles of multiple robotic arms, joint velocities of multiple robotic arms, and end effector poses of multiple robotic arms; inputting the top view images and side view images into a preset target detection model, and extracting and recording the three-dimensional position, pose, category, and upper limit of the time window of the candidate targets.

3. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 2, characterized in that, The collision detection includes: within the execution time range of the candidate grasping scheme, acquiring the motion sweep space of each robotic arm end effector, link sampling point, candidate target and adjacent robotic arm, and calculating the minimum safe distance between each robotic arm and obstacles, non-candidate targets and adjacent robotic arms; when it is detected that there is volume overlap in the motion sweep space, the minimum safe distance is lower than the safe distance threshold or the motion trajectory intersects, the collision risk area is determined to be the volume overlap area or the spatial area with the point corresponding to the minimum safe distance as the center and a preset radius.

4. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 3, characterized in that, The process of extracting risk labels includes: extracting the geometric center of the collision risk area, determining the scope of influence based on the spatial distribution of the collision risk area, and using the minimum bounding sphere radius corresponding to the geometric center of the collision risk area and the scope of influence as the risk label. The process of generating optimal topology tags based on comprehensive cost includes: calculating the comprehensive cost of candidate crawling schemes, using the following formula: ； in, As candidate crawling solutions, The estimated execution time for the candidate crawling solutions. The collision risk cost of the candidate grasping scheme is calculated as the sum of the products of the risk level weight of each collision risk area and the time taken for the robotic arm to traverse that area. The timeout cost or waiting cost for candidate crawling solutions is calculated as the product of the timeout duration and the unit time penalty coefficient, or as the product of the waiting duration and the unit time penalty coefficient. The sorting efficiency benefit of the candidate grasping scheme is the target number of items successfully sorted per unit time, or the total number of items sorted within a fixed time window. , , , These are the weighting coefficients; The candidate grasping scheme with the lowest overall cost, which satisfies the following conditions: the robotic arm can reach the workspace, the posture meets the grasping requirements, the grasping is completed before the upper limit of the grasping time window, the motion sweep space has no volume overlap, and the minimum safe distance is greater than the safety threshold, is selected as the optimal topology label.

5. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 4, characterized in that, The fusion process involves sequentially applying a unified coordinate system, unified timestamp, and unified data format to the multimodal data, collision detection results, risk labels, and optimal topology labels to obtain the fused data. The construction of the state vector includes: taking the three-dimensional position, attitude, category, upper limit of the time window, state of multiple robotic arms, and running state of the conveyor belt as the state vector.

6. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 5, characterized in that, The extraction of high-level spatial semantic information includes: based on CoT text reasoning logic, parsing and extracting occlusion relationships, stacking relationships, spatial adjacency relationships, overlapping relationships of robotic arm operation areas, and potential trajectory interference relationships of candidate targets from the multimodal data, and forming a standardized high-level spatial semantic information set after structured extraction and verification.

7. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 6, characterized in that, The conversion into a target allocation topology matrix and a high-risk interference region set includes: transforming the standardized high-level spatial semantic information set in two ways according to semantic type: the first way generates a target allocation topology matrix based on the occlusion relationship, stacking relationship, spatial adjacency relationship, time window upper limit, robotic arm reachability, and risk assessment value of candidate targets. The risk assessment value The risk is calculated by weighting path collision risk, minimum safe distance risk, occlusion stacking risk, and time constraint risk. The formula is as follows: ； in, The collision detection risk is represented by the average risk level parameter of all collision risk areas involved in the pairing of the robotic arm with the candidate target. The minimum safe distance risk is represented by the quotient of the safe distance threshold divided by the actual minimum distance, obtained by normalization after taking the logarithm mapping. This represents the target occlusion or stacking risk, obtained by weighting the sum of the number of occlusion sources and the number of stacking layers, and then normalizing by dividing by the sum of the maximum number of observed occlusion sources and the maximum number of stacking layers. This indicates a tight time window and is the ratio of the estimated execution time to the upper limit of the capture time window. These are the weighting coefficients; The target allocation topology matrix Recorded as: ； in, Indicates the number of robotic arms. Indicates the number of candidate targets. Indicates the first The robotic arm and the first Whether a valid pairing is allowed between candidate targets satisfies the following formula: ； Indicates the first One candidate target, Indicates the first The reachable workspace of a robotic arm Indicates the first The robotic arm reached the first... The estimated execution time for each candidate target Indicates the first The upper limit of the capture time window for each candidate target. Indicates the first Risk assessment value for each robotic arm performing a corresponding grasping action. Indicates the risk threshold; The second approach identifies areas with difficult detours based on the overlapping relationships of the robotic arm's operating areas and potential trajectory interference. It then generates a set of high-risk interference areas, considering collision risk areas, areas with difficult detours, historical risk samples, and areas with high incidence of cross-movement. The historical risk samples and high-incidence areas of cross-movement are the spatial range in which the frequency of multiple robotic arm trajectories crossing or passing close to each other exceeds a preset statistical threshold in a preset round of simulation. The set of high-risk interference areas Recorded as: ； in, Indicates the first The three-dimensional center coordinates of a high-risk interference region Indicates the first Parameters regarding the impact range of high-risk interference areas. Indicates the first Risk level parameters for high-risk interference areas This indicates the number of high-risk intervention areas.

8. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 7, characterized in that, The action mask constraint for discrete task allocation actions includes: assuming the preference value vector of the discrete task allocation actions is... The formula is: ； in, This represents the task assignment branch in the Actor network used to output discrete task assignment action preference values. For network parameters, This is the current state; Assign a topology matrix according to the target. Constructing a mask penalty term The formula is: ； in, It is a pre-set large positive number; The probability of assigning actions to discrete tasks satisfies the following formula: ； in, For the first Assign actions to discrete tasks.

9. The multi-robotic arm collaborative control method for coal and gangue sorting based on VLM prior constraint reinforcement learning according to claim 8, characterized in that, The construction of risk penalty items, and the updating and calculation of the comprehensive reward based on the risk penalty items, includes: The key points of the robotic arm are determined, and a risk penalty term is constructed based on the spatial distance between the key points of the robotic arm and the centers of each risk area in the high-risk interference area cluster. The formula is as follows: ； in, Indicates the first The key points of the robotic arm are relative to the first The penalty value for each risk area, Indicates the first Risk level parameters for each risk area Indicates the first Attenuation parameters for each risk region, Indicates the first The impact range parameter of each risk area This is a preset proportional coefficient. Indicates the safe buffer distance; The current state is formed by all the key points and all the risk areas of the robotic arm. The risk penalty for each moment is calculated using the following formula: ； Calculate the current The comprehensive reward function at time t is given by the formula: ； in, This indicates the successful capture reward, representing the total number of targets successfully captured by each robotic arm; Weighting for successfully capturing rewards; This represents a sorting efficiency bonus, calculated based on the number of items effectively sorted per unit time or the reciprocal of the completion time. The weighting of sorting efficiency rewards This represents the actual collision penalty, calculated using a collision indicator function or the number of collisions. The weight of the real collision penalty term, Indicates risk penalty items, The weight of the risk penalty item, This represents the delay penalty, calculated based on the target missing the grasping time window, the robotic arm's waiting time, or the idle time. This represents the weight of the delay penalty term.

10. A collaborative control system for coal and gangue sorting using multi-robotic arms based on VLM prior constraint reinforcement learning, characterized in that, include: The simulation modeling module is used to build a virtual simulation environment corresponding to the real scene, collect and process multimodal data, perform collision detection on candidate grasping schemes generated by calling the trajectory planner or heuristic allocator, extract risk labels, and generate the optimal topology label based on the comprehensive cost. The data generation module is used to fuse multimodal data, collision detection results, risk labels, and optimal topology labels, generate CoT text from the fused data according to a preset template, and construct a state vector. The visual language constraint generation module is used to input multimodal data and CoT text into a pre-trained VLM model, extract high-level spatial semantic information, and convert it into a target assignment topology matrix and a set of high-risk interference regions. The reinforcement learning model building module is used to build a multi-robotic arm collaborative sorting reinforcement learning model. The state vector is used as the state space, and the joint action space containing discrete task allocation actions and continuous motion control actions is output based on the Actor network. The collaborative obstacle avoidance module is used to input the target allocation topology matrix into the action constraint end of the multi-robotic arm collaborative sorting reinforcement learning model, perform action mask constraints on the discrete task allocation actions, input the high-risk interference region set into the reward calculation end of the multi-robotic arm collaborative sorting reinforcement learning model, construct risk penalty terms, and update and calculate the comprehensive reward based on the risk penalty terms, thereby obtaining the constrained multi-robotic arm collaborative sorting reinforcement learning model. The reinforcement learning training module is used to perform constrained reinforcement learning training on the constrained multi-robotic arm collaborative sorting reinforcement learning model to obtain a multi-robotic arm collaborative control strategy network. An edge control deployment module is used to deploy the multi-robotic arm collaborative control strategy network on an edge controller.

Images