Multi-robot control method used in multi-task scene

Abstract

A multi-robot control method used in a multi-task scene comprises a general strategy network and a task special strategy network, the general strategy network processes robot sensor observation data through a multi-layer perceptron or a graph neural network, and basic behavior characteristics are extracted; a parameter matrix of the task special strategy network is dynamically generated by the super network according to discrete task identifiers and is used for adapting to task specificity behavior correction such as sorting strength control. The output of the two networks generates a final control action through weighted summation of an adjustable fusion weight, the stability is ensured by taking a general strategy as a leading strategy in the initial training stage, and the task specificity is enhanced in the later stage. According to the method, a multi-agent reinforcement learning framework can be embedded for integrated training, and distributed strategy deployment is supported. In addition, the invention further provides deformation schemes such as a simplified version and a cascade structure for removing universal branches, the method can be effectively applied to scenes such as task switching of storage robots and multi-task execution of industrial mechanical arms, and efficient and light-weight multi-robot multi-task control is achieved.

Description

Technical Field

[0001] The present invention relates to robot control and multi-agent reinforcement learning technology, and in particular to a multi-robot control method for multi-task scenarios. Background Art

[0002] In recent years, multi-agent reinforcement learning (MARL) has made significant progress in the field of multi-robot collaborative control. However, in dynamic multi-task scenarios (such as warehouse robot clusters that need to simultaneously handle heterogeneous tasks such as cargo sorting, path obstacle avoidance, and fault recovery), it faces two core bottlenecks: policy representation conflicts and imbalanced computational efficiency. Existing methods have systematic flaws when dealing with multi-task coupling:

[0003] 1. Policy representation conflict problem

[0004] Existing single network architectures (such as MAPPO-based shared strategies) force all tasks to share the same parameter space, resulting in mutually exclusive gradient directions during strategy optimization. Specifically,

[0005] Inter-task feature entanglement: The policy gradients of different tasks interfere with each other during backpropagation (for example, the transport task requires learning a stable trajectory, while the exploration task requires high exploratory power), causing the network to converge to a suboptimal solution.

[0006] Representation space collapse: Methods (such as QTRAN variants) that dynamically adjust weights through attention mechanisms (such as Transformer) do not constrain the orthogonality of task-specific features, resulting in feature vector cosine similarity greater than 0.8 for highly similar tasks (such as carrying different loads), severely weakening the ability to express policy specificity.

[0007] 2. Computational efficiency issues

[0008] In order to decouple different tasks from each other, a dedicated policy network can be trained for each task, which can improve the performance of a single task. However, the number of parameters will increase linearly with the number of tasks k, making the entire control system not lightweight enough.

[0009] 3. Migration Capability Issues

[0010] Building multi-task strategies through skill combination is also a common technical solution, but its skill base relies on the distribution of offline pre-training data. When online tasks exceed the data coverage (such as new types of cargo handling), the skill combination failure rate is high, and cross-task knowledge transfer cannot be achieved.

[0011] A multi-task reinforcement learning method based on an emergent mixture of experts model uses a mixture of experts model (MoE) with a routing network. Through expert clustering and policy gradient training, it realizes the dynamic allocation of tasks to experts. This MoE-based scheme can reduce the number of parameters during activation. However, for a large number of tasks, a relatively large total number of parameters is still required for learning, and it is not lightweight enough.

[0012] A reinforcement learning method for controlling an agent to perform multiple tasks while interacting with the environment. In multi-task reinforcement learning, it uses meta-gradient to learn action selection options, where the options include the original action sequences executed by the agent under the control of an option policy neural network. In implementation, while the agent interacts with the environment, the system discovers options useful for multiple different tasks through meta-learning rewards for training the option policy neural network. This method lacks decoupling of the policy network, which will result in a loss of effect.

[0013] A method for alleviating catastrophic forgetting in multi-task reinforcement learning. By allocating dedicated memories for each task to store interaction samples and redefining the loss function based on the network gradients of the current task and past tasks, the gradients of the Q-value network are updated, thereby maintaining the agent's memory ability for the learned tasks. This method solves the problem of new tasks overwriting old knowledge in multi-task learning. However, there are network redundancy problems brought about by multiple independent memories and gradient calculations, significantly increasing the computational cost and memory occupancy, and its practicality in large-scale task scenarios is limited.

[0014] It should be noted that the information disclosed in the above background art section is only used for understanding the background of the present application. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0015] The main object of the present invention is to overcome the defects existing in the above background art and provide a multi-robot control method for multi-task scenarios.

[0016] To achieve the above object, the present invention adopts the following technical solutions:

[0017] A multi-robot control method for multi-task scenarios includes the following steps:

[0018] S1. Construct a dual-branch policy network: including a general policy network and a task-specific policy network; the general policy network processes the robot sensor observation data through a multi-layer perceptron or a graph neural network to extract basic behavior features shared across tasks; the parameter matrix of the task-specific policy network is dynamically generated by a hypernetwork according to discrete task identifiers and is used to adapt the correction of the robot task-specific behaviors.

[0019] S2. Policy Fusion and Action Generation: The output vector of the general policy network and the correction of the task-specific policy network are weighted and summed through adjustable fusion weights to generate the final control action of the robot;

[0020] S3. Reinforcement Learning Integrated Training: Embed the dual-branch network into the multi-agent reinforcement learning framework and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0021] S4. Distributed Policy Deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0022] Furthermore, in step S1:

[0023] The general policy network processes the standardized multi-modal sensor observation data, including lidar features, joint states, and pose information;

[0024] The hypernetwork takes the task identifier as input, generates the weight matrix of the fully connected layer of the task-specific policy network through a linear output layer, and directly applies it to policy calculation after dimension reshaping.

[0025] Furthermore, step S2 specifically includes:

[0026] The task-specific policy network constrains the numerical range of the output correction through a non-linear activation function;

[0027] The fusion weight is set to a value close to 1 at the beginning of training, with the general policy as the dominant, and gradually decreased during the training process to enhance task specificity.

[0028] Furthermore, step S3 includes one or more of the following processing mechanisms:

[0029] a) Data preprocessing: Perform mean-variance standardization on the observation data and perform magnitude clipping with sign preservation on the reward value;

[0030] b) Policy optimization: Use the clipped objective function to optimize the executor network, and calculate the temporal difference error through the general advantage estimation to optimize the evaluator network;

[0031] c) Parameter update: The gradient of the task-specific policy network only backpropagates through the hypernetwork, and the general policy network updates parameters independently.

[0032] Furthermore, the policy optimization specifically includes:

[0033] The executor loss function includes a policy ratio clipping mechanism to limit the policy update amplitude;

[0034] The evaluator loss function uses the squared loss of the temporal difference error of the target value network.

[0035] Furthermore, the hypernetwork realizes dynamic parameter generation through the following mechanism:

[0036] Adopt a multi-layer perceptron structure, and apply non-linear activation to the hidden layer;

[0037] The output layer generates a vector through a non-activated linear layer, and forms a task-specific policy network weight matrix through a reshaping operation.

[0038] Furthermore, step S4 specifically includes:

[0039] The evaluator network adds a task identifier input channel to adapt to multi-task value estimation;

[0040] Each robot independently performs forward calculation of the double-branch network and outputs an action to control the mechanical unit in real time.

[0041] A multi-robot control method for multi-task scenarios, including the following steps:

[0042] S1. Construct a task-specific policy network: The parameter matrix of the task-specific policy network is dynamically generated by the hypernetwork according to the discrete task identifier, and is used to directly adapt to the task-specific behaviors of the robot;

[0043] S2. Action generation: Use the output of the task-specific policy network as the final control action of the robot;

[0044] S3. Reinforcement learning integrated training: Embed the task-specific policy network into a multi-agent reinforcement learning framework, and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0045] S4. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0046] A multi-robot control method for multi-task scenarios, including the following steps:

[0047] S1. Construct a cascaded policy network: The general policy network processes the robot sensor observation data and outputs a feature representation shared across tasks; The task-specific policy network takes the feature representation and the task identifier as joint inputs, and its parameter matrix is dynamically generated by the hypernetwork according to the discrete task identifier, and is used to generate the final control action of the robot;

[0048] S2. Reinforcement learning integrated training: Embed the cascaded network into a multi-agent reinforcement learning framework, and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0049] S3. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0050] The method is applied to any of the following scenarios: dynamic switching of goods sorting, obstacle avoidance, and fault recovery tasks in a warehousing robot cluster; parallel execution of assembly operations and anomaly detection tasks by an industrial robotic arm; collaborative processing of multi-modal perception tasks by an edge computing device.

[0051] A computer program product includes a computer program which, when executed by a processor, implements the multi-robot control method for multi-task scenarios described above.

[0052] The present invention has the following beneficial effects:

[0053] The present invention proposes a multi-robot control method for multi-task scenarios, innovatively designs a multi-task reinforcement learning control architecture for multi-robot systems, and achieves a unified breakthrough in high-efficiency computing and high performance in the field of multi-robot collaborative control. The core of the present invention is to use a hypernetwork to dynamically generate task-specific parameters, and on the premise of keeping the total system computing amount constant, accurately adapt to the requirements of multi-task scenarios: for the two-branch structure including a general policy network, by sharing the general feature extractor and hypernetwork parameters, the number of parameters is independent of the number of tasks; for the simplified scheme that removes the general policy network, it completely relies on the hypernetwork to dynamically generate parameters to maintain the lightweight feature. This innovative mechanism completely solves the problem of resource inflation caused by traditional methods (such as increasing the number of experts in the mixture of experts model and linearly increasing storage in the independent memory scheme) with the expansion of tasks.

[0054] At the performance level, the control architecture is significantly improved by using a two-branch collaborative mechanism: the general policy network explicitly models cross-task common features (such as collaborative obstacle avoidance and path planning), laying a foundation for multi-task processing; the task-specific network then uses the dynamic parameters of the hypernetwork to accurately correct specific requirements (such as sorting force and handling speed). Actual tests show that comprehensive performance optimization is achieved in complex scenario tests covering multiple heterogeneous tasks (such as football confrontation, etc.): in terms of control accuracy, the single-task performance is close to the level of an independent dedicated network; the collaborative efficiency in large-scale team collaborative confrontation is significantly improved; in terms of training efficiency, the convergence speed of multi-task joint training is greatly accelerated. The dynamic fusion weight technology (the general policy dominates in the initial stage of training to ensure stability, and the task specificity is enhanced in the later stage to achieve fine-tuning) effectively balances the exploration and optimization processes.

[0055] Resource efficiency optimization is also prominent. Taking a typical multi-task scenario as an example, traditional solutions need to maintain independent parameters for each task, while the present invention only requires shared general parameters and hypernetwork dynamic parameters, and the total number of parameters is much lower than that of traditional solutions. This lightweight design enables the system to be deployed on edge devices with limited computing resources, such as warehouse mobile robots, field operation clusters, etc., significantly expanding the application scenario boundaries. The simultaneously implemented independent parameter update path mechanism avoids multi-task gradient interference and further ensures the training efficiency.

[0056] The core technical advantages of the present invention extend to diverse architecture adaptations. The cascaded structure drives task-specific decisions through the output feature representation of the general network, while the simplified version directly generates actions with hypernetwork parameters, remaining efficient in resource-sensitive scenarios. This flexibility enables the system to select the optimal architecture according to the task coupling degree and hardware conditions, realizing a lightweight multi-task control paradigm of "basic sharing + specific task fine-tuning" in fields such as robot cluster collaboration, industrial automation, and edge computing.

[0057] Other beneficial effects in the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the overall flowchart of a multi-robot control method according to an embodiment of the present invention.

[0059] Figure 2 It is the network structure diagram according to an embodiment of the present invention.

[0060] Figure 3 It is the flowchart of the application algorithm of the double-branch network strategy according to an embodiment of the present invention.

[0061] Figure 4 It is the overall flowchart of a multi-robot control method according to another embodiment of the present invention.

[0062] Figure 5 It is the network structure diagram according to another embodiment of the present invention.

[0063] Figure 6 It is the overall flowchart of a multi-robot control method according to yet another embodiment of the present invention.

[0064] Figure 7 It is the network structure diagram according to yet another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] The following provides a detailed description of the embodiments of the present invention. It should be emphasized that the following description is merely exemplary and not intended to limit the scope of the present invention and its applications.

[0066] In addition, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "a plurality of" means two or more unless otherwise specifically defined.

[0067] Currently, the reinforcement learning algorithms in multi-task scenarios lack the consideration of both computational efficiency and algorithm performance. The novel network architecture proposed by the present invention can achieve high performance while maintaining efficient computing. Training different policy networks for different task scenarios is a common solution. For example, in a multi-task reinforcement learning method based on the emergent mixture of experts model, it is proposed to use a mixture of experts network for training. However, the total number of parameters of the mixture of experts network is relatively large. In order to improve the representation ability of the network, the number of experts needs to be increased accordingly, which increases the storage cost. In a method for alleviating catastrophic forgetting in multi-task reinforcement learning, it is proposed to add a memory for each task to independently store the training data. As the number of tasks increases, the computational cost also rises accordingly.

[0068] The present invention proposes a multi-robot control method for multi-task scenarios and establishes a multi-task reinforcement learning control architecture for multi-robot systems. The core of the present invention is to use a hypernetwork to calculate the policy parameters used for different tasks to achieve different policy corrections for different tasks without increasing additional computational and storage costs. In some embodiments, in order to learn the common features of different tasks, the present invention adds a general policy network outside the hypernetwork. Thus, the total policy network parameters are the sum of the general policy network parameters and the hypernetwork parameters. Compared with the existing solutions, the computational amount required by the present invention for any task scenario is constant and does not increase with the increase in the number of tasks.

[0069] In some embodiments, the present invention designs a dual-branch policy network, which can be seamlessly integrated into any multi-agent reinforcement learning algorithm (such as MAPPO, QMIX, etc.) to replace the existing policy network structure.

[0070] Refer to Figures 1 to 3 , in one embodiment, the multi-robot control method for multi-task scenarios includes the following steps:

[0071] Step S1, construct a dual-branch policy network: including a general policy network and a task-specific policy network; the general policy network processes the robot sensor observation data through a multi-layer perceptron or a graph neural network to extract the basic behavior features shared across tasks; the parameter matrix of the task-specific policy network is dynamically generated by the hypernetwork according to the discrete task identifier for adapting the correction of the robot task-specific behavior.

[0072] Specifically, in step S1, the general policy network processes the standardized multi-modal sensor observation data, including lidar features, joint states, and pose information; the hypernetwork takes the task identifier as input, generates the weight matrix of the fully connected layer of the task-specific policy network through a linear output layer, and directly applies it to policy calculation after dimension reshaping.

[0073] Step S2, Policy Fusion and Action Generation: The output vector of the general policy network and the correction amount of the task-specific policy network are weighted and summed through adjustable fusion weights to generate the final control action of the robot.

[0074] In some embodiments, step S2 specifically includes: the task-specific policy network constrains the numerical range of the output correction amount through a non-linear activation function; the fusion weight is set to a value close to 1 at the beginning of training, with the general policy as the dominant, and gradually decreases during the training process to enhance task specificity.

[0075] Step S3, Reinforcement Learning Integrated Training: Embed the dual-branch network into the multi-agent reinforcement learning framework, and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0076] In some embodiments, step S3 includes one or more of the following processing mechanisms: a) Data preprocessing: Perform mean-variance standardization on the observation data and perform magnitude clipping with sign retention on the reward value; b) Policy optimization: Optimize the executor network using a clipped objective function, and optimize the evaluator network by calculating the temporal difference error through general advantage estimation; c) Parameter update: The gradient of the task-specific policy network is only backpropagated through the hypernetwork, and the general policy network updates parameters independently. In some embodiments, the policy optimization specifically includes: the executor loss function includes a policy ratio clipping mechanism to limit the policy update amplitude; the evaluator loss function uses the squared loss of the temporal difference error of the target value network.

[0077] Step S4, Distributed Policy Deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0078] In some embodiments, the hypernetwork realizes dynamic parameter generation through the following mechanism: adopting a multi-layer perceptron structure, applying non-linear activation to the hidden layer; the output layer generates a vector through a non-activated linear layer, and forms the weight matrix of the task-specific policy network through a reshaping operation.

[0079] In some embodiments, step S4 specifically includes: adding a task identifier input channel to the evaluator network to adapt to multi-task value estimation; each robot independently performs the forward calculation of the dual-branch network and outputs the action to control the mechanical unit in real time.

[0080] Refer toFigure 4 and Figure 5 In another embodiment, a multi-robot control method for a multi-task scenario includes the following steps:

[0081] S1. Construct a task-specific policy network: The parameter matrix of the task-specific policy network is dynamically generated by a hypernetwork according to a discrete task identifier and is used to directly adapt to the task-specific behaviors of the robot;

[0082] S2. Action generation: Use the output of the task-specific policy network as the final control action of the robot;

[0083] S3. Reinforcement learning integrated training: Embed the task-specific policy network into a multi-agent reinforcement learning framework and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0084] S4. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0085] Refer to Figure 6 and Figure 7 In yet another embodiment, a multi-robot control method for a multi-task scenario includes the following steps:

[0086] S1. Construct a cascaded policy network: The general policy network processes the robot sensor observation data and outputs a feature representation shared across tasks; the task-specific policy network takes the feature representation and the task identifier as joint inputs, and its parameter matrix is dynamically generated by a hypernetwork according to a discrete task identifier and is used to generate the final control action of the robot;

[0087] S2. Reinforcement learning integrated training: Embed the cascaded network into a multi-agent reinforcement learning framework and jointly optimize the network parameters through the interaction data between the robot and the environment;

[0088] S3. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

[0089] The present invention uses the task-specific parameters dynamically generated by the hypernetwork to perform task feature calculation. In the embodiment including a general policy network (general feature extractor), by sharing the parameters of the extractor and the hypernetwork, the total number of parameters does not increase with the number of tasks; in the embodiment where the general policy network is removed, only the parameters dynamically generated by the hypernetwork are used to keep the number of parameters constant.

[0090] The method of the present invention can be applied to any of the following scenarios, including but not limited to: dynamic switching of cargo sorting, obstacle avoidance, and fault recovery tasks in a cluster of warehouse robots; parallel execution of assembly operations and anomaly detection tasks by industrial robotic arms; collaborative processing of multi-modal perception tasks by edge computing devices.

[0091] The following further describes specific embodiments of the present invention, its algorithm examples, and experimental verification.

[0092] A multi-robot control method for multi-task scenarios, which establishes a multi-task reinforcement learning control architecture for a multi-robot system. A dual-branch policy network that can be seamlessly integrated into any multi-agent reinforcement learning algorithm (such as MAPPO, QMIX, etc.) is designed.

[0093] The working principle includes two parts. First is the output process of the network structure, and second is the integrated training process of the network. The processing process of the network structure mainly includes three steps: 1. Dynamic parameter generation, where the hypernetwork maps the task encoding to the weights of the task-specific policy network. 2. Dual-branch parallel processing, first using a multi-layer perceptron or a recurrent neural network to extract cross-task general features of the robot observations, and then using the dynamic weights of the task-specific policy network to calculate the task-specific correction amount. 3. Combining the two parts of the policy output in step 2 through policy fusion to obtain the final action output. The integrated training process of the network means that the network can be embedded in any multi-agent reinforcement learning algorithm for training to improve its performance in multi-task scenarios.

[0094] Specifically, the multi-task reinforcement learning control architecture consists of the following two parts:

[0095] I. Network structure design:

[0096] The dual-branch network structure includes two parts - the general policy network π g and the task-specific policy network π t , as Figure 2 shown. The input of the general policy network π g is the robot sensor observation data where t is the current time step, d o is the dimension of the observation space. The robot sensor observation data may include but is not limited to: preprocessed lidar point cloud features (e.g., occupancy grid, distance feature vector, point cluster feature), joint angles (usually represented as radian values), joint velocities, the body pose of the robot (such as position, orientation), depth map information, etc. These inputs usually need to be standardized or normalized before entering the network; the output is the general policy vector d is the dimension of the action space for the final low-level control of the robot, including the linear velocity v of the mobile robot ∈ [v min ,vmax ] and angular velocity ω∈[ω min ,ω max ] (d = 2 at this time); the desired torque vector of the manipulator joint (m is the number of joints, d = m); a composite platform may include both movement velocity and joint torque (d = 2 + m). Its function is to learn basic strategies shared across tasks (such as obstacle avoidance and collaborative navigation). This can be implemented using a multi-layer perceptron (MLP) or a graph neural network (GNN). Its parameters can be initialized using any method (such as all-zero initialization or orthogonal random initialization). The core goal of this network is to learn basic skill strategies that are common to all tasks k = 1, ..., K (such as obstacle avoidance, approaching the target, maintaining stability, and basic collaborative behaviors). This is used to capture underlying patterns and behavioral patterns that are common and beneficial across different tasks. Depending on the nature of the input data, the following different structures can be used:

[0097] Multilayer Perceptron (MLP): Usually contains multiple fully connected layers (FC Layers), and nonlinear activation functions (such as ReLU, tanh) are used between layers. The input layer dimension is d o , the output layer dimension is d, and the intermediate hidden layer dimension (e.g. 256, 512) is adjustable. The choice of output layer activation function depends on the action space: for example, speed with a limited range might use tanh + scaling, and torque might use linear activation.

[0098] Graph Neural Networks (GNNs): particularly suitable for sensor observations t This applies to situations where graph structure information naturally exists in the model, such as representing the topological relationships of a multi-robot system (using robots or key points as nodes and communication / collaboration relationships as edges), or modeling the spatial relationships between entities in the scene (such as the positions of obstacles and target points relative to the robots). Specific architectures include graph convolutional networks (GCNs), graph attention networks (GATs), or message passing neural networks (MPNNs). The output of a GNN is typically flattened or pooled before being connected to the final action output layer (usually a FC layer).

[0099] π g The weight parameter (denoted as θ g ) can be randomly initialized using common initialization strategies (such as Glorot / Xavier initialization, He initialization, or orthogonal initialization). Because it needs to learn a universal representation, the initial weights are usually not zero.

[0100] Mission-Specific Policy Network π t Parameters Dynamically generated by the hypernetwork g. The input of the hypernetwork g is the task code e k, the task encoding method in the present invention is the one - hot vector of the task ID, k = 1, …, K is the task index, that is, e k ∈ {0, 1} K , and its k - th element is 1, and the rest are 0. This is a discrete and explicit identifier of the task; the output of the hyper - network is the parameter matrix used to obtain the policy adapted to the characteristics of the current task (such as the grasping force control parameter of the sorting task). Its input is the task encoding e k , and the output is the core weight matrix of the entire task - specific policy network π t Here, the hyper - network g generates the "main body" parameter matrix of the task - specific policy network. The state space d × d ensures that π o can receive the same observation input o t as π g (dimension d t ), and outputs an action correction amount with the same dimension as the output of π o (dimension d). This design allows π g to act on the output space of π t in the simplest and most direct way. The hyper - network g itself is also a learnable neural network, usually using an MLP structure. The dimension of its input layer is K (task encoding dimension). The output layer needs to generate a vector of dimension (d g × d), so its last layer is a linear layer (without an activation function), and the dimension is precisely adjusted to d o × d. This output vector is reshaped into the required matrix o The hyper - network g can include one or more hidden layers (optional dimensions, such as 128, 256) and activation functions (such as ReLU). The weight parameters of the hyper - network g are denoted as

[0101] The input of the task - specific policy network π t is the same as that of the general policy network π g , and its output is the task - specific action correction amount. Its forward calculation process is as follows: Given the encoding e k of the current task k, generate the parameter through the hyper - network gπ t The input of π is also the robot sensor observation data (consistent with the input of π g ). The output of π t is the task - specific action correction amount Here: · represents matrix multiplication, b t is π tLearnable bias vector. σ is usually a non-linear activation function, and its selection principle is similar to that of π g The activation function of the output layer. It depends on the constraints of the action space. To make the correction amount positive and negative and easy to fuse, tanh is often used to limit the range (such as [-δ, δ]) or linear activation.

[0102] The output of the entire double-branch policy network is the weighted sum of the two policies a t = α · π g (o t ) + (1 - α) · π t (o t ), where α ∈ [0, 1] is the policy fusion hyperparameter, which can be dynamically adjusted according to different task effects. This fusion mechanism combines the g robust and generalizable general behaviors learned by π (such as avoiding any obstacles), and the t fine-tuned behaviors learned by π and customized by the hypernetwork g for the details of the current task (such as the precise grasping posture and force for a specific target object).

[0103] II. Integrated application method

[0104] This network can be embedded into the policy module of the existing multi-robot reinforcement learning algorithm. Taking the MAPPO algorithm as an example, the implementation process is described as follows:

[0105] (1) Algorithm adaptation and network reconstruction

[0106] The executor network of the standard MAPPO algorithm is transformed into a double-branch structure, where the general policy branch π g adopts a three-layer fully connected network (input dimension d o , hidden layers 256 - 128 - 64, output dimension d), and the parameter matrix of the task-specific branch π t is dynamically generated by the hypernetwork g. The input of the hypernetwork g is the task one-hot encoding (K is the total number of tasks), and the output layer uses a linear mapping to generate . The critic network retains the original structure but adds a task encoding input channel, and its state value function is expressed as V(s , e t ). k )

[0107] (2) Training data preparation and processing

[0108] In a robot simulation platform (such as Gazebo, MuJoCo, etc.), a training scenario containing K tasks (such as material sorting, collaborative handling, etc.) is constructed. Task parameters (such as target position, object mass, etc.) are randomly initialized in each episode. The current strategy is used to interact with the environment to obtain training data (o t ,a t ,r t ,o t+1 ,e k ) is stored in the experience replay pool In which observation o t Contains data such as all robot joint states, force sensor readings, etc., and rewards r t Designed according to the task type (e.g., sorting tasks include whether the grasping is successful or not, the distance between the end of the robot arm and the target, etc.). After data collection, perform the following preprocessing operations: i) Observation standardization: ii) Reward Clipping: r t =sign(r t )·min(|r t |,r max ). where μ o , σ o are the mean and standard deviation of the observed data, sign is the sign function, r max is a preset reward clipping hyperparameter.

[0109] (3) Training strategy

[0110] Select the appropriate policy fusion hyperparameter α (multiple experiments may be required to get the best value), and use gradient descent to optimize the policy network π g , g, the loss function is

[0111]

[0112] Among them A t is the advantage function, estimated by the general advantage estimation GAE:

[0113]

[0114] θ is the parameter of the entire two-branch network, π final Represents the entire two-branch policy network. For the evaluator network used in training, the temporal difference error loss is used:

[0115]

[0116] Where V φ represents the evaluator network, φ represents the parameters of the evaluator network, After updating the parameters of the executor network and the evaluator network using the gradient descent method, it is determined whether the upper limit of the number of iterations is reached or the expected reward requirement is met. If so, the training is stopped; otherwise, return to S1 to continue collecting training data for updating.

[0117] (4) Policy Deployment

[0118] The central controller assigns the task code e to each robot k , and each robot generates the action a in real time according to the local sensor data o t and the task code to control the robotic arm or the robot chassis. The overall application algorithm process of the dual-branch network policy is as t shown. Figure 3 shown.

[0119] Effect Verification

[0120] In the field of multi-robot cooperation to complete complex tasks, the dual-branch policy network proposed by the present invention shows significant advantages, which are mainly reflected in three aspects: performance improvement, resource efficiency optimization, and strong migration ability.

[0121] In terms of performance improvement, the dual-branch policy network has achieved remarkable results in actual tests. Through systematic tests in highly simulated experimental environments such as the Google Football research platform, in complex test scenarios covering 8 heterogeneous tasks, the network has achieved efficient and stable task processing. These scenarios include the 3-on-1 offense and defense scenario, which is used to verify the cooperation and decision-making ability of robots in asymmetric confrontations; the passing and shooting scenario, which focuses on testing the precise action execution of robots in a static environment; the running passing and shooting scenario, which simulates the cooperation and decision-making of robots in a dynamic environment; the classic 5-on-5 team confrontation scenario, which is used to study complex tactical cooperation and strategic games; the corner kick scenario, which focuses on the timing of robots in set-piece tactics; the simple counterattack scenario and the difficult counterattack scenario, which respectively simulate the offense and defense conversion situations of different difficulties; and the 11-on-11 full-scenario confrontation, which comprehensively examines the performance of robots in large-scale team cooperation. Experimental data shows that the control accuracy of the dual-branch policy network for a single task can reach more than 98% of that of an independent dedicated network. In the 11-on-11 football confrontation scenario, it has successfully improved the team cooperation efficiency by 23% and effectively reduced the number of mistakes caused by policy conflicts. In the process of multi-task joint training, the convergence rate of this network has increased by 40% compared with traditional methods, and it can reach the converged policy state only in 60% of the traditional training time, significantly reducing the training time cost. The realization of this performance benefits from the unique architecture design of the dual-branch policy network: the general policy network explicitly models cross-task common features, such as cooperation and obstacle avoidance, path planning, etc., laying a solid foundation for multi-task processing; the task-specific network accurately adapts to task-specific features, such as sorting force, handling speed, etc., with the help of parameters dynamically generated by the hypernetwork, ensuring that each task can be optimally processed.

[0122] In terms of resource efficiency optimization, the dual-branch policy network also shows strong competitiveness. Through its parameter dynamic generation mechanism of the hypernetwork, it has achieved a significant optimization of computational costs. Taking an experimental scenario with 8 tasks (K = 8) as an example, the traditional independent network solution needs to maintain 8 sets of complete policy network parameters, which not only occupies a large amount of storage space but also places extremely high requirements on computing resources. However, the dual-branch policy network solution proposed in this invention only requires 1 set of general network parameters and dynamic parameters generated by the hypernetwork, and the total number of parameters is reduced by 76%. This lightweight design enables this method to be easily deployed on edge devices with limited computing resources, such as mobile robots in intelligent warehousing, small robot clusters for field operations, etc., greatly expanding the application scenarios of multi-robot systems.

[0123] The reason why the dual-branch policy network can achieve such excellent technical effects lies in a series of innovative technical mechanisms behind it. First of all, the dual-branch network ensures that the parameter update paths of the general policy network and the task-specific policy network are independent of each other, avoiding performance degradation caused by gradient interference during the training of different tasks, enabling the two networks to perform their respective functions and co-evolve. Secondly, the hypernetwork endows the system with powerful dynamic adaptation capabilities. The task-specific parameters generated by it can give precise action corrections in real time according to the characteristics of different tasks, enabling the robot to quickly adjust its strategy when facing complex and changing task requirements. Finally, policy fusion can adjust the contribution ratio of the general policy and the specific policy according to the training process by introducing the learnable fusion weight α. For example, in the exploration stage at the beginning of training, α≈0.8, and at this time the general policy dominates to ensure the stability and reliability of the policy and avoid training divergence caused by excessive exploration; it can be gradually reduced as the training process progresses to enhance task specificity. In the optimization stage at the end of training, α≈0.3, and the specific policy plays a major role, highlighting task specificity, enabling the robot to make refined policy adjustments for specific tasks. This dynamic balance mechanism significantly improves the training efficiency and policy quality.

[0124] In some alternative embodiments, the general features and task-specific features can be concatenated, and then the final action can be generated through a multi-layer perceptron. Other task encoding methods can also be used to replace the one-hot encoding used in the original scheme, such as task semantic vectors, etc.

[0125] Variant Scheme 1: Remove the general branch

[0126] This variant scheme simplifies the structure of the original technical scheme, retains the core hypernetwork parameter generation mechanism, but removes the general policy network and the policy fusion part, as Figure 4 and Figure 5 shown. The specific implementation method is as follows:

[0127] (1) Network architecture: Only retain the task-specific policy network π t , whose parameters are completely dynamically generated by the hypernetwork g according to the task encoding e k , that is

[0128] (2) Policy output: Directly use π t (o t ) as the final action output, and cancel the weighted fusion step with the general policy.

[0129] The advantage of this solution is that it further reduces the model complexity. The total number of parameters can be reduced by 30-40% compared with the original solution, which is more advantageous for deployment on resource-constrained embedded devices (such as industrial robot controllers). In scenarios with significantly different task characteristics (such as the coexistence of sorting and inspection tasks), its single-task performance is comparable to that of the original solution. This simplified design also has limitations: due to the lack of the common feature learning ability of the general policy network, knowledge transfer cannot be achieved between similar tasks (such as the handling of objects with different weights). When there are some commonalities between the new task and the existing tasks, its training efficiency is lower than that of the double-branch policy network solution.

[0130] (3) Application method: Similar to the original solution, this variant solution can be applied to any multi-agent reinforcement learning algorithm. Replace the default multi-layer perceptron network in the algorithm and operate according to the Figure 3 shown training and deployment process.

[0131] Variant solution 2: Double-branch network structure with a cascade structure

[0132] This variant solution retains the core idea of "shared basis + task-specific", but changes the way of branch interaction. As Figure 6 and Figure 7 shown, it adopts a cascade structure instead of the parallel structure of the original solution. The basic network is responsible for providing general state representations that are strongly related to the environment, while the task-specific network further "fine-tunes" or "gates" based on this representation and task encoding to generate the final action decision. Formally, it is close to "mounting" a task-specific adaptation layer on the general network. The specific network structure design includes three parts: a general feature extraction network (corresponding to the general policy network in the original network structure), a hypernetwork, and a task-conditioned feature processing network (corresponding to the task-specific policy network in the original solution).

[0133] Application

[0134] The application scenarios of the present invention mainly include intelligent systems that require lightweight processing of multiple tasks. The present invention is particularly suitable for devices or systems that need to run on limited computing resources (such as edge devices, embedded systems), and these devices are faced with the requirement of flexibly coping with multiple different tasks. Typical scenarios include:

[0135] 1. Robot cluster collaboration: For example, warehouse robots need to carry goods in a crowded environment (task A), avoid obstacles in real time (task B), and take over tasks when teammates fail (task C). The present invention enables a single robot to instantaneously adjust its strategy in a lightweight manner according to the specific task instructions it currently receives (such as carrying a specific item or going for rescue).

[0136] 2. Industrial Automation: For example, a robotic arm on a production line might perform tasks such as part assembly (Task A), equipment status monitoring (Task B), or emergency response to defective products (Task C). This invention enables the robotic arm to dynamically fine-tune execution details based on the current production step (task instruction) while maintaining its core operational capabilities.

[0137] Smart edge devices, such as security cameras or in-vehicle systems, need to simultaneously perform target recognition (Task A), abnormal behavior detection (Task B), and specific event response (Task C). This invention helps these devices efficiently coordinate multiple task processing modules based on different preset modes (task instructions), avoiding overwhelming resource-constrained processors with the increased number of tasks.

[0138] 3. Resource-constrained game AI: For example, a large number of NPCs need to perform multiple interactions (patrol, combat, gathering, etc.). This invention provides an efficient way to share a model base and then make differentiated adjustments based on the NPC role or current goal (task instructions).

[0139] As long as a device or system needs to be ready to perform multiple preset tasks at any time (and there may be differences or even conflicts between tasks) under the premise of limited resources (computing power, storage), the lightweight strategy architecture of "shared foundation + task-based fine-tuning" of the present invention can be used.

[0140] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0141] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0142] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0143] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but not limited to, these and any other suitable types of memories.

[0144] In several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling or communication connection between the various components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0145] The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, in each embodiment of the present invention, the various functional units can all be integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in a unit; the above integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0147] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0148] Alternatively, if the above integrated units are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present invention essentially or the parts that contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the foregoing storage medium includes: various media such as removable storage devices, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0149] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0150] The features disclosed in the several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0151] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0152] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those skilled in the technical field to which the present invention belongs, without departing from the concept of the present invention, several equivalent substitutions or obvious variations can be made, and as long as the performance or use is the same, they should all be regarded as belonging to the protection scope of the present invention.

Claims

1. A multi-robot control method for multi-task scenarios, characterized in that, It includes the following steps: S1. Construct a dual-branch policy network: including a general policy network and a task-specific policy network; the general policy network processes the robot sensor observation data through a multi-layer perceptron or a graph neural network to extract basic behavior features shared across tasks; The parameter matrix of the task-specific policy network is dynamically generated by a hypernetwork according to discrete task identifiers and is used to adapt the correction of the robot task-specific behavior; S2. Policy fusion and action generation: The output vector of the general policy network and the correction amount of the task-specific policy network are weighted and summed through adjustable fusion weights to generate the final control action of the robot; S3. Reinforcement learning integrated training: Embed the dual-branch network into a multi-agent reinforcement learning framework and jointly optimize the network parameters through the interaction data between the robot and the environment; S4. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

2. The method according to claim 1, wherein In step S1: The general policy network processes the standardized multi-modal sensor observation data, including lidar features, joint states, and pose information; The hypernetwork takes the task identifier as input, generates the weight matrix of the fully connected layer of the task-specific policy network through a linear output layer, and directly applies it to policy calculation after dimensional reshaping.

3. The method according to claim 1 or 2, characterized in that Step S2 specifically includes: The task-specific policy network constrains the numerical range of the output correction amount through a non-linear activation function; The fusion weight is set to a value close to 1 at the beginning of training, with the general policy as the dominant, and gradually decreases during the training process to enhance task specificity.

4. The method according to any one of claims 1 to 3, characterized in that, Step S3 includes one or more of the following processing mechanisms: a) Data preprocessing: Perform mean-variance standardization on the observation data and perform magnitude clipping with sign retention on the reward value; b) Policy optimization: Use a clipped objective function to optimize the executor network, and calculate the temporal difference error through general advantage estimation to optimize the evaluator network; c) Parameter update: The gradient of the task-specific policy network only backpropagates through the hypernetwork, and the general policy network updates parameters independently.

5. The method according to claim 4, wherein The policy optimization specifically includes: The executor loss function includes a policy ratio clipping mechanism to limit the policy update amplitude; The evaluator loss function uses the squared loss of the temporal difference error of the target value network.

6. The method according to any one of claims 1 to 5, characterized in that The hypernetwork realizes dynamic parameter generation through the following mechanism: Adopt a multi-layer perceptron structure, and apply non-linear activation to the hidden layer; The output layer generates a vector through a non-activated linear layer, and forms the weight matrix of the task-specific policy network through a reshaping operation.

7. The method according to any one of claims 1 to 6, characterized in that Step S4 specifically includes: The evaluator network adds a task identifier input channel to adapt to multi-task value estimation; Each robot independently performs the forward calculation of the dual-branch network and outputs the action to control the mechanical unit in real time.

8. A multi-robot control method for multi-task scenarios, characterized in that, It includes the following steps: S1. Construct a task-specific policy network: The parameter matrix of the task-specific policy network is dynamically generated by a hypernetwork according to discrete task identifiers and is used to directly adapt the robot task-specific behavior; S2. Action generation: Use the output of the task-specific policy network as the final control action of the robot; S3. Reinforcement learning integrated training: Embed the task-specific policy network into the multi-agent reinforcement learning framework, and jointly optimize the network parameters through the interaction data between the robot and the environment; S4. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

9. A multi-robot control method for multi-task scenarios, characterized in that, It includes the following steps: S1. Construct a cascaded policy network: The general policy network processes the robot sensor observation data and outputs a feature representation shared across tasks; The task-specific policy network takes the feature representation and task identifier as joint inputs, and its parameter matrix is dynamically generated by the hypernetwork according to the discrete task identifier, and is used to generate the final control action of the robot; S2. Reinforcement learning integrated training: Embed the cascaded network into the multi-agent reinforcement learning framework, and jointly optimize the network parameters through the interaction data between the robot and the environment; S3. Distributed policy deployment: The central controller assigns task identifiers, and each robot generates control actions in real time based on local sensor observations and task identifiers.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-robot control method for multi-task scenarios according to any one of claims 1 to 9.

Images