A reinforcement learning and multi-expert hybrid model-based

Abstract

The application belongs to the technical field of dexterous hand grasping model, and in particular to a reinforcement learning and multi-specialist hybrid model, and the establishment of the reinforcement learning and multi-specialist hybrid model comprises the following steps: step 1: preparation of a diversified data set: an object data set containing various morphological and geometric characteristics is constructed, and a simulation model of a dexterous hand is combined for training and evaluation; step 2: training of an expert strategy using a reward function enhanced PPO: a generalist-specialist hybrid model for dexterous hand operation based on reinforcement learning is constructed. The application realizes efficient generalization of dexterous hand grasping through a phased learning strategy; first, a basic expert strategy for dexterous operation is trained using reinforcement learning, and a high-performance expert model targeted at different objects and operation modes is obtained; subsequently, under the Generalist-Specialist Learning framework, multiple expert strategies are gradually distilled into a more compact generalist strategy; unlike the traditional direct distillation method.

Description

Technical Field

[0001] This invention relates to the field of dexterous hand grasping model technology, specifically a hybrid model based on reinforcement learning and multiple experts. Background Technology

[0002] Dexterous hand grasping, a key challenge in robotics manipulation, has long attracted widespread attention. Compared to common two- or three-finger grippers, multi-fingered dexterous hands possess a structure and manipulative capabilities more closely resembling those of the human hand. However, their high degrees of freedom necessitate coordinated control of policy learning within a high-dimensional action space. This significantly increases the complexity of modeling, planning, and optimization, and also makes policies prone to performance degradation when faced with diverse, unseen, or geometrically complex objects in real-world scenarios. Therefore, achieving lightweight training, efficient generalization, and stable deployment of models while maintaining grasping capabilities has become a significant research challenge in this field.

[0003] In recent years, much work has been dedicated to improving the generality and efficiency of dexterous hand grasping strategies. For example, the ResDex framework, by introducing a geometry-unaware base expert policy and combining a Mixture-of-Experts mechanism with residual learning, achieved excellent grasping success rate and cross-object generalization performance on the DexGraspNet large-scale benchmark, while significantly reducing training time. This method demonstrates that residual correction and expert fusion can effectively mitigate the instability caused by differences in task distribution. However, the inherent complexity of the MoE structure introduces additional computational and inference overhead, limiting its large-scale deployment in real-world robotic systems.

[0004] Meanwhile, UniDexGrasp++ introduces geometry-aware curriculum learning and an iterative generalist–specialist learning paradigm. This method first trains a generalist policy under visual conditions, then progressively refines it into multiple specialist policies, significantly improving the policy's generalization ability to different object geometry and appearance variations. However, this type of training method based on policy distillation and multi-stage transfer may still face problems such as inconsistency in cross-expert supervision, conflicting distillation objectives, and unstable cross-domain knowledge transfer, affecting its final performance on complex tasks. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a hybrid model based on reinforcement learning and multiple experts, which solves the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0009] A hybrid model based on reinforcement learning and multiple experts is proposed. The steps involved in establishing this model are as follows:

[0010] Step 1: Preparation of diverse datasets: Construct a dataset of objects with various shapes and geometric features, and combine it with a simulation model of dexterous hands for training and evaluation;

[0011] Step 2, using reward function-enhanced PPO training expert policy: constructing a general-expert hybrid model for dexterous hand manipulation based on reinforcement learning;

[0012] Step 3: GSL-driven expert strategy distillation and generalist strategy construction;

[0013] Step 4: Training and dynamic routing mechanism of hybrid expert structure MoE.

[0014] Furthermore, the simulation model of the dexterous hand in step 1 is the ShadowHand simulated dexterous hand, a highly biomimetic five-fingered robotic hand. Its structural design fully simulates the bone and joint distribution of a real human right hand, with more than twenty degrees of freedom, enabling complex and precise grasping and manipulation movements. The thumb has a more flexible multi-level joint structure, enabling palm opposition, rotation, and bending movements, thus supporting fine manipulation in three-dimensional space. The other four fingers are composed of four segments connected in series: metacarpophalangeal joint, proximal segment, mid-segment, and distal segment, which can bend independently and coordinate grasping.

[0015] Furthermore, the object dataset is DexGraspNet, which provides accurate hand pose annotations, complete object geometry information, and detailed contact point data, supporting the training and evaluation of various dexterous hand models in simulation and real-world environments. Its high diversity in object shape and grasping posture makes it an important benchmark dataset for current research on general grasping strategies, cross-object generalization capabilities, and vision-to-action mapping.

[0016] Furthermore, the reinforcement learning in step 2 specifically includes:

[0017] First, within the reinforcement learning framework, the sequential decision-making process is formalized as a Markov decision-making process. Markov Decision Process (MDP) is a mathematical framework for describing sequential decision-making in uncertain environments. It uses states, actions, state transition probabilities, reward functions, and discount factors as basic elements to characterize the interaction between an agent and its environment.

[0018] In a given state, an agent selects an action based on a policy. The environment then transitions to the next state with a certain probability based on the action and returns a corresponding reward. The agent's goal is to maximize the cumulative reward by learning the optimal policy through long-term interaction.

[0019] The core characteristic of a Markov Decision Process (MDP) is that it satisfies the Markov property, meaning that the state in the next time step depends only on the current state and actions and is independent of the past. An MDP is typically defined as a quintuple. ,in The state space represents all the states that an agent may be in in the environment. This represents the action space, which is all the actions that an agent can choose in each state; It is the state transition probability function, representing the probability of the agent transitioning from state to state. Execute action Then, it transitions to the next state with a certain probability. The probability distribution; It is the reward function, representing the agent's state change. After the action Transfer to The instant reward received at that time; It is a discount factor used to measure the importance of future rewards relative to current rewards, and also to ensure that cumulative rewards are limited; in MDP, the agent's goal is to achieve the desired results through the policy. Select an action.

[0020]

[0021] This indicates the evaluation of strategies in reinforcement learning. The expected return, of which It is a strategy The performance metric or target value characterizes the expected long-term cumulative reward that the agent can obtain when sampling state and action sequences in accordance with the policy; Representing state and actions It is based on strategy The defined probability distribution is sampled, meaning that each action is selected according to the policy in the current state; From the start time Until the end time The total of the rewards within, of which Indicates at time step Execute action The instant reward received, and Discount factor The power of is used to reduce the impact of long-term rewards, making the agent focus more on short-term gains. The goal of this formula is to maximize the expected cumulative discounted reward, thereby learning the optimal strategy.

[0022] In a standard single-task reinforcement learning setting, the agent at time... Action selection depends on the current state and learned strategies conduct;

[0023]

[0024] in, Indicates the current state. This represents the learning strategy; however, in multi-task scenarios, in order to adapt to the geometry of different objects, each shape category with fundamental differences is treated as an independent task.

[0025] Furthermore, the specific content applied to the reward function in S2 is as follows:

[0026] First, to alleviate the exploration difficulties caused by sparse rewards, dense reward items are introduced, such as hand-object distance reward, posture consistency reward, and contact formation reward, so that the strategy can obtain effective learning signals in the early stages of grasping.

[0027] Secondly, considering the multi-stage characteristics of grasping tasks, a staged reward structure was designed, breaking down the task into four stages: "approaching - enveloping - applying force - lifting". Each stage is provided with independent and distinguishable rewards to guide the strategy to learn complex skills step by step in the correct order. In addition, rewards related to grasping quality were added, including force closure quality, object posture deviation penalty and stable contact score, to ensure that the final grasping action has higher reliability and stability.

[0028] Finally, motion smoothness and energy consumption penalties are added to prevent the strategy from exhibiting jitter, invalid movements, or other undesirable behaviors.

[0029] Through the above-mentioned multi-level reward reconstruction, the improved PPO framework significantly enhances the learning efficiency, grasping stability and generalization ability of expert policies, laying a higher quality foundation for subsequent GSL distillation and MoE extension.

[0030] The total reward function at each time step can be mathematically expressed as:

[0031]

[0032] In reinforcement learning tasks, the overall reward will be... It is decomposed into a weighted combination of rewards for multiple sub-goals, where Rewards for the approach phase are used to encourage agents to approach target objects or locations quickly and accurately. The reward for the contact phase is used to incentivize the agent to correctly establish stable contact or successfully grasp the object. and These are the weight coefficients corresponding to the reward items, used to adjust the importance of different stages or different objectives in the training process. By adjusting these weights, the optimization objectives of the strategy between approaching the target and establishing contact can be balanced, making the training process more stable and efficient, and ultimately improving the task success rate and strategy performance.

[0033] To encourage the finger to gradually move closer to the target object, a proximity reward based on Euclidean distance is introduced:

[0034]

[0035] This formula describes how the "proximity reward" is calculated in dexterous hand grasping tasks, encouraging the robotic finger to gradually approach the target object during the grasping process. The formula... This indicates the number of fingers involved in the grasping action. Indicates the first The actual distance between the finger and the target object, and This indicates the maximum effective distance threshold set; when the finger is far from the object, that is... ≥ The value inside the parentheses is negative, after... The reward becomes after restrictions This means the system will not award a reward; however, as the finger gradually approaches the object, that is...

[0036] At that time, the reward increases linearly as the distance decreases; the closer the finger is to the object, the closer the reward is to its maximum value. Therefore, the reward mechanism can guide the robotic arm to maintain a stable movement towards the target in the early stages of grasping, avoid ineffective movements, and improve grasping efficiency and stability.

[0037] Considering that the strength of contact significantly affects grasping stability, a distinction is introduced between weak and strong contact: This formula describes the contact reward in dexterity hand grasping tasks, encouraging fingers to establish effective contact with the target object. In the formula... This indicates the number of fingers involved in the grasping action, while Indicates the first The distance between the fingers and the surface of an object is crucial during a grasping action. When the fingers are very close to the surface and a firm contact is established, that is... When <0.01m, the system provides A larger reward is given to encourage stable contact; when the finger is in a position that is close but not yet in full contact, 0.01m < If the value is less than 0.02m, a smaller reward will be given. This guides the finger to continue moving closer to the object; if the finger distance exceeds a threshold, If the distance is greater than 0.02m, the reward is 0, indicating that this long-distance action is meaningless for grasping the target. The overall design reflects a segmented contact incentive strategy. By distinguishing the reward levels of close contact and near contact, the reinforcement learning strategy can more easily transition from coarse proximity to precise contact, thereby improving the stability and success rate in the dexterous grasping process.

[0038]

[0039] Once training converges, the encoder parameters are frozen, and a set of expert policies are initialized using the parameters of the base policy. Each expert policy is then fine-tuned on its corresponding shape-specific class data to focus on the operational behavior for a specific geometric class.

[0040] Furthermore, in step 3,

[0041] To enhance the generalization ability of policies in multi-task and multi-geometric scenarios, a Generalist-Specialist Learning framework is introduced, which deeply integrates reinforcement learning with demonstration-based imitation learning to form a stable, efficient reinforcement learning training paradigm with structured knowledge transfer capabilities.

[0042] The Generalist-Specialist Learning framework achieves simultaneous improvement in policy performance and generalization ability through a cyclical three-stage training process of "generalist first, then specialist, and then generalist back". It is particularly suitable for reinforcement learning scenarios with high task diversity and complex action space, such as robot manipulation and dexterous hand grasping.

[0043] Specifically, in the initial training phase, GSL first trains a generalist policy using a complete task set containing all task variation dimensions, enabling it to learn basic skills shared across tasks. As training progresses, when the generalist policy reaches a performance plateau in overall task performance and is difficult to improve further, GSL clones multiple specialist policies from the current generalist policy parameters. Based on the structure of the task space, the original task set is divided into several subsets with the same characteristics or geometric attributes. Each specialist policy is continuously optimized only on its respective subset, enabling it to quickly obtain high-quality policy performance in a more local, stable, and less complex learning environment, achieving in-depth mining of fine-grained operational skills. After the specialist policies are trained, GSL will backfeed the generalist policy with the high-quality experience, action trajectories, or policy distribution information obtained by these specialists in their respective task subsets through policy distillation, behavior cloning, or advantage-weighted update knowledge integration mechanisms. This allows the generalist to absorb the local optimal capabilities of multiple specialists, completing an overall leap in policy performance, and ultimately forming a unified policy with excellent performance and strong generalization ability across all tasks.

[0044] In the training process, the initial generalist model obtained in the first stage uses its parameter checkpoint as the unified starting point for subsequent expert training. In order to capture the diversity of the environment, the environmental task space is divided into several subsets with structural similarity, and an independent expert group is launched for each subset. Each expert is initialized from the shared generalist checkpoint and then undergoes adaptive reinforcement learning training in the corresponding subdomain, thereby forming a more specialized policy on a specific task distribution. After training, demonstration trajectories are collected from all experts, and auxiliary reward signals are constructed based on these high-quality expert behaviors, so that the generalist model can fully absorb expert knowledge in subsequent training stages, improve policy quality and convergence efficiency.

[0045] Furthermore, the hybrid expert structure in step 4 is a neural network structure that completes the task by modeling in parallel by multiple experts and selectively activating some experts using a dynamic routing mechanism. Its core objective is to improve computational efficiency and task expressive ability while ensuring that the model capacity is large enough. In MoE, the network contains multiple expert networks with different functions and good at handling different sub-tasks or different data patterns, as well as a gating network for determining which experts should perform processing for each sample based on the input features.

[0046] During training, the input data is first passed through a gating network to generate a set of weight distributions, representing the importance or fit of each expert to the current input. Then, only the top k experts with the highest weights are selected to perform forward computation, thereby achieving sparse activation. This allows the model to maintain a large number of experts to enhance expressive power while significantly reducing computational overhead. Finally, the outputs of the selected experts are weighted and summed according to the gating weights to form the final model response.

[0047] During the backpropagation phase, gradients only flow to activated experts and gating networks, thereby enabling task-adaptive expert division of labor learning. At the same time, to avoid all data being concentrated on a few experts, load balancing loss is usually added during training to encourage gating networks to uniformly schedule experts, thereby improving training stability and generalization ability.

[0048] Dynamic routing plays a key role in this process. It enables the model to select the “most suitable expert” based on the input, rather than calling all experts at once, forming a conditional computation. This allows MoE to significantly reduce computational costs while maintaining the capacity of a huge neural network, and it exhibits significant task decomposition and capability layering effects. It is particularly suitable for complex scenarios involving cross-distributed data, multi-task learning, and high-dimensional control, such as policy reinforcement learning, dexterous hand grasping, and multimodal visual language tasks.

[0049] Each expert policy is implemented as an independent neural network with an independent set of parameters. By introducing a decoupled representation learning mechanism between experts, the ability dilution problem caused by a single policy being forced to cover all geometric changes is avoided, thereby significantly improving the behavior generalization ability on different structural objects.

[0050] Number of experts The model's complexity is not directly determined by the number of samples, but rather by the inherent geometric correlation between object categories. This design avoids the risk of linear growth in model complexity with data size, ensuring that the system maintains manageable training costs even with larger datasets. In the actual implementation, three expert modules are used, all sharing a frozen point cloud encoder. This provides a consistent and stable perception infrastructure. Based on this shared coding, the three experts only need to perform fine-grained optimization and customization on their respective strategies, thereby efficiently achieving capability differentiation.

[0051] In terms of data preprocessing, we first use domain knowledge to cluster the shape features of objects, thereby dividing the data into multiple subdomains. This lays a regularized data structure foundation for the learning of specialized experts. Then, we independently train the corresponding reinforcement learning experts for each cluster, enabling them to focus on the policy optimization process for specific subdomains.

[0052] After completing expert-level policy learning, a router layer is further introduced to realize dynamic selection and allocation among experts during execution. Specifically, multiple generalist networks are first combined with a trainable router network. During runtime, the router automatically determines which generalist policy should handle the current task based on observed state characteristics, thereby realizing dynamic allocation of input and policy routing. During the training phase, the parameters of the generalist policies are fixed, and only the router network is optimized, so that it can gradually learn the optimal expert selection and routing policies while maintaining stable policy expression capabilities.

[0053] To more clearly describe the mathematical mechanism of the entire hybrid expert model, the following symbol definitions and calculation process are given: Let the shared shape description vector be...

[0054]

[0055] This represents the input point cloud data. Geometric latent feature representation obtained after encoding Where P typically represents a set of 3D point clouds from the scene; and This represents the encoding function of the point cloud feature extraction network; this encoder models the local structure, geometric relationships, and positional distribution within the point cloud, transforming the original high-dimensional point cloud into a low-dimensional, compact, and highly expressive latent vector. Used for subsequent crawling planning; shape vector Input into a gating network The network consists of two multi-layer sensing mechanisms, with weight matrices as follows:

[0056]

[0057] in It is to change the input features from the original dimensions Mapped to The weight matrix of the hidden space is typically used for feature compression, embedding transformation, or dimensionality enhancement, enabling the model to obtain a more compact or expressive internal representation; while This is to replace the previous layer The weight matrix for further linear transformation, fusion, or reconstruction of the hidden features, with dimension 1. It is typically used for deep abstract feature modeling, nonlinear representation enhancement, or cross-feature interaction processing;

[0058] The two layers are used Activation function; output of the gated network go through The normalized weights for each expert are calculated as follows:

[0059]

[0060] in The gating score for the i-th expert is calculated by the gating network. Mapping scores to a positive number space improves discrimination. Summ the results from all experts to achieve normalization; No. The routing weights of each expert satisfy 0 ≤ ≤1 and .

[0061] (III) Beneficial Effects

[0062] Compared with existing technologies, this invention provides a hybrid model based on reinforcement learning and multiple experts, which has the following beneficial effects:

[0063] This invention achieves efficient generalization of dexterous hand grasping through a phased learning strategy. First, it uses reinforcement learning to train basic expert policies for dexterous operations, obtaining high-performance expert models that are targeted at different objects and operation modes. Then, under the Generalist–Specialist Learning framework, multiple expert policies are gradually distilled into more compact generalist policies, which is different from the traditional direct distillation method.

[0064] This invention introduces a generalist to construct a hierarchical knowledge transfer path between experts and generalists: the generalist is located between the expert and student models, and can provide a smooth intermediate representation between the supervision signals of different experts, effectively alleviating the supervision conflict caused by the inconsistency of goals among experts, and significantly improving the transfer stability across tasks and across object categories, thus achieving a more stable and consistent policy fusion process.

[0065] Building upon this foundation, this invention further introduces a Mixture-of-Experts architecture to enhance the expressive power of the policy and the expert division of labor mechanism. By designing an efficient MoE fusion module and superimposing a lightweight residual correction network on it, the systematic bias of MoE on boundary tasks and unseen objects is adaptively compensated. This residual mechanism not only improves the policy's adaptability to new objects, long-tailed objects, and complex geometry while maintaining the diversity and selectivity of MoE output, but also significantly reduces the overall computational burden and enhances the inference efficiency and deployability of the model in real robot systems. Attached Figure Description

[0066] Figure 1 This is a flowchart of the general-expert hybrid model for dexterous hand manipulation based on reinforcement learning in this invention;

[0067] Figure 2This is a diagram illustrating the overall architecture of the general-expert hybrid model for dexterous hand manipulation based on reinforcement learning, as described in this invention.

[0068] Figure 3 This is a schematic diagram of the PPO training expert policy structure based on reward function enhancement in the general-expert hybrid model of dexterous hand operation based on reinforcement learning in this invention;

[0069] Figure 4 This diagram illustrates a comparison of the success rate of the second-stage training of the general-expert hybrid model for dexterous hand manipulation based on reinforcement learning in this invention with other methods.

[0070] Figure 5 This diagram illustrates the comparison of the success rate of the third-stage training of the general-expert hybrid model for dexterous hand manipulation based on reinforcement learning in this invention with other methods.

[0071] Figure 6 This diagram illustrates the success rate based on experts after the first stage of training and the success rate based on specialists after the second stage of training for the general-expert hybrid model of dexterous hand manipulation based on reinforcement learning, according to the present invention.

[0072] Figure 7 This is a schematic diagram illustrating the scene of the Shadowhand dexterous hand grasping different objects in Issacgym. The top left is a milk carton, the top right is a mineral water bottle, the bottom left is a camera, and the bottom right is an apple. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] Example

[0075] like Figure 1-7 As shown, an embodiment of the present invention proposes a hybrid model based on reinforcement learning and multiple experts, which includes the following steps:

[0076] Step 1: Preparation of diverse datasets: Construct a dataset of objects with various shapes and geometric features, and combine it with a simulation model of dexterous hands for training and evaluation;

[0077] The dexterous hand simulation model is the ShadowHand, a highly biomimetic five-fingered robotic hand. Its structural design fully simulates the skeletal and joint distribution of a real human right hand, possessing over twenty degrees of freedom and capable of complex and precise grasping and manipulation movements. The thumb has a more flexible multi-level joint structure, enabling palm opposition, rotation, and flexion movements, thus supporting fine manipulation in three-dimensional space. The other four fingers are each composed of four segments connected in series: metacarpophalangeal joint, proximal, mid-shaft, and distal segment, allowing for independent flexion and coordinated grasping.

[0078] The ShadowHand simulated dexterous hand, through precise modeling of joint range, damping, friction, and collision structures, exhibits realistic motion characteristics and stable contact behavior in a physical simulation environment, enabling it to perform complex manipulation tasks such as pinching, wrapping grasping, object rotation, and flipping.

[0079] With its high degrees of freedom, multi-joint cooperative capabilities, and controllability suitable for reinforcement learning and robot manipulation research, it has become an important benchmark device for many advanced algorithms and multi-task control platforms, and is one of the most representative bionic manipulators in the field of robot dexterity manipulation and generalization strategy research. The image shows a scenario where the Shadowhand dexterous hand trains with an object in Issacgym.

[0080] This method does not require strict division of the training and test sets, but it needs to perform generalization tests on unseen objects to verify the universality and robustness of the strategy in cross-object and multi-pose scenarios.

[0081] DexGraspNet provides accurate hand pose annotations, complete object geometry information, and detailed contact point data, supporting the training and evaluation of various dexterous hand models in simulation and real-world environments. Its high diversity in object shape and grasping posture makes it an important benchmark dataset for current research on general grasping strategies, cross-object generalization capabilities, and vision-to-action mapping.

[0082] Step 2, using reward function-enhanced PPO training expert policy: Construct a general-expert hybrid model for dexterous hand manipulation based on reinforcement learning. The structure of the reward function-enhanced PPO training expert policy is as follows: Figure 3 As shown;

[0083] First, within the reinforcement learning framework, the sequential decision-making process is formalized as a Markov decision-making process. Markov Decision Process (MDP) is a mathematical framework for describing sequential decision-making in uncertain environments. It uses states, actions, state transition probabilities, reward functions, and discount factors as basic elements to characterize the interaction between an agent and its environment.

[0084] In a Markov Decision Process (MDP), an agent chooses an action based on a policy in a given state. The environment then transitions to the next state with a certain probability based on the action and returns a corresponding reward. The agent's goal is to maximize the cumulative reward by learning the optimal policy through long-term interactions. A core characteristic of MDPs is that they satisfy the Markov property, meaning the next state depends only on the current state and action and is independent of the past. A Markov Decision Process is typically defined as a quintuple. ,in The state space represents all the states that an agent may be in in the environment. This represents the action space, which is all the actions that an agent can choose in each state; It is the state transition probability function, representing the probability of the agent transitioning from state to state. Execute action Then, it transitions to the next state with a certain probability. The probability distribution; It is the reward function, representing the agent's state change. After the action Transfer to The instant reward received at that time; It is a discount factor used to measure the importance of future rewards relative to current rewards, and also to ensure that cumulative rewards are limited. In MDP, the agent's goal is to achieve the desired results through the policy. Select action,

[0085]

[0086] This indicates the evaluation of strategies in reinforcement learning. The expected return, of which It is a strategy The performance metric or target value characterizes the expected long-term cumulative reward that the agent can obtain when sampling state and action sequences in accordance with the policy; Representing state and actions It is based on strategy The defined probability distribution is sampled, meaning that each action is selected according to the policy in the current state; From the start time Until the end time The total of the rewards within, of which Indicates at time step Execute action The instant reward received, and Discount factor The power of is used to reduce the impact of long-term rewards, making the agent focus more on short-term gains. The goal of this formula is to maximize the expected cumulative discounted reward, thereby learning the optimal strategy.

[0087] In a standard single-task reinforcement learning setting, the agent at time... Action selection depends on the current state and learned strategies conduct.

[0088]

[0089] in, Indicates the current state. This represents the learning strategy. However, in multi-task scenarios, to adapt to the geometric structures of different objects, each fundamentally different shape category is treated as an independent task. In traditional multi-task reinforcement learning, different tasks often have different objective functions, reward mechanisms, or dynamic transition models, even if they share the same state-action space. In contrast, this method employs a unified reward function and consistent state representation across all tasks to achieve general learning across tasks.

[0090] In dexterous hand grasping tasks, the original PPO reinforcement learning framework typically relies on sparse and coarse-grained reward signals, such as providing feedback only upon successful or failed grasping. However, this reward design fails to reflect the complexity and multi-stage characteristics of grasping tasks, easily leading to slow convergence, inefficient exploration, and even getting stuck in local optima during training. Furthermore, strategies relying solely on terminal rewards often struggle to learn stable and reliable grasping actions because the rewards cannot characterize key factors such as contact quality, grasping posture, finger closure, and object stability. To address these issues, this study systematically optimizes the reward function of PPO.

[0091] To address these issues, firstly, to alleviate the exploration difficulties caused by sparse rewards, dense reward items are introduced, such as hand-object distance rewards, posture consistency rewards, and contact formation rewards, enabling the strategy to obtain effective learning signals in the early stages of grasping. Secondly, considering the multi-stage nature of grasping tasks, a staged reward structure is designed, breaking the task down into four stages: "approach—envelop—apply force—lift," and providing independent and distinguishable rewards for each stage to guide the strategy to learn complex skills step by step in the correct order. Furthermore, rewards related to grasping quality are added, including force closure quality, object posture deviation penalties, and stable contact scores, to ensure higher reliability and stability of the final grasping action.

[0092] Finally, motion smoothness and energy consumption penalties are added to prevent the policy from exhibiting undesirable behaviors such as jitter and invalid actions. Through the above multi-level reward reconstruction, the improved PPO framework significantly enhances the learning efficiency, grasping stability, and generalization ability of expert policies, laying a higher-quality foundation for subsequent GSL distillation and MoE extensions.

[0093] The total reward function at each time step can be mathematically expressed as:

[0094]

[0095] In reinforcement learning tasks, the overall reward will be... It is decomposed into a weighted combination of rewards for multiple sub-goals, where Rewards for the approach phase are used to encourage agents to approach target objects or locations quickly and accurately. The reward for the contact phase is used to incentivize the agent to correctly establish stable contact or successfully grasp the object. and These are the weight coefficients corresponding to the reward items, used to adjust the importance of different stages or objectives in the training process. By adjusting these weights, the optimization objectives of the strategy between approaching the target and establishing contact can be balanced, making the training process more stable and efficient, and ultimately improving the task success rate and strategy performance.

[0096] To encourage the finger to gradually move closer to the target object, a proximity reward based on Euclidean distance is introduced:

[0097]

[0098] This formula describes how the "proximity reward" is calculated in dexterous hand grasping tasks, encouraging the robotic finger to gradually approach the target object during the grasping process. In the formula... This indicates the number of fingers involved in the grasping action. Indicates the first The actual distance between the finger and the target object, and This indicates the maximum effective distance threshold. When the finger is far from the object, that is... ≥ The value inside the parentheses is negative, after... The reward becomes after restrictions This means the system will not award a reward; however, as the finger gradually approaches the object, that is...

[0099] At that time, the reward increases linearly as the distance decreases; the closer the finger is to the object, the closer the reward is to its maximum value. Therefore, this reward mechanism can guide the robotic arm to maintain a stable approach to the target in the early stages of grasping, avoiding ineffective movements and improving grasping efficiency and stability.

[0100] Considering that the strength of contact significantly affects grasping stability, a distinction is introduced between weak and strong contact: this formula describes the contact reward in dexterity hand grasping tasks, encouraging fingers to establish effective contact with the target object. In the formula... This indicates the number of fingers involved in the grasping action, while Indicates the first The distance between the fingers and the surface of the object. During a grasping action, when the fingers are very close to the object's surface and a truly tight contact is established (i.e.,... When <0.01m), the system provides A larger reward is given to encourage stable contact; when the finger is in a position that is close but not yet in complete contact (0.01m < If the value is less than 0.02m, a smaller reward will be given. Guide the finger to continue moving closer to the object; if the finger distance exceeds the threshold ( If the distance is greater than 0.02m, the reward is 0, indicating that this long-distance action is meaningless for grasping the target. The overall design embodies a segmented contact incentive strategy. By distinguishing the reward levels of close contact and near contact, the reinforcement learning strategy can more easily transition from coarse proximity to precise contact, thereby improving the stability and success rate in the dexterous grasping process.

[0101]

[0102] Once training converges, the encoder parameters are frozen, and a set of expert policies are initialized using the parameters of the base policy. Each expert policy is then fine-tuned on its corresponding shape-specific class of data to focus on operations targeting a particular geometric class.

[0103] Step 3: GSL-Driven Expert Policy Distillation and Generalist Policy Construction; To further enhance the generalization ability of policies in multi-task and multi-geometric scenarios, a Generalist-Specialist Learning framework is introduced, deeply integrating reinforcement learning with demonstration-based imitation learning to form a stable, efficient reinforcement learning training paradigm with structured knowledge transfer capabilities. The Generalist-Specialist Learning framework achieves simultaneous improvement in policy performance and generalization ability through a cyclical three-stage training process of "generalist first, then specialist, then back to generalist," making it particularly suitable for reinforcement learning scenarios with high task diversity and complex action spaces, such as robot manipulation and dexterous hand grasping. Specifically, in the initial training phase, GSL first trains a generalist policy using a complete task set containing all task variation dimensions, enabling it to learn basic skills shared across tasks. As training progresses, when the generalist policy reaches a performance plateau in overall task performance and is difficult to improve further, GSL clones multiple specialist policies from the current generalist policy parameters and divides the original task set into several sub-task sets with the same characteristics or geometric attributes according to the structure of the task space. Each specialist strategy is continuously optimized only on its respective subset, enabling it to quickly achieve high-quality policy performance in a more localized, stable, and less complex learning environment, thus achieving in-depth mining of fine-grained operational skills. After the specialist strategies are trained, GSL feeds back the high-quality experience, motion trajectories, or policy distribution information gained by these specialists in their respective task subsets to the generalist strategy through knowledge integration mechanisms such as policy distillation, behavior cloning, or advantage-weighted updates. This allows the generalist to absorb the local optimal capabilities of multiple specialists, achieving an overall leap in policy performance and ultimately forming a unified policy with excellent performance and strong generalization ability across all tasks. Through continuous iteration between generalists and specialists, GSL not only avoids the training bottleneck of a single generalist struggling to handle tasks with high distribution differences but also solves the problem of a single specialist strategy lacking cross-scenario transfer ability. It realizes a "divide and conquer" learning model for complex tasks and has been proven to significantly improve policy stability, learning efficiency, and generalization ability in multi-task, multi-object, and multi-scenario robot learning and dexterous hand grasping.

[0104] In the training process, the initial generalist model obtained in the first stage uses its parameter checkpoint as a unified starting point for subsequent expert training. To capture environmental diversity, the environmental task space is partitioned into several structurally similar subsets, and an independent expert group is initiated for each subset. Each expert is initialized from the shared generalist checkpoint and then undergoes adaptive reinforcement learning training within the corresponding subdomain, thereby forming more specialized policies on specific task distributions. After training, demonstration trajectories are collected from all experts, and auxiliary reward signals are constructed based on these high-quality expert behaviors, enabling the generalist model to fully absorb expert knowledge in subsequent training stages, improving policy quality and convergence efficiency.

[0105] Step 4: Training and Dynamic Routing Mechanism of Hybrid Expert Structure; A hybrid expert structure is a neural network architecture that uses parallel modeling by multiple experts and a dynamic routing mechanism to selectively activate some experts to complete a task. Its core objective is to improve computational efficiency and task expressiveness while ensuring sufficient model capacity. In MoE, the network includes multiple expert networks with different functions, each adept at handling different sub-tasks or data patterns, and a gating network used to determine which experts should process each sample based on input features. During training, the input data first passes through the gating network to generate a set of weight distributions, representing the importance or fit of each expert to the current input. Then, only the top k experts with the highest weights are selected for forward computation, thus achieving sparse activation. This allows the model to maintain a large number of experts to enhance expressiveness while significantly reducing computational overhead. Finally, the outputs of the selected experts are weighted and summed according to the gating weights to form the final model response. During the backpropagation phase, gradients only flow to activated experts and gating networks, enabling task-adaptive expert-based learning. Simultaneously, to avoid all data concentrating on a few experts, load balancing loss is typically incorporated into training to encourage gating networks to evenly distribute experts, improving training stability and generalization ability. Dynamic routing mechanisms play a crucial role in this process, allowing the model to select the "most suitable expert" based on the input, rather than calling all experts, thus creating a conditional computation. This enables MoE to significantly reduce computational costs while maintaining a large neural network capacity, exhibiting remarkable task decomposition and capability stratification effects. It is particularly suitable for complex scenarios such as cross-distributed data, multi-task learning, and high-dimensional control, including policy reinforcement learning, dexterous hand grasping, and multimodal visual-language tasks.

[0106] In this study, each expert policy is implemented as an independent neural network with an independent set of parameters. By introducing a decoupled representation learning mechanism between experts, the ability dilution problem caused by a single policy being forced to cover all geometric changes is avoided, thereby significantly improving the behavior generalization ability on different structural objects.

[0107] Number of experts The model size is not directly determined by the number of samples, but rather by the inherent geometric correlation between object categories. This design avoids the risk of model complexity increasing linearly with data size, ensuring that the system maintains manageable training costs even with larger datasets. In the actual implementation, three expert modules are used, sharing a frozen point cloud encoder. This provides a consistent and stable perception infrastructure. Based on this shared encoding, the three experts only need to perform fine-grained optimization and customization on their respective policy levels, thereby efficiently achieving capability differentiation.

[0108] In terms of data preprocessing, domain knowledge is first used to cluster the shape features of objects, thereby dividing the data into multiple subdomains and laying a regularized data structure foundation for the learning of specialized experts. Subsequently, reinforcement learning experts are independently trained for each cluster, enabling them to focus on the policy optimization process for specific subdomains.

[0109] After completing expert-level policy learning, a router layer is introduced to enable dynamic selection and allocation of experts during execution. Specifically, multiple generalist networks are first combined with a trainable router network. During runtime, the router automatically determines which generalist policy should handle the current task based on observed state characteristics, thus achieving dynamic input allocation and policy routing. During the training phase, the parameters of the generalist policies are fixed, and only the router network is optimized, allowing it to gradually learn the optimal expert selection and routing policies while maintaining stable policy expression capabilities.

[0110] To more clearly describe the mathematical mechanism of the entire hybrid expert model, the following symbol definitions and calculation process are given: Let the shared shape description vector be...

[0111]

[0112] This represents the input point cloud data. Geometric latent feature representation obtained after encoding Where P typically represents a set of 3D point clouds from the scene. This represents the encoding function of the point cloud feature extraction network. This encoder models the local structure, geometric relationships, and positional distribution within the point cloud, transforming the original high-dimensional point cloud into a low-dimensional, compact, and highly expressive latent vector. This is used for subsequent crawling planning. Shape vector Input into a gating network The network consists of two multi-layer sensing mechanisms, with weight matrices as follows:

[0113]

[0114] in It is to change the input features from the original dimensions Mapped to The weight matrix of the hidden space is typically used for feature compression, embedding transformation, or dimensionality enhancement, enabling the model to obtain a more compact or expressive internal representation; while This is to replace the previous layer The weight matrix for further linear transformation, fusion, or reconstruction of the hidden features, with dimension 1. It is typically used for deep abstract feature modeling, nonlinear expression enhancement, or cross-feature interaction processing.

[0115] The two layers are used Activation function. The output of the gated network. go through The normalized weights for each expert are calculated.

[0116]

[0117] in The gating score for the i-th expert is calculated by the gating network. Mapping scores to a positive number space improves discrimination. The results from all experts are summed to achieve normalization. No. The routing weights of each expert satisfy 0 ≤ ≤1 and .

[0118] This design enables the policy network to decouple perceptual input from action generation. Through structured expert division of labor and dynamic gating mechanisms, the entire system achieves stronger generalization ability and higher data utilization efficiency in multi-object and multi-shape task scenarios.

[0119] Figure 4 , Figure 5The report presents state-based results for 11,000 training objects (Train.), 141 unseen objects in the seen category (Test seenCat.), and 100 unseen objects in the unseen category (Test unseen Cat.). It shows that the proposed method consistently outperforms UniDexGrasp++ in both training phases, demonstrating strong generalization ability across different tasks and data distributions. Although the performance improvement in the second phase is slightly lower than in the first phase, the overall trend remains positive, indicating that the model gradually converges in the later stages, and the hybrid policy continues to refine the decision boundaries of the grasping action. Overall, these results fully validate the effectiveness of the method. By introducing the fusion of multiple basic policies, the superpolicy can more fully integrate the feature representations of different sub-policies during training, thereby better aligning with the optimization direction provided by the grasping planning reward, ultimately achieving a more robust and adaptive improvement in grasping performance.

[0120] Figure 6 The results of ablation experiments are presented to analyze the impact of different expert numbers on model performance. The results show that as the number of experts increases, the model's performance generally shows a steady upward trend, but the performance improvement tends to saturate after the number of experts exceeds a certain threshold. This indicates that an appropriate number of experts not only helps enhance the model's expressive power and policy resolution but also achieves a reasonable balance between computational cost and performance. Furthermore, the ablation experiments further verify the important role of multi-expert fusion in the crawling task; that is, a reasonable expert design can significantly improve the overall crawling success rate and policy stability.

[0121] Overall, Figures 4 to 6 The experimental results systematically validated the effectiveness and progressive improvement process of the proposed method. From reinforcement learning optimization and the introduction of visual policies to the adjustment of the number of experts, the design of each stage provided complementary support for the final performance improvement, demonstrating the comprehensive advantages of the proposed method in policy generalization and grasping robustness.

[0122] Please see Figure 2 This is a diagram illustrating the overall architecture of the general-expert hybrid model for dexterous hand manipulation based on reinforcement learning, as described in this invention.

[0123] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hybrid model based on reinforcement learning and multiple experts, characterized in that: Building a hybrid model based on reinforcement learning and multiple experts includes the following steps: Step 1: Preparation of diverse datasets: Construct a dataset of objects with various shapes and geometric features, and combine it with a simulation model of dexterous hands for training and evaluation; Step 2, using reward function-enhanced PPO training expert policy: constructing a general-expert hybrid model for dexterous hand manipulation based on reinforcement learning; The reinforcement learning in step 2 specifically includes: First, within the reinforcement learning framework, the sequential decision-making process is formalized as a Markov decision-making process. Markov Decision Process (MDP) is a mathematical framework for describing sequential decision-making in uncertain environments. It uses states, actions, state transition probabilities, reward functions, and discount factors as basic elements to characterize the interaction between an agent and its environment. In a given state, an agent selects an action based on a policy. The environment then transitions to the next state with a certain probability based on the action and returns a corresponding reward. The agent's goal is to maximize the cumulative reward by learning the optimal policy through long-term interaction. The core characteristic of a Markov Decision Process (MDP) is that it satisfies the Markov property, meaning that the state in the next time step depends only on the current state and actions and is independent of the past. An MDP is typically defined as a quintuple. ,in The state space represents all the states that an agent may be in in the environment. This represents the action space, which is all the actions that an agent can choose in each state; It is the state transition probability function, representing the probability of the agent transitioning from state to state. Execute action Then, it transitions to the next state with a certain probability. The probability distribution; It is the reward function, representing the agent's state change. After the action Transfer to The instant reward received at that time; It is a discount factor used to measure the importance of future rewards relative to current rewards, and also to ensure that cumulative rewards are limited; in MDP, the agent's goal is to achieve the desired results through the policy. Select an action. ; This indicates the evaluation of strategies in reinforcement learning. The expected return, of which It is a strategy The performance metric or target value characterizes the expected long-term cumulative reward that the agent can obtain when sampling state and action sequences in accordance with the policy; Representing state and actions It is based on strategy The defined probability distribution is sampled, meaning that each action is selected according to the policy in the current state; From the start time Until the end time The total of the rewards within, of which Indicates at time step Execute action The instant reward received, and Discount factor The power of is used to reduce the impact of long-term rewards, making the agent focus more on short-term gains. The goal of this formula is to maximize the expected cumulative discounted reward, thereby learning the optimal strategy. In a standard single-task reinforcement learning setting, the agent at time... Action selection depends on the current state and learned strategies conduct; ; in, Indicates the current state. This represents the learning strategy; however, in multi-task scenarios, in order to adapt to the geometry of different objects, each shape category with essential differences is treated as an independent task. Step 3: GSL-driven expert strategy distillation and generalist strategy construction; Step 4: Training and dynamic routing mechanism of hybrid expert structure MoE.

2. The hybrid model based on reinforcement learning and multiple experts according to claim 1, characterized in that: The simulation model of the dexterous hand in step 1 is the ShadowHand simulation dexterous hand, which is a highly biomimetic five-finger robotic hand. Its structural design fully simulates the bone and joint distribution of a real human right hand. It consists of a forearm, wrist, palm and five fingers, and has more than 20 degrees of freedom, enabling it to perform complex and precise grasping and manipulation actions. The wrist has two degrees of rotational freedom, allowing the palm to bend forward and backward and swing left and right, while the thumb has a more flexible multi-level joint structure, enabling palm opposition, rotation and bending movements, thus supporting fine manipulation in three-dimensional space; the other four fingers are composed of four segments connected in series: metacarpophalangeal joint, proximal, middle and distal segments, which can bend independently and coordinate grasping.

3. The hybrid model based on reinforcement learning and multiple experts according to claim 1, characterized in that: The object dataset is DexGraspNet, which provides accurate hand pose annotations, complete object geometry information, and detailed contact point data, supporting the training and evaluation of various dexterous hand models in a simulation environment. Its high diversity in object shape and grasping posture makes it an important benchmark dataset for current research on general grasping strategies, cross-object generalization ability, and vision-to-action mapping.

4. The hybrid model based on reinforcement learning and multiple experts according to claim 1, characterized in that: The specific content applied to the reward function in S2 is as follows: First, to alleviate the exploration difficulties caused by sparse rewards, dense reward items are introduced, which include hand-object distance reward, posture consistency reward, and contact formation reward, so that the strategy can obtain effective learning signals in the early stages of grasping. Secondly, considering the multi-stage characteristics of grasping tasks, a staged reward structure was designed, breaking down the task into four stages: "approaching - enveloping - applying force - lifting". Each stage is provided with independent and distinguishable rewards to guide the strategy to learn complex skills step by step in the correct order. In addition, rewards related to grasping quality were added, including force closure quality, object posture deviation penalty and stable contact score, to ensure that the final grasping action has higher reliability and stability. Finally, motion smoothness and energy consumption penalties are added to prevent the strategy from exhibiting jitter, invalid movements, or other undesirable behaviors. Through the above-mentioned multi-level reward reconstruction, the improved PPO framework significantly enhances the learning efficiency, grasping stability and generalization ability of expert policies, laying a higher quality foundation for subsequent GSL distillation and MoE extension. The total reward function at each time step can be mathematically expressed as: ; In reinforcement learning tasks, the overall reward will be... It is decomposed into a weighted combination of rewards for multiple sub-goals, where Rewards for the approach phase are used to encourage agents to approach target objects or locations quickly and accurately. The reward for the contact phase is used to incentivize the agent to correctly establish stable contact or successfully grasp the object. and These are the weight coefficients corresponding to the reward items, used to adjust the importance of different stages or different objectives in the training process. By adjusting these weights, the optimization objectives of the strategy between approaching the target and establishing contact can be balanced, making the training process more stable and efficient, and ultimately improving the task success rate and strategy performance. To encourage the finger to gradually move closer to the target object, a proximity reward based on Euclidean distance is introduced: ; This formula describes how the "proximity reward" is calculated in dexterous hand grasping tasks, encouraging the robotic finger to gradually approach the target object during the grasping process. The formula... This indicates the number of fingers involved in the grasping action. Indicates the first The actual distance between the finger and the target object, and This indicates the maximum effective distance threshold set; when the finger is far from the object, that is... ≥ The value inside the parentheses is negative, after... The reward becomes after restrictions This means the system will not award a reward; however, as the finger gradually approaches the object, that is... At that time, the reward increases linearly as the distance decreases; the closer the finger is to the object, the closer the reward is to its maximum value. Therefore, the reward mechanism can guide the robotic arm to maintain a stable movement towards the target in the early stages of grasping, avoid ineffective movements, and improve grasping efficiency and stability. Considering that the strength of contact significantly affects grasping stability, a distinction is introduced between weak and strong contact: This formula describes the contact reward in dexterity hand grasping tasks, encouraging fingers to establish effective contact with the target object. In the formula... This indicates the number of fingers involved in the grasping action, while Indicates the first The distance between the fingers and the surface of an object is crucial during a grasping action. When the fingers are very close to the surface and a firm contact is established, that is... When <0.01m, the system provides A larger reward is given to encourage stable contact; when the finger is in a position that is close but not yet in full contact, 0.01m < If the value is less than 0.02m, a smaller reward will be given. This guides the finger to continue moving closer to the object; if the finger distance exceeds a threshold, If the distance is greater than 0.02m, the reward is 0, indicating that this long-distance action is meaningless for grasping the target. The overall design reflects a segmented contact incentive strategy. By distinguishing the reward levels of close contact and near contact, the reinforcement learning strategy can more easily transition from coarse proximity to precise contact, thereby improving the stability and success rate in the dexterous grasping process. ; Once training converges, the encoder parameters are frozen, and a set of expert policies are initialized using the parameters of the base policy. Each expert policy is then fine-tuned on its corresponding shape-specific class data to focus on the operational behavior for a specific geometric class.

5. The hybrid model based on reinforcement learning and multiple experts according to claim 1, characterized in that: In step 3 To enhance the generalization ability of policies in multi-task and multi-geometric scenarios, the Generalist-Specialist Learning framework is introduced, which deeply integrates reinforcement learning with demonstration-based imitation learning to form a stable, efficient reinforcement learning training paradigm with structured knowledge transfer capabilities. The Generalist-Specialist Learning framework achieves simultaneous improvement in policy performance and generalization ability through a cyclical three-stage training process of "first generalist, then specialist, and then back to generalist". It is particularly suitable for reinforcement learning scenarios with high task diversity and complex action space, such as robot manipulation and dexterous hand grasping. Specifically, in the initial training phase, GSL first trains a generalist policy using a complete task set containing all task variation dimensions, enabling it to learn basic skills shared across tasks. As training progresses, when the generalist policy reaches a performance plateau in overall task performance and is difficult to improve further, GSL clones multiple specialist policies from the current generalist policy parameters. Based on the structure of the task space, the original task set is divided into several subsets with the same characteristics or geometric attributes. Each specialist policy is continuously optimized only on its respective subset, enabling it to quickly obtain high-quality policy performance in a more local, stable, and less complex learning environment, achieving in-depth mining of fine-grained operational skills. After the specialist policies are trained, GSL will backfeed the generalist policy with the high-quality experience, action trajectories, or policy distribution information obtained by these specialists in their respective task subsets through policy distillation, behavior cloning, or advantage-weighted update knowledge integration mechanisms. This allows the generalist to absorb the local optimal capabilities of multiple specialists, completing an overall leap in policy performance, and ultimately forming a unified policy with excellent performance and strong generalization ability across all tasks. In the training process, the initial generalist model obtained in the first stage uses its parameter checkpoint as the unified starting point for subsequent expert training. In order to capture the diversity of the environment, the environmental task space is divided into several subsets with structural similarity, and an independent expert group is launched for each subset. Each expert is initialized from the shared generalist checkpoint and then undergoes adaptive reinforcement learning training in the corresponding subdomain, thereby forming a more specialized policy on a specific task distribution. After training, demonstration trajectories are collected from all experts, and auxiliary reward signals are constructed based on these high-quality expert behaviors, so that the generalist model can fully absorb expert knowledge in subsequent training stages, improve policy quality and convergence efficiency.

6. The hybrid model based on reinforcement learning and multiple experts according to claim 1, characterized in that: The hybrid expert structure described in step 4 is a neural network structure that completes tasks by modeling multiple experts in parallel and selectively activating some experts using a dynamic routing mechanism. Its core objective is to improve computational efficiency and task expressiveness while ensuring that the model capacity is large enough. In MoE, the network contains multiple expert networks with different functions that are good at handling different sub-tasks or different data patterns, as well as a gating network for determining which experts should perform processing for each sample based on the input features. During training, the input data is first passed through a gating network to generate a set of weight distributions, representing the importance or fit of each expert to the current input. Then, only the top k experts with the highest weights are selected to perform forward computation, thereby achieving sparse activation. This allows the model to maintain a large number of experts to enhance expressive power while significantly reducing computational overhead. Finally, the outputs of the selected experts are weighted and summed according to the gating weights to form the final model response. During the backpropagation phase, gradients only flow to activated experts and gating networks, thereby enabling task-adaptive expert division of labor learning. At the same time, to avoid all data being concentrated on a few experts, load balancing loss is usually added during training to encourage gating networks to uniformly schedule experts, thereby improving training stability and generalization ability. Dynamic routing plays a key role in this process. It enables the model to select the "most suitable expert" based on the input, rather than calling all experts at once, forming a conditional computation. This allows MoE to significantly reduce computational costs while maintaining the capacity of a huge neural network, and it exhibits significant task decomposition and capability layering effects. It is particularly suitable for complex scenarios involving cross-distributed data, multi-task learning, and high-dimensional control, such as policy reinforcement learning, dexterous hand grasping, and multimodal visual language tasks. Each expert policy is implemented as an independent neural network with an independent set of parameters. By introducing a decoupled representation learning mechanism between experts, the ability dilution problem caused by a single policy being forced to cover all geometric changes is avoided, thereby significantly improving the behavior generalization ability on different structural objects. Number of experts The model's complexity is not directly determined by the number of samples, but rather by the inherent geometric correlation between object categories. This design avoids the risk of linear growth in model complexity with data size, ensuring that the system maintains manageable training costs even with larger datasets. In the actual implementation, three expert modules are used, all sharing a frozen point cloud encoder. This provides a consistent and stable perception infrastructure. Based on this shared coding, the three experts only need to perform fine-grained optimization and customization on their respective strategies, thereby efficiently achieving capability differentiation. In terms of data preprocessing, we first use domain knowledge to cluster the shape features of objects, thereby dividing the data into multiple subdomains. This lays a regularized data structure foundation for the learning of specialized experts. Then, we independently train the corresponding reinforcement learning experts for each cluster, enabling them to focus on the policy optimization process for specific subdomains. After completing expert-level policy learning, a router layer is introduced to enable dynamic selection and allocation among experts during execution. Specifically, multiple generalist networks are first combined with a trainable router network. During operation, the router automatically determines which generalist policy should handle the current task based on observed state characteristics, thereby achieving dynamic input allocation and policy routing. During the training phase, the parameters of the generalist policy are fixed, and only the router network is optimized, so that it can gradually learn the optimal expert selection and traffic splitting policy while maintaining stable policy expression capabilities. To more clearly describe the mathematical mechanism of the entire hybrid expert model, the following symbol definitions and calculation process are given: Let the shared shape description vector be... ; This represents the input point cloud data. Geometric latent feature representation obtained after encoding Where P typically represents a set of 3D point clouds from the scene; and This represents the encoding function of the point cloud feature extraction network; this encoder models the local structure, geometric relationships, and positional distribution within the point cloud, transforming the original high-dimensional point cloud into a low-dimensional, compact, and highly expressive latent vector. Used for subsequent crawling planning; shape vector Input into a gating network The network consists of two multi-layer sensing mechanisms, with weight matrices as follows: ; in It is to change the input features from the original dimensions Mapped to The weight matrix of the hidden space is typically used for feature compression, embedding transformation, or dimensionality enhancement, enabling the model to obtain a more compact or expressive internal representation; while This is to replace the previous layer The weight matrix for further linear transformation, fusion, or reconstruction of the hidden features, with dimension 1. It is typically used for deep abstract feature modeling, nonlinear representation enhancement, or cross-feature interaction processing; The two layers are used Activation function; output of the gated network go through The normalized weights for each expert are calculated as follows: ; in The gating score for the i-th expert is calculated by the gating network. Mapping scores to a positive number space improves discrimination. Summ the results from all experts to achieve normalization; No. The routing weights of each expert satisfy 0 ≤ ≤1 and .

Images