A Course-Based Training Method for Balancing and Handling Robotic Arms

Abstract

This invention discloses a training method for balanced handling of a robotic arm based on curriculum learning. The method includes breaking down the hybrid constraint training for balanced handling into several sub-tasks that can be progressively learned and used for single-constraint training, and constructing a task reward function for the current sub-task. Based on the initial base model and task reward function of the current sub-task, the robotic arm's motion transitions are controlled to move its end effector towards the desired target. The state transition information corresponding to each motion transition is stored in an experience replay buffer. The initial base model is updated based on the state transition information in the experience replay buffer. The success rate of the updated initial base model is evaluated, and based on the success rate, the method determines whether to continue training the initial base model for the current sub-task, train the initial base model for the next sub-task, and generate a balanced handling model. This invention significantly improves training efficiency and success rate, effectively solving complex handling tasks that require simultaneous satisfaction of position accuracy, attitude constraints, and stability requirements.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically to a training method for balancing and transporting robotic arms based on course learning. Background Technology

[0002] In fields such as automated warehousing, logistics sorting, and smart manufacturing, robotic arms need to perform complex balancing and handling tasks. Typical scenarios include transferring items from conveyor belts to containers, handling fragile items on production lines, and precise placement in confined spaces. These tasks impose three constraints on robotic arms: target position accuracy (usually requiring centimeter-level or even millimeter-level accuracy), end effector attitude control (such as maintaining a horizontal position to prevent items from tipping over), and dynamic stability of the handling process (preventing items from falling during movement).

[0003] Reinforcement learning, as a mainstream method in the field of robot control, learns the optimal policy through trial and error in interaction between the agent and the environment. However, traditional reinforcement learning faces the following technical challenges when dealing with multi-constraint balance transport tasks:

[0004] Low sample efficiency: When position, pose, and balance constraints coexist, the successful states that satisfy all three constraints simultaneously are extremely sparse in the state space. The agent needs to optimize three interdependent objectives simultaneously, making it difficult to obtain effective learning signals. In traditional methods, the agent may need millions of interactions to obtain the first successful sample, resulting in extremely slow training convergence or even complete failure.

[0005] Difficulty in Reward Trade-offs: In multi-constraint tasks, the importance and learning difficulty of different constraints vary significantly. Using a reward function with fixed weights can easily lead to the following problems: if the weight is too small, the agent ignores the constraint, resulting in insufficient learning; if the weight is too large, the agent optimizes the constraint too early, sacrificing other objectives and getting stuck in a local optimum. For example, if the pose weight is set too high in the early stages of training, the agent will focus too much on pose adjustment and struggle to learn basic positional reach.

[0006] Target Substitution Failure: Hindsight Experience Replay (HER) is an important technique for improving sample efficiency in goal-oriented tasks. Its core idea is to replace the target of a failed experience with the actually reached state, thereby transforming "failure" into "success" and increasing the number of effective training samples. However, in multi-constraint tasks, HER faces a serious target substitution failure problem: when the agent's position is achieved but the pose is failed, traditional HER sets this state as the new target, but this target itself violates the pose constraints, causing the generated virtual experience to mislead the learning direction. This failure is particularly prominent in tasks with strong pose constraints and can even lead to a significant decline in training performance.

[0007] Sparse rewards, which only provide a positive reward upon complete success of the task, offer a clear learning objective. However, in tasks with multiple constraints, the rewards become too sparse, making it difficult for the agent to discover the successful path through random exploration. Dense rewards, which provide continuous rewards based on distance from the target, offer a learning gradient but are prone to introducing misleading signals—for example, the agent might learn to approach the target quickly but ignore pose constraints. Existing methods typically employ a single type of reward mechanism, failing to balance exploration efficiency with precise control.

[0008] Insufficient physical constraint modeling: Existing methods mostly use simple geometric thresholds to judge the falling of objects, which cannot accurately reflect the real physical process, resulting in a difference between training and actual application. Summary of the Invention

[0009] The purpose of this invention is to provide a training method for robotic arm balance handling based on curriculum learning. This method integrates multiple stages such as constraint decomposition, reward design, experience generation, and physical modeling, thereby improving the training efficiency of multi-constraint balance handling tasks. The technical solution of this invention is as follows:

[0010] A training method for balancing and handling robotic arms based on curriculum learning, comprising:

[0011] Furthermore, the hybrid constraints include position constraints, attitude constraints, and balance constraints, and the training sequence of the subtasks includes, in turn, the first subtask of performing position constraint training, the second subtask of performing attitude constraint training, and the third subtask of performing balance constraint training.

[0012] Furthermore, the constraint reward function for the first subtask is:

[0013] ,

[0014] in, This represents the Euclidean distance between the desired target position and the actual position of the robotic arm's end effector. This represents the Euclidean distance threshold, and r represents the preset value.

[0015] Furthermore, the constrained reward function for the second subtask is:

[0016] ,

[0017] in, This represents the quaternion distance between the desired target pose and the actual pose of the robotic arm's end effector. Indicates the attitude weighting coefficient;

[0018] The reward function for the second subtask is:

[0019] ,

[0020] in, Indicates the location reward value. This represents the posture reward value.

[0021] Furthermore, the constraint reward function for the third subtask is:

[0022] ,

[0023] ,

[0024] , ,

[0025] ,

[0026] = ,

[0027] ,

[0028] in, Indicates the basic penalty. Indicates continuous punishment. This indicates the penalty value for the comfort angle zone. This indicates the penalty value for the warning angle zone. This indicates the penalty value for the dangerous angle zone. This indicates the tilt angle at the end of the robotic arm. , and These represent the base penalty coefficients for the corresponding angle regions. and These represent the upper limit angle values ​​for the comfort angle zone and the warning angle zone, respectively. This indicates the number of steps within the sliding window where the tilt angle exceeds the upper limit of the warning angle zone. This indicates that the sliding window contains a number of steps. This represents the threshold number of steps required to trigger a continuous penalty. Indicates the continuous penalty coefficient;

[0029] The reward function for the third subtask is:

[0030] .

[0031] Furthermore, the state transition information includes a current observation vector representing the current state of the robotic arm. , representing the motion vector for the robotic arm's motion transformation. Reward Value , representing the next observation vector of the robotic arm's next state. , indicates the end of the last step in a round of transportation. ;

[0032] The current observation vector Next observation vector All include the actual posture information of the robotic arm. Actual pose information of the robotic arm end effector Desired pose information of the robotic arm end effector .

[0033] Furthermore, the model training includes a friction angle drop detection strategy: constructing a friction angle detection criterion to detect whether an object slips or flips, a contact detection criterion to detect whether an object falls and collides, and an energy analysis criterion to detect whether the kinetic and potential energy of the object meets the physical conditions for falling; determining whether the friction angle detection criterion is triggered, and arbitrarily triggering the contact detection criterion or the energy analysis criterion, if yes, the item falls; if no, the item does not fall.

[0034] Determine whether to award a reward value based on the results of the friction angle drop detection strategy. Replace with terminal drop penalty and effective / termination mark .

[0035] Furthermore, the model training includes task-aware adaptive HER; generating virtual targets, acquiring state transition information of all action transitions in a failed transport round, filtering out several valid state transition information where the item did not fall, determining whether there is valid state transition information within the target area; if yes, several valid state transition information within the target area are defined as candidate targets; otherwise, several valid state transition information closest to the target area are defined as candidate targets; and using a two-layer hybrid sampling strategy to filter candidate targets to generate virtual targets. ;

[0036] Generating Virtual Experiences: Based on Virtual Goals Redetermine the next virtual observation vector within the state transition information. Virtual reward value Generate virtual experiences with the same format as state transition information. And store it in the experience playback buffer.

[0037] Furthermore, the next step virtual observation vector Call the task reward function and virtual target of the current subtask. Recalculate virtual reward value .

[0038] Furthermore, the model training step includes a progressive fusion stage, which includes a progressive weight calculation function for progressively adjusting the weight coefficients of the constraint reward function within the task reward function:

[0039] ,

[0040] in, This indicates the current weight of the constraint reward function within the task reward function of the current subtask; t represents the current number of robot arm motion transitions; This indicates the number of times the robotic arm's motion transitions at the start of the progressive fusion phase of the current subtask; This indicates the initial weight of the constraint reward function within the constraint reward function corresponding to the current subtask; This indicates the target weight of the constraint reward function in the current subtask; This represents the total number of robotic arm motion transitions within the progressive fusion phase. This indicates the percentage of times the robotic arm's actions transition between the progressive fusion stage and the current subtask; This indicates the total number of robot arm motion transitions in the current subtask.

[0041] The beneficial effects of this invention are:

[0042] By decomposing the complex multi-constraint balance transport training task into multiple progressive single-constraint learning subtasks, the model avoids directly learning a high-difficulty training task at the beginning, which leads to low efficiency in generating effective samples and slow training convergence in the early stage of training.

[0043] By setting a progressive fusion stage, when subtasks are switched progressively, the weights of each constraint reward function within the task reward function are gradually adjusted based on the progressive weight calculation function. This avoids the sudden change in the value of the task reward function when the subtask is switched, which could cause the model to crash and reduce the instability of the training process.

[0044] By setting a hierarchical reward mechanism (different reward mechanisms for different sub-tasks), the position constraint reward function is a sparse reward, which can explore more unknown areas during model training. The posture and balance constraint reward functions are dense rewards, which can linearly guide the posture of the robotic arm end to maintain balance, effectively balancing exploration efficiency and precise control, and reducing the probability of the model encountering local optima.

[0045] The fall penalty is determined by setting a friction angle detection criterion to detect whether an object slips or flips, a contact detection criterion to detect whether an object falls and collides, and an energy analysis criterion to determine whether the object's kinetic and potential energy meet the physical conditions for falling. The multi-criteria detection mechanism comprehensively considers three dimensions: friction angle, contact, and energy. Compared with simple geometric threshold judgment, it effectively narrows the gap between training and actual application and improves the accuracy of model training.

[0046] During training, candidate targets are determined based on the position of the robotic arm's end effector and the object drop detection. Virtual targets are then selected from these candidates to avoid the problem of virtual target generation failure caused by the increase in constraints during training under strong pose constraints (second or third subtask). The selection criteria for the object drop detection ensure that all candidate targets meet the pose requirements of the training, avoiding misleading the model's learning direction and improving the model's ability to grasp and learn effective information. The end effector position detection criteria guarantee the generation of a certain number of candidate targets, improving the efficiency of virtual target generation and thus improving the model's convergence speed.

[0047] Calculate virtual reward value based on virtual target At that time, the format of virtual experience is The sparse reward with position constraint is used for the success determination and relabeling of virtual targets. The explicit "failure" and "success" signals of the sparse reward enable the model to quickly learn the position arrival first. When the dense reward with attitude and balance constraint is used for target relabeling, its continuous guidance signal gradually improves attitude and balance control while maintaining position capability, thus realizing the functional decoupling of the two reward mechanisms. Attached Figure Description

[0048] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0049] Figure 1 This is a task environment diagram of a robotic arm balance handling training method based on course learning according to the present invention;

[0050] Figure 2 This is a schematic diagram of a robotic arm balance handling training method based on course learning according to the present invention.

[0051] Figure 3 This is a learning diagram of different sub-tasks in a robotic arm balance handling training method based on course learning according to the present invention.

[0052] Figure 4 This is a training trajectory diagram of a robotic arm balance handling training method based on course learning according to the present invention;

[0053] Figure 5 This is a trajectory diagram of a round of handling in the third subtask of a robotic arm balance handling training method based on course learning according to the present invention.

[0054] Among them, 1. Expected target; 2. Item; 3. Conveyor belt; 4. End effector; 5. Robotic arm. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0056] A course-based training method for robotic arm balance handling, such as... Figure 1 As shown, the task environment required for model training includes the items to be transported, a conveyor belt for placing the items, a robotic arm for transporting the items, and a container for transferring the items to their destination (the area where the desired target is located). The end effector of the robotic arm is in the form of a flat pallet, used to receive items falling from the conveyor belt.

[0057] To facilitate the observation of the robotic arm's state during training, observation dimensions for recording the robotic arm's state are defined within the task environment. These observation dimensions include the robotic arm's actual posture information, the robotic arm's end effector's actual pose information, and the robotic arm's end effector's desired pose information.

[0058] The observation dimensions of the robotic arm's state are organized using a dictionary structure, and the observation dimensions include the actual posture information of the robotic arm. Actual pose information of the robotic arm end effector Desired pose information of the robotic arm end effector . , and The observation vector is obtained after splicing. .

[0059] One embodiment of this application is: the actual posture information of the robotic arm. It contains 21-dimensional observation vectors. Used to fully describe the motion state of the robotic arm and the pose of the end effector.

[0060] ,

[0061] Among them, joint position vector Includes the current position angles or displacements of the robot arm's seven joints, which is a direct description of the robot arm's posture configuration; joint velocity vectors. This includes the current angular or linear velocities of the robotic arm's seven joints, providing information about the robotic arm's motion state, which is crucial for learning smooth trajectory control; end effector position... Represents the three-dimensional spatial coordinates of the end effector in the world coordinate system. attitude quaternions of the end effector In quaternion form Quaternions represent the spatial orientation of the end effector. Compared to Euler angles, quaternions avoid gimbal lock problems and have better mathematical properties in interpolation and gradient calculation.

[0062] Desired pose information of the robotic arm end effector for

[0063] ,

[0064] in, Specifies the three-dimensional spatial coordinates that the end effector should reach; target pose quaternion. The spatial orientation that the end effector should reach is specified.

[0065] Actual pose information of the robotic arm end effector Directly from the actual posture information of the robotic arm Extracting the end effector pose , The format obtained is consistent with the desired pose information of the robotic arm's end effector. Exactly the same.

[0066] Complete observation vector By , and The three parts are pieced together:

[0067] ,

[0068] This structured observation definition enables the model to simultaneously perceive the current state of the robotic arm, the current target reached by the end effector, and the desired target, thereby learning to guide the current state toward the desired target.

[0069] like Figure 2 As shown, the robotic arm balance handling training method includes constraint decomposition: the hybrid constraints of the robotic arm are decomposed into several individual constraints, and a corresponding sub-task for training individual constraints is constructed based on the number of individual constraints and the training order.

[0070] like Figure 3As shown, the first row of diagrams illustrates the learning process for positional constraints, the second row for pose constraints, and the third row for pose constraints. Skill decomposition results in hybrid constraints, including positional, pose, and balance constraints. This transforms a single, complex training exercise into three progressively learning tasks, with each sub-task requiring the learning of only one constraint. From simply learning to reach the target position, to learning to reach the target pose, and then to learning to maintain balance during transport, this progressive learning approach, compared to end-to-end training that simultaneously learns multiple constraints, avoids the cold start problem caused by the difficulty in gaining successful experience in the early stages of training. This allows the model to converge faster and effectively improves the training rate.

[0071] In the process of progressively training the model through several sub-tasks of single-constraint training, the learning difficulty is gradually increased from simple to complex, and the difficulty of generating virtual learning samples also gradually increases. Compared with directly generating virtual samples that simultaneously meet multiple constraints, this can effectively improve the generation efficiency of virtual training samples, allowing the model to learn as much training data as possible during training, thereby improving the model's convergence rate.

[0072] One embodiment of this application is: the observation vectors of all subtasks The observation dimensions are the same, which avoids the problem of inconsistent data dimensions between models when subtasks are progressive.

[0073] Generate task reward function: Based on the training order of a single constraint, the constraint reward functions of the first subtask to the current subtask are sequentially superimposed to generate the task reward function of the current subtask.

[0074] Based on the progressive training principle of balanced handling training, the training sequence of constraints is determined. The training sequence of subtasks includes the first subtask of performing position constraint training, the second subtask of performing posture constraint training, and the third subtask of performing balance constraint training. Therefore, the first subtask only includes position constraints, used to learn how to reach the target position; the second subtask includes both position and posture constraints, building upon learning how to reach the target position, and further learning the handling posture of the end effector; the third subtask includes position, posture, and balance constraints, building upon learning how to reach the target position and the handling posture, and further learning how to maintain the horizontal stability of the running end effector during handling to prevent the item from falling. By gradually increasing the learning difficulty and comprehensiveness of the subtasks, the entire training process of learning balanced handling is finally completed.

[0075] The position-constrained reward function for the positional task uses a sparse reward mechanism: the core idea of ​​sparse rewards is to provide binary success and failure signals, avoiding misleading gradients that dense rewards may introduce; when calculating virtual reward values ​​based on virtual targets, the binary reward signal can generate virtual reward values ​​with a clear transition from "failure" to "success," providing the model with clear and unbiased learning signals. Given the actual target of the robotic arm's end effector... and expected goals The calculation of the task reward function first requires extracting the location component (actual location). With the target location ), actual location With the target location Euclidean distance between Defined as:

[0076] ,

[0077] in, These are the actual three-dimensional position coordinates of the end effector. The three-dimensional position coordinates of the target location.

[0078] The expression for the location-based task reward function is:

[0079] ,

[0080] in, This indicates the set reward value. Indicates the location reward value. This indicates the location threshold. Location threshold As subtasks are progressively learned, adjustments are made to gradually improve the accuracy of positional arrival during the progressive learning process.

[0081] One embodiment of this application is: a position threshold for the first task stage. With a precision of 0.05, it can quickly learn to roughly reach the target position, meeting the accuracy requirements of the initial learning while effectively controlling the learning difficulty. As the sub-tasks progress, the second sub-task is adjusted to 0.03m, and the third sub-task is finally tightened to 0.01m to ensure that the high precision requirements are ultimately met.

[0082] The pose-constrained reward function for the pose task uses a dense reward form to provide continuous learning gradients for the second subtask. The pose error is quantified using quaternion distance, a standard method for measuring the difference between two poses in a 3D rotational space. Given the actual pose quaternion of the robotic arm's end effector... and target attitude quaternion Quaternion distance The calculation formula is:

[0083] ,

[0084] in, This indicates that the absolute value is taken after the quaternion dot product operation. The reason for taking the absolute value is that quaternions have a double covering property, i.e. and To represent the same three-dimensional rotation, taking the absolute value can eliminate this symbolic ambiguity. This represents the cosine function.

[0085] The expression for the posture task reward function is:

[0086] ,

[0087] in, This represents the posture reward value. This represents the attitude weighting coefficients. This reflects the importance of attitude accuracy in the task and provides a clear direction for model control and learning optimization.

[0088] The balance task reward function uses a dense reward form to provide continuous learning gradients for the third subtask. The balance error is measured based on the tilt angle of the end effector relative to the horizontal plane. Calculate the tilt angle. The tilt angle is defined as the local Z-axis direction of the robotic arm's end effector (pallet), that is, the angle between the pallet's normal vector and the vertical direction of the world coordinate system. Physically speaking, this angle reflects the degree to which the pallet deviates from a horizontal state: when the pallet is perfectly horizontal, the tilt angle is 0; the more the pallet is tilted, the larger the tilt angle.

[0089] The actual posture quaternion of the robotic arm end effector The attitude matrix is ​​represented, and the third column vector of the attitude matrix is ​​extracted. This vector represents the direction of the Z-axis of the end effector in the world coordinate system, and the tilt angle. The calculation formula is:

[0090] ,

[0091] in, It is the vertical unit vector in the world coordinate system.

[0092] To accurately reflect the impact of pallet tilt on item stability, the balancing task reward system employs a three-tiered, progressive tilt penalty mechanism, combining basic penalties with progressive penalties. The expression for the balancing task reward function is as follows:

[0093] ,

[0094] in, Indicates the basic penalty. Indicates continuous punishment;

[0095] Basic Penalties The formula is:

[0096] ,

[0097] , ,

[0098] ,

[0099] = ,

[0100] in, This indicates the penalty value for the comfort angle zone. This indicates the penalty value for the warning angle zone. This indicates the penalty value for the dangerous angle zone. , and These represent the base penalty coefficients for the corresponding angle regions. and These represent the upper limit angle values ​​for the comfort angle zone and the warning angle zone, respectively.

[0101] One embodiment of this application is as follows: and The calculation formula is:

[0102] ,

[0103] ,

[0104] in, Indicates the friction angle threshold. and These represent the upper limit angle coefficients for the comfort angle zone and the warning angle zone, respectively.

[0105] One embodiment of this application is as follows: It is 19°. , and The values ​​are 0.05, 0.15, and 2.0 respectively. and Base penalty when the values ​​are 0.4 and 0.8 respectively The formula is:

[0106] (0°~0.4×) ≈ ):

[0107] ,

[0108] When the tilt angle of the robotic arm's end effector is within the comfort angle range, only the penalty value of the comfort angle range is introduced. The base penalty changes slowly and linearly to apply a slight linear penalty, allowing for small tilts during normal movement while gently guiding the robotic arm to remain horizontal.

[0109] (0.4×) ≈ ~0.8× ≈15°):

[0110] when hour: ,

[0111] when hour: ,

[0112] When the tilt angle of the robotic arm's end effector is in the warning angle zone, a new penalty value for the warning angle zone is added. This penalty value increases more sharply in a linear fashion to quickly guide the robotic arm to reduce the tilt angle.

[0113] when hour: :

[0114] when hour: ,

[0115] When the tilt angle of the robotic arm's end effector exceeds 15°, a penalty value is added to the danger angle zone. This penalty value increases quadratically and is used to strongly prevent the tilt angle of the robotic arm's end effector from approaching the friction angle threshold. (19°). The squared growth property ensures that the penalty increases sharply with the angle, effectively reducing the risk of items falling.

[0116] Continuous punishment The formula is:

[0117] ,

[0118] in, This indicates the number of steps within the sliding window where the tilt angle exceeds the upper limit of the warning angle zone. This indicates that the sliding window contains a number of steps. This represents the threshold number of steps required to trigger a continuous penalty. This represents the continuous penalty coefficient.

[0119] One embodiment of this application is: sliding window size. Set to 8 steps, trigger threshold Set to 5 steps, with a continuous penalty coefficient of 0.5.

[0120] The reward function for the first subtask is:

[0121] ,

[0122] The success reward value is set to 0.0, and the failure reward value is set to -1.0. The task reward function of the first subtask guides the learning of more regions through explicit success and failure signals. The first subtask does not assess posture and balance, so the model can focus on learning point-to-point motion during training.

[0123] The reward function for the second subtask is:

[0124] ,

[0125] The success criteria for the second subtask are more stringent than those for the first, adding posture constraints while maintaining positional accuracy. Sparse rewards are used for position to maintain clear success signals, while dense rewards are used for posture to provide fine-grained guidance (the changes in the reward value based on the task reward function continuously guide the robotic arm's movements). This design ensures the priority of positional control, preventing the robotic arm from losing its already mastered positional reach capabilities while learning posture control.

[0126] The sparse-dense hybrid reward design fully leverages the advantages of both reward mechanisms while avoiding their respective drawbacks. Sparse rewards for position maintain the explicitness of the primary task objective (position), preventing the agent from prematurely optimizing secondary metrics (pose) and neglecting the fundamental task requirements. Dense rewards for pose provide continuous gradient signals for pose learning, avoiding the exploration difficulties that pure sparse pose rewards might lead to—if pose also uses sparse rewards, successful states (both position and pose simultaneously achieved) become extremely sparse, making it difficult for the agent to discover successful experiences through random exploration within a reasonable timeframe. Through this hierarchical reward design, the model can first quickly learn position arrival (utilizing the explicit signals of sparse rewards), and then gradually improve pose control while maintaining position arrival capabilities (utilizing the continuous guidance of dense rewards).

[0127] The reward function for the third subtask is:

[0128] ,

[0129] The third sub-task adds a new level to the tiered rewards of the second sub-task: by adjusting the tilt angle at the end of the robotic arm. Rewards are used to further enhance posture constraints and help the robotic arm learn to maintain balance to avoid objects falling.

[0130] Added continuity penalty to the balance task reward function This ensures that the linear guidance logic controlling the robotic arm's operation remains consistent with the guidance logic for preventing items from falling before the end effector reaches the target area in the third subtask, thus preventing the model from learning dangerous behaviors such as "brief high tilt but insufficient continuity." Even if a single tilt does not lead to an immediate fall, a sustained high tilt state will be penalized, guiding the robotic arm to learn a more robust handling strategy.

[0131] Model training: Based on the initial base model and task reward function of the current sub-task, control the robotic arm's motion transitions to move its end effector toward the desired target. Store the state transition information corresponding to each motion transition, which includes the corresponding reward value, in the experience replay buffer. Update the initial base model based on the state transition information in the experience replay buffer.

[0132] State transition information provides a complete description of the robotic arm's transition process from one state to another. State transition information includes the current observation vector representing the robotic arm's current state. , representing the motion vector for the robotic arm's motion transformation. Reward Value , representing the next observation vector of the robotic arm's next state. Termination mark The termination marker True if: the robotic arm's end effector successfully and smoothly reaches the target position and orientation; the robotic arm reaches its maximum number of steps; or the item falls from the tray. Current observation vector. Next observation vector They have the same structure and both include the actual posture information of the robotic arm. Actual pose information of the robotic arm end effector Desired pose information of the robotic arm end effector .

[0133] The motion conversion process of the robotic arm is as follows: the initial basic model is based on the current observation vector. and reward value Determine the action vector for the action transition in the current step. Based on action vectors Control the robotic arm to perform motion transitions; obtain the observation vector for the next step. And calculate the corresponding reward value. and the observation vector of the current step Action vectors Reward Value Observation vector and termination mark Store to the experience replay buffer; continue based on observation vectors Reward Value Control the robotic arm's movements until the current subtask training is completed.

[0134] Model training includes a friction angle drop detection strategy: based on physical events, a friction angle detection criterion is constructed to detect whether an object slips or flips, a contact detection criterion is constructed to detect whether an object falls and collides, and an energy analysis criterion is constructed to detect whether the kinetic and potential energy of the object meets the physical conditions for falling; it is determined whether the friction angle detection criterion is triggered, and if either the contact detection criterion or the energy analysis criterion is triggered, if yes, the item falls; otherwise, the item does not fall.

[0135] The friction angle drop detection strategy determines the drop status of an item based on multiple physical criteria. Compared with simple geometric threshold judgment, it can more accurately reflect the real physical process and reduce the probability of misjudgment caused by instantaneous state disturbances.

[0136] Friction angle detection includes: obtaining the coefficient of friction of the end effector (pallet surface) of the robotic arm. And define the friction angle According to the Coulomb friction model: when the tilt angle of the pallet... Exceeding the friction angle At this time, objects on the pallet will slip or roll. Therefore, the balance constraint can be simplified to the tilt angle limitation of the pallet surface:

[0137] ,

[0138] ,

[0139] in, Indicates the friction angle threshold. This represents the safety factor, used to construct a strict friction angle threshold.

[0140] One embodiment of this application is as follows: the end effector is made of anti-slip material, the coefficient of friction between the object and the end effector is 0.8, and the safety factor is [missing information]. Then when When 0.5 = 19°, the friction angle-based detection mechanism is triggered.

[0141] Contact detection (drop collision) includes: The surface of the object has several contact points; for each contact point, the contact normal force is obtained. and contact normal vector Calculate the alignment between the contact normal and the direction of gravity. :

[0142] ,

[0143] in, This is the unit vector representing the direction of gravity.

[0144] Effective surface contact requires two conditions to be met simultaneously: contact normal force. Exceeding the minimum effective threshold, and the contact direction is basically consistent with the direction of gravity (alignment). (Below the corresponding threshold). Minimum effective contact force threshold. The alignment threshold is used to filter numerical noise from the physics engine and to ensure the correctness of the contact orientation. One embodiment of this application is: a minimum effective contact force threshold. Set the current to 2.0N and the alignment threshold to 0.85 (corresponding to an angular deviation of approximately 32 degrees); count the number of contact points with the ground. ,when At that time, the contact detection criteria are triggered.

[0145] Energy analysis (kinetic and potential energy analysis) includes: determining the impact of a fall by identifying the energy changes characteristic of free fall, including the increase in kinetic energy, decrease in potential energy, and decrease in altitude exhibited by an object during free fall. Kinetic energy includes translational kinetic energy. rotational kinetic energy Total kinetic energy (The sum of translational kinetic energy and rotational kinetic energy); potential energy includes gravitational potential energy. (Including the height of the object's center of mass, h).

[0146] The energy analysis criterion requires three conditions to be met simultaneously: kinetic energy exceeding the free-fall threshold, potential energy decreasing rapidly, and the center of mass being at a relatively low height. (Kinetic energy threshold) The potential energy reduction rate threshold is determined based on the mass of the item and the typical drop height, reflecting a clear falling trend, while the height threshold is used to exclude normal high-altitude movement.

[0147] For an object with a mass of 10 kg, the kinetic energy threshold is set to... (Equivalent to the kinetic energy of a free fall from a height of 1.5 meters), the potential energy decrease rate threshold is set to -10J / action conversion (step), and the height threshold is set to 0.8m.

[0148] To prevent the robotic arm from continuing to perform handling training and generate state transition information after the object has fallen from its end effector during model training, thus avoiding incorrect learning experiences, a terminal drop penalty is introduced. Unlike the task reward function mentioned above, Instead of a continuous reward component, it's a terminal event penalty. When the friction angle drop detection strategy determines that the suitcase has fallen, the current training round is immediately terminated, and the current step reward is replaced with a terminal penalty R= =n, termination flag Effective, where n is a preset value. It is not included in the regular task reward calculation, but is only used as an end-point penalty when a drop event occurs, ensuring that the severity of the drop event is fully reflected.

[0149] Model training includes task-aware adaptive HER; generating virtual targets; acquiring state transition information of all action transitions in a failed transport round; filtering out several valid state transition information where the item did not fall based on a friction angle drop detection strategy; determining whether there is valid state transition information within the target area; if yes, defining several valid state transition information within the target area as candidate targets; otherwise, defining several valid state transition information closest to the target area as candidate targets; and filtering candidate targets based on a two-layer hybrid sampling strategy to generate virtual targets. ;

[0150] Generating Virtual Experiences: Based on Virtual Goals Redetermine the next virtual observation vector within the state transition information. Virtual reward value Generate virtual experiences with the same format as state transition information. And store it in the experience playback buffer.

[0151] The friction angle drop detection strategy is used to eliminate state transition information that causes items to fall during a failed handling round. This ensures that the attitude and stability of effective state transition information meet training requirements. The virtual experiences generated in each subtask are related to the included training constraints, reducing the probability of generated virtual experiences misleading the learning direction. Candidate targets are generated using one of two filtering conditions: candidate targets within the target area and several candidate targets closest to the target area. This ensures that there are always usable candidate targets, improving the accuracy of virtual target detection. The generation efficiency.

[0152] The dual-layer hybrid sampling strategy includes: the main strategy (70% probability): within the range, Future is preferentially selected randomly from the candidate targets after the current step using the Future strategy. If there are no available targets after the current step, the Final strategy is used to select the target for the last step of a round. This strategy prioritizes future states within the range, accelerating the ability to learn the complete trajectory; the auxiliary strategy (30% probability): a hybrid of traditional Final / Future: with 80% probability, the Final strategy is used to select the target for the last step of a round, and with 20% probability, the Future strategy is used to randomly select targets after the current step. This strategy retains the exploratory nature of traditional HER and maintains the diversity of learning.

[0153] The target area is the placeable range of the item. In one embodiment of this application, the target area can be the reachable area of ​​the end effector contained in the container in the mission environment.

[0154] When generating virtual experience, the state transition information is used to guide the next observation. The desired pose information in the quantity is replaced with a virtual target. This could constitute a new direction for the next step of virtual observation. Then call the task reward function of the current subtask ( , or Recalculate virtual reward value Using the next step of virtual observation to... and virtual reward value Replace state transition information By using the corresponding data within the database, a virtual experience with the same format as the original experience can be generated. The virtual reward value is then stored in the experience replay buffer. The virtual reward value is calculated by calling the full task reward function of the current subtask. This ensures that the reward structure and training signals are consistent with those of real-world experiences.

[0155] Virtual reward value The calculation formula is:

[0156] when hour: (success),

[0157] when hour: (fail),

[0158] in, This represents the Euclidean distance between the actual position of the end effector and the virtual target position.

[0159] ,

[0160] Position-constrained sparse rewards are used for success determination and target relabeling in task-aware adaptive HER, providing explicit "failure" and "success" signals (virtual reward values). (From positive to negative values); dense rewards for posture and balance constraints are used for target relabeling in task-aware adaptive HER, solely for guiding training. This design preserves the polarity shift required by HER—after the virtual target is replaced, the experience of "failure" can be clearly transformed into "success," providing the agent with a clear and unbiased learning signal. If only dense rewards were used for relabeling, the smooth score changes would obscure the effectiveness of actions and undermine the core mechanism of HER.

[0161] The initial basic model is an Actor-Critic network based on SAC, and the input of the Actor-Critic network is the observation vector. The output is an action vector used for action transformation. The Q-network outputs a state-action value function Q(o, a), based on action vectors. Control the robotic arm to perform the next action transition.

[0162] The initial base model is updated as follows: when the number of state transition information accumulated in the experience replay buffer exceeds the initial observation period and meets the model update frequency requirements, a batch of state transition information is sampled from the experience replay buffer to form experience data, and multiple rounds of Actor-Critic network updates are performed based on the experience data.

[0163] The Q-network is updated as follows: The loss function of the Q-network is calculated based on empirical data and state transition information. The calculation of the Q-network loss function first requires calculating the target Q-value y.

[0164] ,

[0165] in, The reward for the current moment. Indicates the discount factor. and Let's represent two target Q-networks, where action vector a' is the action of the policy network sampled from observation vector o', and α represents the entropy coefficient. This represents the probability that the policy network (Actor network) outputs the action vector a'. Using a dual Q-network and taking the minimum value can effectively alleviate the problem of Q-value overestimation and improve training stability.

[0166] The loss functions for the two Q-networks are as follows:

[0167] ,

[0168] ,

[0169] Where N represents the batch size. The total network loss of Q is... The loss function is backpropagated and the gradients are clipped. The optimizer is then used to update the Q-network parameters.

[0170] The policy network is updated as follows: the Q-network parameters are frozen, and the action vector is sampled through the policy network based on the observation vector o in the empirical replay buffer. Based on action vectors Calculate the smaller of the two Q-networks The loss function of the policy network is The batch mean is used, and the loss includes an entropy regularization term to encourage exploration. The loss function is backpropagated and gradients are pruned. The policy network parameters are updated using the optimizer, and the Q-network parameters are unfrozen.

[0171] After the network update is complete, update the target Q network parameters using a soft update strategy:

[0172] ,

[0173] Where θ represents the current Q-network parameters, Let represent the target Q-network parameters, and τ represent the soft update coefficients. Soft updates allow the target network to slowly track the current network, providing a stable estimate of the target value.

[0174] One embodiment of this application is as follows: the discount factor γ is 0.99 during model updates, the batch size N of the state transition information is 128, and the learning rate of the initial base model for each update is 3×10. -4 The soft update coefficient τ is 0.995.

[0175] Model iteration: Evaluate the success rate of the updated initial base model; determine if the success rate meets the threshold for several consecutive iterations. If not, proceed to the model training step. If yes, determine if the current subtask is the last subtask. If not, define the updated initial base model as the initial base model for the next subtask and proceed to the model training step. If yes, define the updated initial base model as the balanced transport model.

[0176] The success rate criterion for the updated initial base model is as follows: the model executes a fixed number of test rounds using a deterministic strategy, and the percentage of rounds in which the target is successfully reached is used as the success rate. When the success rate meets the threshold requirement for several consecutive evaluations, the current subtask is considered to have been sufficiently learned, and the process can switch to the next subtask.

[0177] One embodiment of this application is as follows: the success rate threshold for the first subtask is 0.9, the success rate threshold for the second subtask is 0.85, and the success rate threshold for the third subtask is 0.9. Each evaluation consists of 100 test rounds, and the subtask is considered complete when the success rate meets the threshold for five consecutive evaluations. Each subtask has 150 to 200 training rounds, with no more than 2000 steps of robotic arm movement transitions per round. The experience replay buffer can store 100,000 state transition records.

[0178] After the current subtask is trained, the trained model (the updated initial base model) is used as the initial base model for the next subtask. The reward function of the next subtask is a new constrained reward function superimposed on the current subtask's reward function, while the observation space dimension remains unchanged, ensuring that the model parameters can be directly transferred. When all subtasks have been trained in sequence, the final model is the balanced transfer model.

[0179] like Figure 4As shown, this is a transport trajectory diagram of the robotic arm over several rounds. Through constraint decomposition mechanism, virtual experience generation mechanism, hierarchical reward mechanism (position constraints are sparse rewards, posture constraints and balance constraints are dense rewards) and progressive weighting method, the trajectory diagram trained by the model covers the entire workspace. The model learns the transport capability within the space, rather than the transport capability between points (the model learns the general transport capability within the space, rather than the memory of a specific trajectory within the space).

[0180] like Figure 5 As shown, this is a single-point transport trajectory diagram of the robotic arm. The model has learned to keep the end effector balanced during the transport process.

[0181] The model iteration includes a progressive fusion phase: when the current subtask has just started training, the weights of the constraint reward function are progressively adjusted to ensure a smooth transition in the task reward function value. This avoids drastic changes in the task reward function value caused by sudden introduction or enhancement of constraints when switching subtasks, which could lead to instability or even crashes in the training process. In this application's scheme, the balance constraint achieves escalating penalty intensity through a three-layer progressive tilt penalty mechanism (comfort zone, warning zone, danger zone), thus eliminating the need for additional weight scheduling.

[0182] The progressive fusion phase includes a progressive weighting function for progressively adjusting the weight coefficients of the constraint reward function, the formula of which is:

[0183] ,

[0184] in, This indicates the current weight of the constraint reward function within the task reward function of the current subtask; t represents the current number of robot arm motion transitions; This indicates the number of times the robotic arm's motion transitions at the start of the progressive fusion phase of the current subtask; This indicates the initial weight of the constraint reward function within the constraint reward function corresponding to the current subtask; This indicates the target weight of the constraint reward function in the current subtask; This represents the total number of robotic arm motion transitions within the progressive fusion phase. This indicates the percentage of times the robotic arm's actions transition between the progressive fusion stage and the current subtask; This indicates the total number of robot arm motion transitions in the current subtask.

[0185] One embodiment of this application is as follows: When switching to the second subtask, the initial weight of the newly introduced pose constraint is 0, the target weight is 0.5, and the number of attempts accounts for 30%, allowing the model sufficient time to adapt to the introduction of the pose constraint and avoiding the degradation of position learning ability due to the sudden application of pose penalties. When switching to the third subtask, due to the introduction of new and more complex balance constraints, the weight of the pose constraint increases from 0.5 to 1.0, and the number of attempts adopts a longer gradual process of 60%, ensuring that the pose weight smoothly transitions to the final value during the balance constraint learning process, maintaining training stability.

[0186] During model training, the current weight value is calculated according to the progressive weight calculation function, and the pose weight parameters of the environment copy in the training environment and the experience replay buffer are updated synchronously to ensure that the reward calculation and experience generation use consistent weight configuration.

[0187] The embodiments described above are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A training method for balancing and handling robotic arms based on curriculum learning, characterized in that, include, Constraint decomposition: The mixed constraints corresponding to the balance transport are decomposed into several individual constraints. Based on the number of individual constraints and the training order, corresponding sub-tasks that can be progressively learned and used for training individual constraints are constructed. To generate the task reward function, the constraint reward functions of the first subtask to the current subtask are sequentially superimposed based on the training order of a single constraint to generate the task reward function for the current subtask. Model training involves using the initial base model of the current sub-task and the task reward function to calculate the reward value. The robot arm's motion transitions are controlled to move its end effector toward the desired target. The state transition information corresponding to each motion transition, which includes the corresponding reward value, is stored in the experience replay buffer. The initial base model is updated based on the state transition information in the experience replay buffer. Model iteration: Evaluate the success rate of the updated initial base model; determine if the success rate meets the threshold for several consecutive iterations. If not, proceed to the model training step. If yes, determine if the current subtask is the last subtask. If not, define the updated initial base model as the initial base model for the next subtask and proceed to the model training step. If yes, define the updated initial base model as the balanced transport model.

2. The robotic arm balance handling training method based on course learning according to claim 1, characterized in that, The hybrid constraints include position constraints, attitude constraints, and balance constraints. The training sequence of the sub-tasks includes the first sub-task of performing position constraint training, the second sub-task of performing attitude constraint training, and the third sub-task of performing balance constraint training.

3. The robotic arm balance handling training method based on course learning according to claim 2, characterized in that, The constraint reward function for the first subtask is: ， in, This represents the Euclidean distance between the desired target position and the actual position of the robotic arm's end effector. represents the Euclidean distance threshold, and r represents the set reward value.

4. The robotic arm balance handling training method based on course learning according to claim 2, characterized in that, The constraint reward function for the second subtask is: ， in, This represents the quaternion distance between the desired target pose and the actual pose of the robotic arm's end effector. Indicates the attitude weighting coefficient; The reward function for the second subtask is: ， in, Indicates the location reward value. This represents the posture reward value.

5. The robotic arm balance handling training method based on course learning according to claim 2, characterized in that, The constraint reward function for the third sub-task is: ， ， ， ， ， = ， ， in, Indicates the basic penalty. Indicates continuous punishment. This indicates the penalty value for the comfort angle zone. This indicates the penalty value for the warning angle zone. This indicates the penalty value for the dangerous angle zone. This indicates the tilt angle at the end of the robotic arm. , and These represent the base penalty coefficients for the corresponding angle regions. and These represent the upper limit angle values ​​for the comfort angle zone and the warning angle zone, respectively. This indicates the number of steps within the sliding window where the tilt angle exceeds the upper limit of the warning angle zone. This indicates that the sliding window contains a number of steps. This represents the threshold number of steps required to trigger a continuous penalty. Indicates the continuous penalty coefficient; The task reward function for the third subtask is: ， in, Indicates the location reward value. This represents the posture reward value.

6. The robotic arm balance handling training method based on course learning according to claim 1, characterized in that, The state transition information includes a current observation vector representing the current state of the robotic arm. , representing the motion vector for the robotic arm's motion transformation. Reward Value , representing the next observation vector of the robotic arm's next state. , indicates the end of the last step in a round of transportation. ; The current observation vector Next observation vector All include the actual posture information of the robotic arm. Actual pose information of the robotic arm end effector Desired pose information of the robotic arm end effector .

7. The robotic arm balance handling training method based on course learning according to claim 6, characterized in that, The model training includes a friction angle drop detection strategy: constructing a friction angle detection criterion to detect whether an object slips or flips, a contact detection criterion to detect whether an object falls and collides, and an energy analysis criterion to detect whether the kinetic and potential energy of the object meets the physical conditions for falling; determining whether the friction angle detection criterion is triggered, and arbitrarily triggering the contact detection criterion or the energy analysis criterion, if yes, the item falls; if no, the item does not fall. Determine whether to award a reward value based on the results of the friction angle drop detection strategy. Replace with terminal drop penalty and effective / termination mark .

8. The robotic arm balance handling training method based on course learning according to claim 1, characterized in that, The model training includes task-aware adaptive HER; generating virtual targets; acquiring state transition information of all action transitions in a failed transport round; filtering out several valid state transition information where the item did not fall; determining whether there is valid state transition information within the target area; if yes, defining several valid state transition information within the target area as candidate targets; otherwise, defining several valid state transition information closest to the target area as candidate targets; and using a two-layer hybrid sampling strategy to filter candidate targets to generate virtual targets. ; Generating Virtual Experiences: Based on Virtual Goals Redetermine the next virtual observation vector within the state transition information. Virtual reward value Generate virtual experiences with the same format as state transition information. ,in, Represents the current observation vector. The motion vector represents the motion transformation of the robotic arm. This indicates the termination of the effect and is stored in the experience replay buffer.

9. A training method for balancing and handling robotic arms based on course learning according to claim 8, characterized in that, The next virtual observation vector ,in, This indicates the actual posture information of the robotic arm in the next step. This indicates the actual pose information of the robotic arm's end effector in the next step, and calls the task reward function and virtual target of the current subtask. Recalculate virtual reward value .

10. A training method for robotic arm balance handling based on course learning according to claim 1, characterized in that, The model training steps include a progressive fusion stage, which includes a progressive weight calculation function for progressively adjusting the weight coefficients of the constraint reward function within the task reward function: ， in, This indicates the current weight of the constraint reward function within the task reward function of the current subtask; t represents the current number of robot arm motion transitions; This indicates the number of times the robotic arm's motion transitions at the start of the progressive fusion phase of the current subtask; This indicates the initial weight of the constraint reward function within the constraint reward function corresponding to the current subtask; This indicates the target weight of the constraint reward function in the current subtask; This represents the total number of robotic arm motion transitions within the progressive fusion phase. This indicates the percentage of times the robotic arm's actions transition between the progressive fusion stage and the current subtask; This indicates the total number of robot arm motion transitions in the current subtask.

Images