A robot local action policy gradient calculation and update method and related device

Abstract

This application discloses a method and related apparatus for calculating and updating the local action policy gradient of a robot, used to optimize the policy gradient and improve the convergence efficiency of the policy network. The application includes: inputting the robot's current state information into the policy network and outputting action commands for each action dimension; after the robot executes the action commands, acquiring the reward signal and the state information at the next moment, and recording trajectory segments; calculating an advantage function based on the trajectory segments and the reward signal, and weighting the gradient of the policy network using the advantage function to obtain a weighted policy gradient; calculating the local importance weight of each action dimension based on the advantage function, and determining the corresponding gradient pruning threshold based on the local importance weight; pruning the components belonging to the corresponding dimension in the weighted policy gradient according to the gradient pruning threshold of each action dimension to obtain the pruned policy gradient; and updating the parameters of the policy network based on the pruned policy gradient.

Description

Technical Field

[0001] This application relates to the field of intelligent detection technology, and in particular to a method and related apparatus for calculating and updating the gradient of a robot's local motion strategy. Background Technology

[0002] With the deep integration of robotics and artificial intelligence, robot action decision-making methods based on deep reinforcement learning have attracted widespread attention. Among them, the policy gradient method has become an important technical route in the field of robot motion control because it can directly optimize the policy network parameters and is applicable to continuous action spaces.

[0003] However, the contributions of different motion dimensions to task performance often vary significantly. For example, in robotic arm operations, changes in the end-effector position dimension may directly determine the success or failure of the task, while the impact of certain joint angle dimensions is relatively limited. Existing technologies use a uniform pruning threshold to treat the gradient components of all motion dimensions equally, failing to distinguish the differences in importance between different dimensions. This leads to the gradients of critical motion dimensions, which have a significant impact on task performance, being easily over-pruned and losing important learning signals; while the gradients of non-critical dimensions, which have a smaller impact on the task, lack effective constraints, thus affecting the convergence efficiency of the policy network and the final control performance. Summary of the Invention

[0004] This application discloses a method and related apparatus for calculating and updating the local action policy gradient of a robot, which is used to optimize the policy gradient and improve the convergence efficiency of the policy network.

[0005] In a first aspect, this application discloses a method for calculating and updating the gradient of a robot's local action policy, including:

[0006] Input the robot's current state information into the policy network and output action instructions for each action dimension; After the robot executes the action command, it acquires the reward signal and the state information of the next moment, and records the trajectory segment. The reward signal is used to characterize the execution performance of the action command, and the trajectory segment is the trajectory segment from the current moment to the next moment and the trajectory segment is stored in the calculation buffer. The advantage function is calculated based on the trajectory segment and the reward signal, and the gradient of the policy network is weighted by the advantage function to obtain the weighted policy gradient. Based on the advantage function, calculate the local importance weight of each action dimension, and determine the corresponding gradient clipping threshold according to the local importance weight. According to the gradient clipping threshold of each action dimension, the components belonging to the corresponding dimension in the weighted policy gradient are clipped to obtain the clipped policy gradient. The parameters of the policy network are updated based on the pruned policy gradient.

[0007] Optionally, the step of calculating the advantage function based on the trajectory segment and the reward signal, and then weighting the gradient of the policy network using the advantage function to obtain the weighted policy gradient, includes: Based on the state information of each time step in the trajectory segment, the state value of each time step is calculated; Based on the state value and the reward signal, the advantage function estimate for each time step is calculated; The gradient of the policy network at each time step is weighted using the estimated value of the advantage function at each time step, and the weighted results of all time steps are summed to obtain the weighted policy gradient.

[0008] Optionally, the step of calculating the local importance weight of each action dimension based on the dominance function, and determining the corresponding gradient clipping threshold according to the local importance weight, includes: For each action dimension, calculate the partial derivative of the advantage function with respect to the action instruction of the action dimension in the current state, and use the partial derivative as the local importance weight of the action dimension; Based on the preset clipping threshold and the local importance weight, the gradient clipping threshold corresponding to each action dimension is determined.

[0009] Optionally, after calculating the local importance weight of each action dimension based on the dominance function and determining the corresponding gradient clipping threshold according to the local importance weight, the method further includes: Based on the current state information, the local geometric curvature of the robot in the current configuration is calculated, and the local geometric curvature is used to characterize the dexterity of the robot in the current posture. The gradient clipping threshold is corrected based on the local geometric curvature to obtain the corrected gradient clipping threshold.

[0010] Optionally, the local geometric curvature is the ellipsoidal curvature; The step of calculating the local geometric curvature of the robot in the current configuration based on the current state information includes: Based on the current state information, construct the Jacobian matrix of the robot in the current configuration; Singular value decomposition is performed on the Jacobian matrix to obtain a set of singular values. The Jacobian matrix is ​​a mapping matrix from the robot joint space to the operation space. The ellipsoid curvature is calculated based on the minimum and maximum values ​​in the set of singular values.

[0011] Optionally, after executing the action instruction, obtaining the reward signal and the next moment's state information, and recording the trajectory segment, the method further includes: Based on the preset termination conditions, determine whether the trajectory segment is an abnormal trajectory; If the trajectory segment is determined to be an abnormal trajectory, the trajectory segment is removed from the calculation buffer.

[0012] Optionally, the preset termination condition includes the reward signal falling below a preset reward lower limit threshold.

[0013] Secondly, this application provides a device for calculating and updating the gradient of a robot's local motion policy, comprising: The instruction unit is used to input the robot's current state information into the policy network and output action instructions for each action dimension. The acquisition unit is used to acquire a reward signal and the state information of the next moment after executing the action instruction, and to record the trajectory segment. The reward signal is used to characterize the execution performance of the action instruction, and the trajectory segment is the trajectory segment from the current moment to the next N moments and the trajectory segment is stored in the calculation buffer. The first computing unit is used to calculate the advantage function based on the trajectory segment and the reward signal, and to weight the gradient of the policy network with the advantage function to obtain the weighted policy gradient. The second calculation unit is used to calculate the local importance weight of each action dimension based on the advantage function, and determine the corresponding gradient clipping threshold according to the local importance weight. The pruning unit is used to prune the components belonging to the corresponding dimension in the weighted policy gradient according to the gradient pruning threshold of each action dimension, so as to obtain the pruned policy gradient. The update unit is used to update the parameters of the policy network based on the pruned policy gradient.

[0014] Thirdly, this application provides a device for calculating and updating the gradient of a robot's local motion strategy, comprising: Processor, memory, input / output units, and bus; The processor is connected to memory, input / output units, and a bus; The memory holds a program, which the processor calls to execute, as in the first aspect and any optional method of the first aspect.

[0015] Fourthly, this application provides a computer-readable storage medium on which a program is stored, and when the program is executed on a computer, it performs the methods described in the first aspect and any optional methods of the first aspect.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The dominance function is derived from trajectory segments and reward signals, and policy gradient weighted optimization is implemented based on the dominance function, which can accurately distinguish the execution quality of different action instructions. Adaptive weight allocation is performed using the actual performance represented by the dominance function, effectively improving the effectiveness and relevance of the policy gradient, avoiding interference from ineffective action gradients in network optimization, and enhancing the accuracy of policy learning.

[0017] The quantitative evaluation based on the local importance weight of the action dimension can set gradient pruning thresholds differently according to the actual task contribution of each action dimension. This breaks the rigid mode of sharing a uniform pruning threshold for all action dimensions in traditional technologies, allowing key core action dimensions to obtain a larger gradient update amplitude, ensuring the full optimization of core operational capabilities, while constraining the gradient update of secondary action dimensions to avoid gradient oscillation and parameter erratic updates.

[0018] By employing dimensional independent gradient pruning, precise constraints and optimizations are applied to the components of the weighted policy gradient across different dimensions. This effectively eliminates anomalous noise and abrupt changes in the gradient, preserving only valid gradient information and significantly improving the stability and reliability of the policy gradient. Updating the policy network parameters based on the pruned policy gradient effectively accelerates the convergence speed of the policy network, enhancing the accuracy and operational stability of the robot's action decisions. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an embodiment of the robot local motion strategy gradient calculation and update method in this application; Figure 2 This is a schematic diagram of an embodiment for calculating the advantage function and weighted policy gradient in this application; Figure 3 This is a schematic diagram of an embodiment for obtaining the gradient clipping threshold in this application; Figure 4 This is a schematic diagram of an embodiment for calculating local geometric curvature in this application; Figure 5 This is a schematic diagram of an embodiment for determining abnormal trajectories in this application; Figure 6 This is a schematic diagram of the robot local motion strategy gradient calculation and update device in this application; Figure 7 This is another schematic diagram of the robot local motion strategy gradient calculation and update device in this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not an embodiment," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] Based on this, this application discloses a method and related apparatus for calculating and updating the local action policy gradient of a robot, which is used to optimize the policy gradient and improve the convergence efficiency of the policy network.

[0028] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] The method of this application can be applied to servers, devices, terminals, or other devices with logical processing capabilities; therefore, this application does not limit its application. For ease of description, the following description uses a system as the executing entity.

[0030] Please see Figure 1 This application provides an embodiment of a method for calculating and updating the gradient of a robot's local motion policy, comprising: 101. Input the robot's current state information into the policy network and output action instructions for each action dimension; The robot collects complete status information in real time at the current working moment. This status information can comprehensively reflect the robot's own motion state and the current working environment state, specifically including the robot's joint angles, joint motion angular velocities, joint output torques, robot end effector pose, attitude data collected by the inertial measurement unit, and environmental perception data.

[0031] The collected current state information is used as real-time input data for the policy network. Through feature extraction, feature mapping, and decision reasoning logic within the policy network, synchronous decision outputs are made for all motion dimensions of the robot. Among them, motion dimensions correspond to different degrees of freedom of motion control and task execution dimensions of the robot, including various motion dimensions such as joint rotation, end effector translation, and posture adjustment. Finally, the independent motion command corresponding to each motion dimension is output.

[0032] 102. After the robot executes the action command, it acquires the reward signal and the state information of the next moment, and records the trajectory segment. The reward signal is used to characterize the execution performance of the action command, and the trajectory segment is the trajectory segment from the current moment to the next moment and the trajectory segment is stored in the calculation buffer. After receiving the action instructions from the policy network, the robot synchronously executes the corresponding motion and operation actions to complete the single-step action interaction process.

[0033] After the action command is executed, the system collects the corresponding reward signal and the robot's state information at the next moment from the working environment in real time. The reward signal is a quantitative evaluation indicator used to accurately characterize the execution effect, operational accuracy, motion stability, and task completion degree of each action dimension at the current moment. It is the core evaluation basis for subsequent strategy gradient optimization and performance assessment. Simultaneously, the system fully records the entire process of robot motion and state changes from the current decision moment to the next moment, forming a single-step continuous trajectory segment. This trajectory segment is then uniformly stored in the device's built-in computing buffer, achieving temporary caching and ordered storage of trajectory data.

[0034] 103. Calculate the advantage function based on the trajectory segments and reward signals, and use the advantage function to weight the gradient of the policy network to obtain the weighted policy gradient; The algorithm retrieves the current time-stamped trajectory segment and the corresponding environmental reward signal stored in the computation buffer, and solves for the advantage function under the current state and action matching. The advantage function can objectively quantify the superiority of the action instructions output by the current policy network compared to the average action policy, effectively distinguishing between high-quality action policies and inefficient or biased action policies, and accurately reflecting the actual value of each action execution.

[0035] Based on the obtained advantage function, the original policy gradient of the policy network is weighted and corrected. The gradient weights are adaptively allocated through the advantage value, weakening the gradient weights corresponding to inefficient actions and strengthening the gradient weights corresponding to high-quality and effective actions. Finally, a weighted policy gradient that integrates the information of the quality of action execution is obtained, completing the initial optimization of the gradient.

[0036] 104. Based on the dominance function, calculate the local importance weight for each action dimension, and determine the corresponding gradient clipping threshold based on the local importance weight. Based on the advantage function obtained in step 103, an importance analysis is performed on each independent action dimension of the robot, and the local importance weight of each action dimension in the current work scenario and current state is calculated separately. The weight can represent the degree of contribution of different action dimensions to the robot's task completion, work accuracy, and motion stability, thereby realizing the quantitative differentiation of the value of each action dimension.

[0037] Based on the local importance weights corresponding to each action dimension, an adaptive matching and setting of a dedicated gradient pruning threshold is performed. The threshold size is positively matched with the dimension importance. That is, action dimensions with high task contribution and high task criticality are matched with a larger gradient pruning threshold, allowing for a larger parameter update range; action dimensions with low task contribution and auxiliary function are matched with a smaller gradient pruning threshold, strictly constraining the gradient update range, thus achieving personalized adaptation of the gradient constraint standard for each action dimension.

[0038] 105. According to the gradient clipping threshold of each action dimension, clip the components belonging to the corresponding dimension in the weighted policy gradient to obtain the clipped policy gradient. For the weighted policy gradient obtained in step 103, it is decomposed into components according to each action dimension to accurately distinguish the gradient components corresponding to different action dimensions in the weighted policy gradient. Based on the exclusive gradient pruning threshold preset for each action dimension in step 104, the gradient components of each dimension are pruned and constrained one by one. Invalid interference components such as gradient mutation, gradient anomaly, and magnitude out of bounds in each dimension gradient are specifically removed, while the effective and high-quality update components in each dimension gradient are retained to avoid the problem of over-constraint or under-constraint of gradient caused by a uniform pruning method.

[0039] After completing gradient clipping for all action dimensions, the optimized gradient components of each dimension are integrated to obtain a clipped policy gradient that is stable overall and adaptable to the characteristics of each dimension's operation.

[0040] 106. Update the parameters of the policy network based on the pruned policy gradient.

[0041] The policy gradient, after being optimized through dimensional adaptive pruning, is iteratively corrected and updated according to a preset network update rule, affecting all weight and bias parameters of the policy network. By updating the policy gradient, the decision-making and reasoning logic of the policy network is optimized. This completes a single round of policy gradient calculation and network iteration, and can continuously enter the next round of state acquisition, action decision-making, and gradient optimization iteration, achieving continuous adaptive optimization of the robot's action policy.

[0042] In this embodiment, the advantage function is solved based on trajectory segments and reward signals, and policy gradient weighted optimization is implemented based on the advantage function, which can accurately distinguish the execution quality of different action instructions. The weights are adaptively allocated using the actual job performance represented by the advantage function, effectively improving the effectiveness and relevance of the policy gradient, avoiding interference from invalid action gradients in network optimization, and improving the accuracy of policy learning.

[0043] The quantitative evaluation based on the local importance weight of the action dimension can set gradient pruning thresholds differently according to the actual task contribution of each action dimension. This breaks the rigid mode of sharing a uniform pruning threshold for all action dimensions in traditional technologies, allowing key core action dimensions to obtain a larger gradient update amplitude, ensuring the full optimization of core operational capabilities, while constraining the gradient update of secondary action dimensions to avoid gradient oscillation and parameter erratic updates.

[0044] By employing dimensional independent gradient pruning, precise constraints and optimizations are applied to the components of the weighted policy gradient across different dimensions. This effectively eliminates anomalous noise and abrupt changes in the gradient, preserving only valid gradient information and significantly improving the stability and reliability of the policy gradient. Updating the policy network parameters based on the pruned policy gradient effectively accelerates the convergence speed of the policy network, enhancing the accuracy and operational stability of the robot's action decisions.

[0045] Please see Figure 2 This application provides an embodiment for calculating the advantage function and the weighted policy gradient, including: 201. Calculate the state value of each time step based on the state information of each time step in the trajectory segment; After acquiring the robot trajectory segment stored in the computation buffer, the trajectory segment is decomposed in time sequence, and the state information corresponding to each time step in the trajectory segment is extracted. The state information of each time step completely records the robot's motion state, pose state and environmental state at different times in a single interaction cycle.

[0046] Based on the preset state value assessment logic, the state characteristics of each time step are analyzed and value fitted. Each time step is individually quantitatively evaluated to calculate the state value of the corresponding time step. The state value is used to quantitatively characterize the expected long-term cumulative return that the robot's current time step state can bring, and can objectively judge the quality of the state itself and its potential for continuous completion of subsequent tasks.

[0047] 202. Based on the state value and reward signal, the estimated value of the dominance function at each time step is calculated; After obtaining the state values ​​for all time steps, the advantage difference is calculated step-by-step by combining the reward signals collected in real time after the action is executed at each time step. Specifically, through the coupled calculation of reward signals and state values, the corresponding advantage function estimate is obtained step-by-step. This advantage function estimate can accurately quantify the excess return of the action executed at the current time step compared to the average expected action in that state, and can truly reflect the actual contribution and optimization space of a single action, distinguishing the differentiated performance of high-quality actions from low-quality actions.

[0048] 203. Using the advantage function estimate at each time step as the weight, the gradient of the policy network at the corresponding time step is weighted, and the weighted results of all time steps are summed to obtain the weighted policy gradient.

[0049] The advantage function estimate obtained at each time step is used as the corresponding adaptive weight coefficient to perform weighted correction on the original policy gradient at the corresponding time step. For time steps where the advantage function estimate is positive and the action reward is better than expected, the update weight of the corresponding gradient is amplified to strengthen the optimization of policy parameters corresponding to high-quality actions; for time steps where the advantage function estimate is negative and the action performance is worse than expected, the weight of the corresponding gradient is weakened to suppress the negative impact of ineffective and poor-quality actions on the policy network.

[0050] After completing the gradient weighting processing for all time steps, the weighted gradient results for all time steps are summed in time series, and all effective gradient information of the entire trajectory segment is merged to finally obtain the overall weighted policy gradient, thus completing the fine preprocessing of the policy gradient.

[0051] In this embodiment, the state value of trajectory segments is solved step-by-step. By introducing state value to represent the long-term expected return of a state, both instantaneous operational effectiveness and future task benefits are considered, enabling a more comprehensive and objective evaluation of the actual value of each time-series state of the robot. Based on the coupling of state value and reward signal to solve for the estimated value of the dominance function step-by-step, the excess reward of each action at each time step can be distinguished, accurately exploring the optimization space of each action at each time sequence. This effectively identifies high-quality and effective actions and inefficient and biased actions in the trajectory, achieving a quantitative distinction between the quality of actions and solving the problem that traditional solutions cannot finely identify the contribution of actions at each time sequence.

[0052] By employing a time-step advantage weighted gradient and temporal accumulation method, differentiated temporal optimization of the policy gradient is achieved. Gradient updates are strengthened for high-reward temporal actions, while gradient interference is suppressed for low-reward temporal actions. This amplifies the optimization effect of high-quality learning samples, making the final weighted policy gradient more closely match the robot's actual optimal operation strategy, and significantly improving the effectiveness and learning direction of the policy gradient.

[0053] Meanwhile, by accumulating and fusing gradient information from the entire trajectory segment over time, the randomness and one-sidedness of gradient updates at a single time step are avoided, improving the stability and robustness of the overall weighted policy gradient. This makes the policy network parameter updates more accurate and convergence more stable, further enhancing the accuracy and dynamic adaptation capabilities of robot policy learning.

[0054] Please see Figure 3 This application provides an embodiment for obtaining a gradient clipping threshold, including: 301. For each action dimension, calculate the partial derivative of the advantage function with respect to the action instruction of the action dimension in the current state, and use the partial derivative as the local importance weight of the action dimension. After obtaining the estimated values ​​of the dominance function at each time step, importance quantification is performed for all independent action dimensions of the robot. Using the robot's current state information as the calculation benchmark, the partial derivative of the dominance function with respect to the action command variable for each action dimension is calculated. This partial derivative can intuitively reflect the sensitivity and gradient of the dominance function as the corresponding action dimension command changes.

[0055] The larger the partial derivative value of a certain action dimension, the greater the contribution of adjusting that action dimension to the quality of the task and the level of benefit, indicating that it belongs to the key core dimension of robot operation; conversely, the smaller the value, the smaller the impact of that action dimension on the overall task effect, indicating that it belongs to a secondary auxiliary dimension. This step directly uses the obtained partial derivatives of each dimension as the local importance weight of each action dimension, achieving accurate, quantitative, and adaptive evaluation of the task contribution of each action dimension.

[0056] 302. Determine the gradient clipping threshold for each action dimension based on the preset clipping threshold and local importance weight; The preset pruning threshold is used to limit the baseline range of gradient updates, avoiding excessively large or small overall gradient update magnitudes and ensuring the basic stability of network iteration.

[0057] Based on a system-preset pruning threshold combined with differentiated local importance weights for each action dimension, dynamic replication is performed on each action dimension to establish a positive correlation between weights and pruning thresholds. Core action dimensions with higher local importance weights are matched with larger gradient pruning thresholds, allowing for larger gradient parameter update amplitudes; auxiliary action dimensions with lower local importance weights are matched with smaller gradient pruning thresholds, constraining their gradient update range and preventing invalid updates of secondary dimensions from interfering with overall strategy optimization.

[0058] 303. Based on the current state information, calculate the local geometric curvature of the robot in the current configuration. The local geometric curvature is used to characterize the robot's dexterity in the current posture. Different joint extension, folding, and deflection configurations of a robot correspond to different motion redundancy, operational dexterity, and motion fault tolerance. Local geometric curvature is a quantitative characterization index of the robot's current posture geometric characteristics and motion performance.

[0059] Based on the robot's current state information, including geometric state parameters such as joint angles, joint rotation configurations, end effector pose, and link spatial positions, the local geometric curvature corresponding to the robot's current overall motion configuration is calculated.

[0060] The specific calculation process is described in detail in subsequent embodiments and will not be repeated here.

[0061] 304. The gradient clipping threshold is corrected based on the local geometric curvature to obtain the corrected gradient clipping threshold.

[0062] After calculating the local geometric curvature of the robot's current configuration in step 303, if the local geometric curvature of the robot's current configuration is high, the posture is highly dexterous, and the motion adjustment space is sufficient, the gradient clipping threshold of the corresponding action dimension is appropriately increased. If the robot's current configuration has low local geometric curvature, restricted posture, low motion tolerance, and is close to a singular configuration, the gradient clipping threshold should be appropriately reduced to constrain the gradient update amplitude and avoid problems such as robot motion jitter, sudden posture changes, and motion over-limit caused by large parameter updates.

[0063] Threshold correction is achieved through real-time feedback of geometric posture, ultimately obtaining a corrected gradient clipping threshold that adapts to the robot's current posture and fits real-time motion performance, thus realizing dynamic adaptation between gradient constraint standards and the robot's physical configuration.

[0064] In this embodiment, the partial derivative of the advantage function with respect to the action instruction is used as the local importance weight of the dimension, realizing adaptive quantitative evaluation of the importance of each action dimension. Based on the matching of the basic threshold and the local importance weight with the dimension-specific pruning threshold, the dimension-differentiated optimization of gradient constraints is realized. Gradient update restrictions are relaxed for core high-contribution action dimensions, while ineffective gradient updates are constrained for secondary action dimensions, effectively suppressing gradient oscillations and redundant parameter updates. This solves the dual drawbacks of insufficient optimization of core dimensions and excessive updates of secondary dimensions caused by the traditional uniform threshold, significantly improving the targeting and effectiveness of policy gradient updates.

[0065] By introducing local geometric curvature quantification to assess current posture dexterity, the robot's physical configuration features are integrated into the policy gradient optimization process. Through dynamic correction of the gradient clipping threshold using geometric curvature, real-time dynamic adaptive adjustment of gradient constraint criteria is achieved, effectively improving the robot's actual motion stability and posture adaptability.

[0066] Local geometric curvature can be ellipsoidal curvature; the robot's ellipsoid can intuitively map the ability to transfer velocities from joint space to end effector space. The regularity and anisotropic scale differences of the ellipsoid directly correspond to the robot's dexterity, redundancy, and degree of constraint in different motion directions, and can accurately characterize the robot's real-time motion performance. Please refer to [link / reference]. Figure 4 This application provides an embodiment for calculating local geometric curvature, including: 401. Based on the current state information, construct the Jacobian matrix of the robot in the current configuration; The Jacobian matrix is ​​the core transformation matrix for mapping robot joint space to maneuver space. It can establish a linear mapping relationship between robot joint motion velocity and motion increment and end effector pose velocity and pose increment.

[0067] Based on the robot's current state information, such as joint angles, joint link structural parameters, joint offset parameters, and end-effector pose information, and combined with the robot's forward kinematic mapping relationship, a Jacobian matrix corresponding to the robot's current configuration is constructed. Specifically, firstly, the actual rotation angles of all joints at the current moment are read, and combined with the robot's DH parameters, the spatial pose transformation relationship of each link is derived link by link, solving for the differential influence relationship of each joint motion on the spatial position and attitude of the link end. Then, based on the linear mapping principle of small motion increments in joint space and small pose increments in robot end-effector operation space, the motion mapping coefficients corresponding to each joint are solved column by column and dimension by dimension, determining the independent driving contribution and coupling correlation of each joint degree of freedom to the translational and rotational axes of the end effector. Finally, the mapping coefficients corresponding to all joints are arranged in a dimensional regularized manner to form a complete Jacobian matrix from joint space to operation space.

[0068] 402. Perform singular value decomposition on the Jacobian matrix to obtain the set of singular values. The Jacobian matrix is ​​a mapping matrix from the robot joint space to the operation space. Singular value decomposition can decompose high-dimensional motion mapping matrices into features, extracting core feature parameters that characterize the robot's motion transmission capability in various directions.

[0069] For the Jacobian matrix under the current configuration, singular value decomposition is performed to obtain all the singular values ​​corresponding to the Jacobian matrix, forming a complete set of singular values. Each singular value in this set corresponds to the motion transmission gain and motion capability of the robot's end effector in different motion directions in the operating space. The magnitude of the singular value directly corresponds to the motion sensitivity, transmission efficiency, and operational controllability in the corresponding direction, and can comprehensively reflect the differences in motion performance in various dimensions of the robot under the current posture.

[0070] 403. The curvature of the ellipsoid is calculated based on the minimum and maximum values ​​in the set of singular values.

[0071] After obtaining the complete set of singular values, all singular values ​​within the set are filtered and compared to extract the maximum and minimum singular values. The maximum singular value corresponds to the motion direction with the highest motion transmission efficiency and the strongest operational flexibility under the robot's current configuration, representing the optimal motion dimension performance under the robot's current posture; the minimum singular value corresponds to the weakest motion dimension with the lowest motion transmission efficiency, the most significant posture limitation, and the easiest to approach motion singularity.

[0072] Coupled computation is performed based on the ratio between the maximum and minimum singular values. Combining the geometric construction principles of the robot ellipsoid and curvature quantification rules, the ellipsoid curvature under the current configuration is calculated. This curvature value can quantitatively characterize the regularity of the ellipsoid: the better the curvature value, the more regular the ellipsoid shape, the more balanced the robot's anisotropy, the higher its posture dexterity, and the further it is from singular configurations; the worse the curvature value, the higher the degree of ellipsoid distortion, the more prominent the robot's anisotropy, and the more severely restricted the motion in some dimensions. This enables precise quantification of the robot's real-time posture dexterity and motion constraint characteristics.

[0073] In this embodiment, the Jacobian matrix of the robot's current configuration is constructed in real time, dynamically establishing the motion mapping relationship between the joint space and the operation space. By performing singular value decomposition on the Jacobian matrix, the multi-dimensional motion coupling characteristics of the robot can be effectively decomposed, the controllable motion capabilities in each direction can be accurately extracted, and the advantageous and limited dimensions of the operation can be distinguished. Based on the calculation of ellipsoid curvature using singular value extrema, the robot's abstract configuration dexterity, motion isotropy, and singular approach risk can be transformed into quantitative values, providing a reliable physical basis for adaptive correction of gradient clipping threshold, and effectively improving the adaptability, stability, and control accuracy of the robot's strategy iteration.

[0074] Please see Figure 5 This application provides an embodiment for determining abnormal trajectories, including: 501. Based on the preset termination conditions, determine whether the trajectory segment is an abnormal trajectory; After the trajectory segment is cached in the computation buffer and before it participates in the policy gradient calculation, the legality and validity of a single complete time-series trajectory segment are verified.

[0075] The preset termination condition can be whether the reward signal corresponding to the detected trajectory segment is lower than a preset reward threshold. When the reward signal corresponding to the trajectory is lower than the preset threshold, it indicates that the action sequence corresponding to that trajectory segment has abnormal problems such as operation failure, motion deviation, collision risk, task failure, etc., and does not have effective learning value.

[0076] 502. If a trajectory segment is determined to be an abnormal trajectory, the trajectory segment will be removed from the calculation buffer.

[0077] After determining the anomalies in the trajectory segments, a differentiated sample screening process is performed. For failed trajectory segments that meet the anomaly determination criteria, a sample removal mechanism is triggered immediately to completely remove the abnormal trajectory segment from the computation buffer, preventing it from participating in subsequent iterations such as advantage function calculation, gradient weighted accumulation, and network parameter updates.

[0078] Normal trajectory segments that do not trigger abnormal conditions are retained in the computation buffer and participate normally in subsequent policy learning iterations.

[0079] In this embodiment, a pre-screening mechanism for policy learning samples is constructed by using a preset reward threshold to determine and selectively remove trajectory anomalies. By quantifying reward signals to identify failed trajectories, low-quality samples such as robot operation failures, motion anomalies, and action deviations can be identified. By promptly removing abnormal trajectories from the buffer, erroneous samples are blocked from participating in gradient calculation and parameter updates at the data source, effectively avoiding gradient distortion, incorrect parameter updates, and policy drift caused by abnormal trajectories, further improving the convergence stability and control accuracy of robot policy learning.

[0080] Please see Figure 6 This application provides an embodiment of a robot local motion policy gradient calculation and update device, comprising: The instruction unit 601 is used to input the robot's current state information into the policy network and output action instructions for each action dimension. The acquisition unit 602 is used to acquire the reward signal and the state information of the next moment after executing the action command, and record the trajectory segment. The reward signal is used to characterize the execution performance of the action command, and the trajectory segment is the trajectory segment from the current moment to the next N moments and the trajectory segment is stored in the calculation buffer. The first computing unit 603 is used to calculate the advantage function based on the trajectory segment and the reward signal, and to weight the gradient of the policy network with the advantage function to obtain the weighted policy gradient. The second calculation unit 604 is used to calculate the local importance weight of each action dimension based on the advantage function, and determine the corresponding gradient clipping threshold according to the local importance weight. The clipping unit 605 is used to clip the components belonging to the corresponding dimension in the weighted policy gradient according to the gradient clipping threshold of each action dimension, so as to obtain the clipped policy gradient. Update unit 606 is used to update the parameters of the policy network based on the pruned policy gradient.

[0081] For detailed implementation methods, please refer to... Figures 1 to 5 Examples are not detailed here.

[0082] Please see Figure 7 This application also provides a device for calculating and updating the gradient of a robot's local motion policy, comprising: Processor 701, memory 702, input / output unit 703, and bus 704.

[0083] The processor 701 is connected to the memory 702, the input / output unit 703, and the bus 704.

[0084] The memory 702 stores a program, and the processor 701 calls the program to execute it, such as... Figures 1 to 5 The method in the middle.

[0085] This application provides a computer-readable storage medium on which a program is stored, and the program is executed when executed on a computer. Figures 1 to 5 The method in the middle.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for calculating and updating the gradient of a robot's local motion strategy, characterized in that, include: Input the robot's current state information into the policy network and output action instructions for each action dimension; After the robot executes the action command, it acquires the reward signal and the state information of the next moment, and records the trajectory segment. The reward signal is used to characterize the execution performance of the action command, and the trajectory segment is the trajectory segment from the current moment to the next moment and the trajectory segment is stored in the calculation buffer. The advantage function is calculated based on the trajectory segment and the reward signal, and the gradient of the policy network is weighted by the advantage function to obtain the weighted policy gradient. Based on the advantage function, calculate the local importance weight of each action dimension, and determine the corresponding gradient clipping threshold according to the local importance weight. According to the gradient clipping threshold of each action dimension, the components belonging to the corresponding dimension in the weighted policy gradient are clipped to obtain the clipped policy gradient. The parameters of the policy network are updated based on the pruned policy gradient.

2. The method for calculating and updating the gradient of a robot's local motion strategy according to claim 1, characterized in that, The step of calculating the advantage function based on the trajectory segment and the reward signal, and then weighting the gradient of the policy network using the advantage function to obtain the weighted policy gradient, includes: Based on the state information of each time step in the trajectory segment, the state value of each time step is calculated; Based on the state value and the reward signal, the advantage function estimate for each time step is calculated; The gradient of the policy network at each time step is weighted using the estimated value of the advantage function at each time step, and the weighted results of all time steps are summed to obtain the weighted policy gradient.

3. The method for calculating and updating the gradient of a robot's local motion strategy according to claim 1, characterized in that, The step of calculating the local importance weight of each action dimension based on the dominance function, and determining the corresponding gradient clipping threshold based on the local importance weight, includes: For each action dimension, calculate the partial derivative of the advantage function with respect to the action instruction of the action dimension in the current state, and use the partial derivative as the local importance weight of the action dimension; Based on the preset clipping threshold and the local importance weight, the gradient clipping threshold corresponding to each action dimension is determined.

4. The method for calculating and updating the gradient of a robot's local motion strategy according to claim 3, characterized in that, After calculating the local importance weight of each action dimension based on the dominance function, and determining the corresponding gradient clipping threshold according to the local importance weight, the method further includes: Based on the current state information, the local geometric curvature of the robot in the current configuration is calculated, and the local geometric curvature is used to characterize the dexterity of the robot in the current posture. The gradient clipping threshold is corrected based on the local geometric curvature to obtain the corrected gradient clipping threshold.

5. The method for calculating and updating the gradient of a robot's local motion strategy according to claim 4, characterized in that, The local geometric curvature is the ellipsoidal curvature; The step of calculating the local geometric curvature of the robot in the current configuration based on the current state information includes: Based on the current state information, construct the Jacobian matrix of the robot in the current configuration; Singular value decomposition is performed on the Jacobian matrix to obtain a set of singular values. The Jacobian matrix is ​​a mapping matrix from the robot joint space to the operation space. The ellipsoid curvature is calculated based on the minimum and maximum values ​​in the set of singular values.

6. The method for calculating and updating the gradient of a robot's local motion strategy according to any one of claims 1 to 5, characterized in that, After executing the action command, acquiring the reward signal and the next moment's state information, and recording the trajectory segment, the method further includes: Based on the preset termination conditions, determine whether the trajectory segment is an abnormal trajectory; If the trajectory segment is determined to be an abnormal trajectory, the trajectory segment is removed from the calculation buffer.

7. The method for calculating and updating the gradient of a robot's local motion strategy according to claim 6, characterized in that, The preset termination condition includes the reward signal falling below a preset reward lower limit threshold.

8. A device for calculating and updating the gradient of a robot's local motion strategy, characterized in that, include: The instruction unit is used to input the robot's current state information into the policy network and output action instructions for each action dimension. The acquisition unit is used to acquire a reward signal and the state information of the next moment after executing the action instruction, and to record the trajectory segment. The reward signal is used to characterize the execution performance of the action instruction, and the trajectory segment is the trajectory segment from the current moment to the next N moments and the trajectory segment is stored in the calculation buffer. The first computing unit is used to calculate the advantage function based on the trajectory segment and the reward signal, and to weight the gradient of the policy network with the advantage function to obtain the weighted policy gradient. The second calculation unit is used to calculate the local importance weight of each action dimension based on the advantage function, and determine the corresponding gradient clipping threshold according to the local importance weight. The pruning unit is used to prune the components belonging to the corresponding dimension in the weighted policy gradient according to the gradient pruning threshold of each action dimension, so as to obtain the pruned policy gradient. The update unit is used to update the parameters of the policy network based on the pruned policy gradient.

9. A device for calculating and updating the gradient of a robot's local motion strategy, characterized in that, include: The processor, memory, input / output unit, and bus are connected to the memory, the input / output unit, and the bus. The memory stores a program, and the processor calls the program to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, performs the method as described in any one of claims 1 to 7.

Images