Long-time-domain robot autonomous operation method based on hierarchical reinforcement learning

Abstract

The application relates to a long-time-domain robot autonomous operation method based on layered reinforcement learning, which comprises the following steps: in a training stage: a layered reinforcement learning model is constructed; state information of a robot is acquired and a long-time-domain task chain is output, and the long-time-domain task chain is disassembled into even-order task chains and odd-order task chains which are parallel and cross; a first skill model is subjected to reinforcement learning training based on the even-order task chains to output a first action, and a second skill model is subjected to reinforcement learning training based on the odd-order task chains to output a second action; a top planning unit is subjected to reinforcement learning training based on global state information to generate an action weight for weighted combination of the first action and the second action; in an execution stage: the first action and the second action are generated based on current state information of the robot; the action weight for weighted combination of the first action and the second action is generated, and an execution action of the robot is generated. The scheme effectively ensures stability of model updating and low calculation cost in the training process.

Description

A Long-Term Autonomous Operation Method for Robots Based on Hierarchical Reinforcement Learning Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a long-term autonomous operation method for robots based on hierarchical reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, autonomous robots are gradually emerging in numerous application fields, such as chemical and materials laboratories, demonstrating enormous application potential. These fields often involve long-duration tasks, which place high demands on the efficiency, autonomy, and safety of robots. Therefore, developing reliable robot operation learning methods has become crucial for promoting the widespread application of autonomous robots in these fields.

[0003] In learning robot manipulation skills, imitation learning (IL) and reinforcement learning (RL) are currently the mainstream methods. Both methods have achieved certain results in various tasks such as grasping, handheld manipulation, and contact-rich manipulation. However, each has significant limitations.

[0004] Imitation learning relies on demonstration data to efficiently learn state-action mapping relationships, which can enable robots to quickly master specific operating patterns to a certain extent. However, its performance is severely limited by the coverage of demonstration data. When facing long-term tasks, due to the complexity and diversity of task scenarios, demonstration data can hardly fully cover all possible situations. This results in limited generalization ability of imitation learning, and the robot may not be able to make accurate and effective operations when encountering situations outside of the demonstration data.

[0005] Reinforcement learning, on the other hand, requires no demonstration data and can learn generalizable policies, theoretically enabling robots to exhibit adaptability in different environments and task scenarios. However, in long-duration tasks, reinforcement learning faces the severe challenges of a vast exploration space and sparse rewards. Long-duration tasks typically involve numerous combinations of states and actions, requiring the robot to search for the optimal policy within this enormous exploration space. This makes the training process extremely complex, time-consuming, and costly. Simultaneously, due to the sparse rewards, the robot struggles to obtain sufficient feedback to guide its exploration during the learning process, easily converging to local optima and failing to find a globally optimal policy, thus impacting the robot's overall performance in long-duration tasks.

[0006] To address these challenges, hierarchical learning has been introduced into long-duration tasks. Hierarchical learning typically employs a two-layer structure, decomposing the long-duration task into multiple sub-tasks or objectives. The lower skill layer focuses on learning the operational skills for specific sub-tasks. Through targeted learning for each sub-task, the robot can more precisely master the skills required to complete each sub-task. The upper planning layer then schedules the execution of these skills based on environmental conditions, ensuring that each sub-task is performed in a reasonable order and at the right time to complete the entire long-duration task. This task decomposition method simplifies the learning process for long-duration tasks to some extent, enabling the robot to gradually overcome each sub-task and ultimately achieve the completion of the overall task.

[0007] However, while hierarchical learning brings advantages, it also raises some new problems. On the one hand, the hierarchical structure increases the number of model parameters, which not only makes model training and optimization more difficult but may also lead to problems such as overfitting, affecting the model's generalization performance. On the other hand, due to the state similarity between lower-level operational skills, upper-level planning may become unstable when scheduling skill execution, leading to deviations in the task execution process.

[0008] More importantly, hierarchical learning faces an imbalance dilemma. When lower-level skill models are oversimplified to simplify lower-level skill learning, it increases the burden on upper-level planners. Especially when long-term tasks are decomposed into too many subtasks or objectives, the solution space that the upper-level planner needs to handle becomes excessively large. This not only increases the complexity of planning but also increases the computational burden, making it difficult for the upper-level planner to make quick and accurate decisions, thus affecting the robot's execution efficiency and effectiveness in long-term tasks.

[0009] This demonstrates that long-term robot manipulation remains a core challenge in the field of robotics. While hierarchical reinforcement learning offers a promising solution to this problem, issues such as the imbalance dilemma it presents urgently need to be addressed. Summary of the Invention

[0010] The technical problem to be solved by this invention is to provide a long-term autonomous operation method for robots based on hierarchical reinforcement learning.

[0011] To achieve the above-mentioned objectives, this invention provides a long-term autonomous robot operation method based on hierarchical reinforcement learning, comprising the following steps:

[0012] S1. Training Phase:

[0013] Construct a hierarchical reinforcement learning model, which includes: task chain decomposition unit, upper-level planning unit and lower-level skill unit;

[0014] The task chain decomposition unit acquires the robot's state information and outputs a long-term task chain, and decomposes the long-term task chain into parallel and intersecting even-order task chains and odd-order task chains.

[0015] The lower-level skill units include: a first skill model and a second skill model, and the first skill model and the second skill model respectively adopt dynamic Kolmogorov-Arnold networks;

[0016] The first skill model is trained using reinforcement learning based on an even-ordered task chain to output the first action, while the second skill model is trained using reinforcement learning based on an odd-ordered task chain to output the second action. In the reinforcement learning training of the skill models, a dynamic tanh function and a low-rank decomposition method are used to assist in the dynamic updating of the skill models.

[0017] The upper-level planning unit performs reinforcement learning training based on global state information to generate action weights that weight the first action and the second action.

[0018] S2. Execution Phase:

[0019] Based on the robot's current state information, a long-term task chain is obtained and decomposed into even-ordered task chains and odd-ordered task chains, and the first action and the second action are generated respectively.

[0020] Generate action weights that are a weighted combination of the first and second actions;

[0021] The robot's actions are generated by weighting and combining the first and second actions based on their weights.

[0022] According to one aspect of the present invention, in step S1, the first skill model and the second skill model are trained based on a policy gradient reinforcement learning method, and the first skill model and the second skill model are updated based on a skill model loss function, wherein the skill model loss function is expressed as:

[0023] ;

[0024] ;

[0025] ;

[0026] in, This represents the loss function of the skill model. Subtasks The loss function corresponding to the state, Subtasks The loss function corresponding to the state, This indicates the state of the subtask, i.e., the state information. Representing state and actions The advantage function, Representing state and actions The advantage function, Indicates in subtask The actions performed in the process, Indicates in subtask The actions performed in the process, This represents the mapping relationship from the advantage function to the loss function. Representing state The state value function under the given conditions This represents the value function of a state-action pair.

[0027] According to one aspect of the invention, the dynamic Kolmogorov-Arnold network used by the first skill model and the second skill model is consistent, wherein the dynamic Kolmogorov-Arnold network is obtained by introducing a grid dynamic update mechanism based on the Kolmogorov-Arnold network;

[0028] Under the dynamic mesh update mechanism, the dynamic Kolmogorov-Arnold network maintains the mesh accuracy when updating the mesh, and dynamically adjusts the mesh size based on the change in mesh range, and adaptively adjusts the number of learnable basis function parameters in the B-spline activation function according to the changed mesh size.

[0029] According to one aspect of the invention, the dynamic Kolmogorov-Arnold network comprises:

[0030] The input layer is used to receive the state information corresponding to the subtasks;

[0031] At least one DyKAN layer, each DyKAN layer containing multiple B-spline activation functions to map the input to the output;

[0032] The output layer is used to generate actions corresponding to the subtasks;

[0033] The grid dynamic update mechanism module is used to dynamically expand the grid of each DyKAN layer to update the number of learnable basis function parameters in the B-spline activation function;

[0034] The grid dynamic update mechanism module dynamically monitors the distribution range of the input data of each DyKAN layer in the dynamic Kolmogorov-Arnold network after each update, compares the distribution range with the grid range of the main grid, determines the newly added forward grid points and newly added backward grid points in the main grid, adds the newly added forward learnable basis function parameters corresponding to the newly added forward grid points to the preset forward parameter set, adds the newly added backward learnable basis function parameters corresponding to the newly added backward grid points to the preset backward parameter set, and periodically merges the main learnable basis function parameters corresponding to the main grid, the newly added forward learnable basis function parameters, and the backward learnable basis function parameters to complete the adaptive update of the grid size and the number of learnable basis function parameters in the B-spline activation function.

[0035] According to one aspect of the present invention, in the reinforcement learning training of a skill model, in the step of using a low-rank decomposition method to assist in the dynamic updating of the skill model, the grids of all B-spline activation functions within the same DyKAN layer are unified into a shared grid with the same size and range based on the low-rank decomposition method, which includes:

[0036] Organize all learnable basis function parameters of the DyKAN layer into a dimension of The parameter matrix; where, This represents the input dimension of the DyKAN layer. This indicates the output dimension of the DyKAN layer. This indicates the dimension of learnable basis functions in a single B-spline activation function. That is, the number of learnable basis functions;

[0037] The parameter matrix is ​​decomposed into the product of a first low-rank matrix and a second low-rank matrix, where the dimension of the first low-rank matrix is... The dimension of the second low-rank matrix is ;in, Indicates a low-rank dimension;

[0038] When the number of learnable basis functions in the B-spline activation function of the same DyKAN layer increases, only the second low-rank matrix is ​​expanded and updated, while the first low-rank matrix remains unchanged. This enables dynamic updating of the shared grid of the same DyKAN layer, which is used to match the update process of the dynamic Kolmogorov-Arnold network.

[0039] According to one aspect of the present invention, in the reinforcement learning training of a skill model, the step of using a dynamic tanh function to assist in the dynamic updating of the skill model includes:

[0040] A dynamic tanh function layer is added before the DyKAN layer, wherein the dynamic tanh function used contains learnable scaling and translation parameters.

[0041] The constraint range of the DyKAN layer input data distribution is dynamically adjusted by scaling and translation parameters to stabilize the training process and maintain the scalability of the second low-rank matrix.

[0042] The iteration parameters are set, and the second low-rank matrix is ​​dynamically expanded based on the iteration parameters and the dynamic tanh function to match the update process of the dynamic Kolmogorov-Arnold network.

[0043] The dynamic tanh function in the dynamic tanh function layer is represented as follows:

[0044] ;

[0045] in, This represents the dynamic tanh function. This indicates the output scaling parameter. This indicates the input scaling parameter. Indicates input data, This indicates the output translation parameters.

[0046] According to one aspect of the invention, in the step of setting iteration parameters and dynamically updating the skill model based on the iteration parameters and a dynamic tanh function, the iteration parameters include: the current number of training steps. Merging parameter step threshold Step increment Upper limit of shared grid range within the same DyKAN layer Lower bound of shared mesh range within the same DyKAN layer Then it includes:

[0047] S11. Construct the upper bound set for expanding the second low-rank matrix. and lower bound set ;

[0048] S12. Upon completion of the current training step, obtain the latest parameter matrix of the DyKAN layer and calculate the current observation based on the dynamic tanh function. ;

[0049] S13. If Then update And add the new network parameter matrix to the upper limit set. ,like Then update And add the new network parameter matrix to the lower bound set. ;in, Represents absolute value;

[0050] S14. If ,and Then the upper limit set and lower bound set Merge into the second low-rank matrix to generate a new second low-rank matrix, and update ;

[0051] S15. Update the merge parameter step threshold for ;

[0052] S16. Update current training steps Then repeat steps S12 to S15 to complete the dynamic expansion of the second low-rank matrix.

[0053] According to one aspect of the present invention, the state information includes: robot body perception information, target object information, and environmental information;

[0054] The first and second actions include: the robot's end effector at... The three accelerations in the axial direction and the opening and closing state of the end effector fixture.

[0055] According to one aspect of the invention, during the training phase, environmental information is added to the state information. Gaussian noise;

[0056] During the training phase, the upper-level planning unit and the lower-level skill unit share the same reward function, which is a weighted average of extrinsic and intrinsic rewards, and is expressed as follows:

[0057] ;

[0058] in, Represents the reward function, This represents an external reward, calculated using a custom reward function. It represents intrinsic reward and is generated by a generative adversarial imitation learning mechanism.

[0059] According to one aspect of the invention, the upper-level planning unit is constructed based on an MLP network.

[0060] According to one aspect of the present invention, this scheme employs a dual-chain design to decompose long-term task chains, thereby decomposing complex tasks into two intersecting sub-chains. This effectively reduces optimization conflicts between sub-tasks, alleviates the computational load on upper-level planning units, and significantly improves the performance of the scheme. Furthermore, the dual-chain decomposition design ensures clear separation of the state distributions of different sub-tasks within the hierarchical reinforcement learning model, greatly reducing the requirement for manually defining sub-task boundaries with high precision.

[0061] According to one aspect of the present invention, the skill model of this scheme is constructed based on DyKAN. DyKAN is applied to the Actor and Critic networks in reinforcement learning, enabling the skill model of this scheme to adaptively adjust the grid and basis functions while retaining prior knowledge, thus effectively achieving optimal learning of robot operation skills.

[0062] According to one aspect of the present invention, this approach, based on the optimization of the DyKAN layer, further introduces a dynamic Tanh function (DyT) and a low-rank decomposition method, effectively ensuring the stability of model updates and low computational cost during training.

[0063] According to one aspect of the present invention, the Dynamic Kolmogorov-Arnold Network (DyKAN) can maintain the grid range and data distribution by updating the number of learnable basis function parameters without destroying the learned basis function parameters, thus effectively ensuring training stability under the exploratory paradigm of reinforcement learning.

[0064] According to one aspect of the present invention, the low-rank decomposition method effectively alleviates the problem of a surge in the number of learnable basis function parameters caused by mesh expansion, while achieving low-cost network parameter expansion. Furthermore, during both the training and inference phases, the complete parameter matrix of the learnable basis function parameters is calculated and reconstructed before subsequent computational processes are executed. Therefore, the low-rank decomposition of the parameter matrix of the learnable basis function parameters does not affect the local influence mechanism of the Dynamic Kolmogorov-Arnold Network (DyKAN), effectively ensuring the performance of this approach. Attached Figure Description

[0065] Figure 1 is a flowchart illustrating the steps of the long-term autonomous operation method for robots based on hierarchical reinforcement learning according to the present invention.

[0066] Figure 2 is a flowchart of the long-term autonomous operation method for robots based on hierarchical reinforcement learning according to the present invention;

[0067] Figure 3 is a structural diagram of the hierarchical reinforcement learning model of the present invention. Detailed Implementation

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0069] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0070] Referring to Figures 1, 2, and 3, according to one embodiment of the present invention, a long-term autonomous operation method for robots based on hierarchical reinforcement learning includes the following steps:

[0071] S1. Training Phase:

[0072] Construct a hierarchical reinforcement learning model, which includes: task chain decomposition unit, upper-level planning unit and lower-level skill unit;

[0073] The task chain decomposition unit acquires the robot's state information and outputs a long-term task chain, and decomposes the long-term task chain into parallel and intersecting even-order task chains and odd-order task chains.

[0074] The lower-level skill units include: a first skill model and a second skill model, and the first skill model and the second skill model respectively adopt dynamic Kolmogorov-Arnold networks;

[0075] The first skill model is trained using reinforcement learning based on an even-ordered task chain to output the first action, while the second skill model is trained using reinforcement learning based on an odd-ordered task chain to output the second action. In the reinforcement learning training of the skill models, a dynamic Tanh function and a low-rank decomposition method are used to assist in the dynamic updating of the skill models.

[0076] The upper-level planning unit performs reinforcement learning training based on global state information to generate action weights that weight the first action and the second action.

[0077] S2. Execution Phase:

[0078] Based on the robot's current state information, a long-term task chain is obtained and decomposed into even-ordered task chains and odd-ordered task chains, and the first action and the second action are generated respectively.

[0079] Generate action weights that are a weighted combination of the first and second actions;

[0080] The robot's actions are generated by weighting and combining the first and second actions based on their weights.

[0081] Referring to Figures 2 and 3, according to one embodiment of the present invention, in a long-term robotic operation task, the robot needs to execute a continuous sequence of actions. These actions can be abstracted into different operational skills (such as pushing, pulling, or grasping). Therefore, they can be derived from action sequences. Abstract a long-term task chain corresponding to the definition of the operation behavior. Therefore, based on this long-term task chain It can be decomposed into two parallel and intersecting coupled task chains, one of which is an even-order task chain. One is an odd-order quest chain. .

[0082] Referring to Figures 2 and 3, according to one embodiment of the present invention, in step S1, both the first skill model and the second skill model are constructed using a Dynamic Kolmogorov-Arnold Network (DyKAN). In this embodiment, for the skill model (i.e., the first skill model or the second skill model), a subtask is assumed during its reinforcement learning training process. , , The state distributions are respectively represented as , , Then the intersection of these state distributions is defined. as follows:

[0083] ;

[0084] ;

[0085] in, Represents state distribution and state distribution The intersection between them Represents state distribution and state distribution The intersection between them.

[0086] In this embodiment, the first skill model and the second skill model are trained using a reinforcement learning method based on policy gradients. Therefore, for any state... The robot in the subtask Actions performed in In subtasks Actions performed in The corresponding advantage function is defined as follows:

[0087] ;

[0088] ;

[0089] in, Representing state and actions The advantage function, Representing state and actions The advantage function, Representing state The state value function under [condition].

[0090] Therefore, when the first skill model and the second skill model are updated based on the skill model loss function, for any state... The skill model loss function can then be defined as:

[0091] ;

[0092] in, This represents the loss function of the skill model. Subtasks The loss function corresponding to the state, Subtasks The loss function corresponding to the state, This indicates the state of the subtask, i.e., the state information. Representing state and actions The advantage function, Representing state and actions The advantage function, Indicates in subtask The actions performed in the process, Indicates in subtask The actions performed in the process, This represents the mapping relationship from the advantage function to the loss function. Representing state The state value function under the given conditions This represents the value function of a state-action pair.

[0093] As shown in Figure 3, according to one embodiment of the present invention, the dynamic Kolmogorov-Arnold network used in the first skill model and the second skill model is consistent. The dynamic Kolmogorov-Arnold network is obtained by introducing a dynamic grid update mechanism based on the Kolmogorov-Arnold network. In this embodiment, under the action of the dynamic grid update mechanism, the dynamic Kolmogorov-Arnold network maintains the grid accuracy unchanged when updating the grid, and dynamically adjusts the grid size based on changes in the grid range, and adaptively adjusts the number of learnable basis function parameters in the B-spline activation function according to the changed grid size. Thus, the dynamic grid update mechanism enables dynamic updating of the number of learnable basis functions, allowing the dynamic Kolmogorov-Arnold network of this scheme to maintain a match between the grid range and the data distribution by updating the number of learnable basis function parameters without destroying the already learned basis function parameters, effectively ensuring training stability under the exploratory paradigm of reinforcement learning. This achieves the effect of dynamically expanding network parameters based on data distribution, enabling stable and efficient mastery of multiple operational skills without task ID embedding.

[0094] As shown in Figure 3, according to one embodiment of the present invention, the dynamic Kolmogorov-Arnold network includes:

[0095] The input layer is used to receive the state information corresponding to the subtasks;

[0096] At least one DyKAN layer, each DyKAN layer containing multiple B-spline activation functions to map the input to the output;

[0097] The output layer is used to generate actions corresponding to the subtasks;

[0098] The grid dynamic update mechanism module is used to dynamically expand the grid of each DyKAN layer to update the number of learnable basis function parameters in the B-spline activation function;

[0099] In this embodiment, the grid dynamic update mechanism module dynamically monitors the distribution range of the input data of each DyKAN layer in the dynamic Kolmogorov-Arnold network after each update, compares the distribution range with the grid range of the main grid, determines the newly added forward grid points and newly added backward grid points in the main grid, adds the newly added forward learnable basis function parameters corresponding to the newly added forward grid points to the preset forward parameter set, adds the newly added backward learnable basis function parameters corresponding to the newly added backward grid points to the preset backward parameter set, and periodically merges the main learnable basis function parameters corresponding to the main grid, the newly added forward learnable basis function parameters, and the backward learnable basis function parameters to complete the adaptive update of the grid size and the number of learnable basis function parameters in the B-spline activation function.

[0100] In this embodiment, the forward learnable basis function parameters stored in the forward parameter set are the parameters added due to the expansion of the upper bound of the input data distribution; the backward learnable basis function parameters stored in the backward parameter set are the parameters added due to the expansion of the lower bound of the input data distribution.

[0101] In this embodiment, the mesh parameters of the main mesh are initialized as follows:

[0102] ;

[0103] in, Indicates the grid range. Indicates the lower limit of the grid range. Indicates the upper limit of the grid range. Indicates the grid size. Let represent the order of the B-spline activation function. Then, the number of initial learnable basis function parameters in the B-spline activation function is equal to the grid size and order. The sum of and All are constants. Therefore, the learnable basis function parameters are denoted as... Each parameter corresponds to a basis function. Then the B-spline activation function can be expressed as:

[0104] ;

[0105] in, This represents the B-spline activation function.

[0106] Furthermore, to enable dynamic expansion of the shared mesh, the mesh dynamic update mechanism module further designed a forward parameter set (which can be represented as...). ) and the backward parameter set (which can be represented as The two sets of parameters (forward and backward learnable basis function parameters) are used to store the newly added learnable basis function parameters resulting from the expansion of the upper and lower bounds of the grid range. Therefore, after each network update, the skill model adjusts the grid range based on the upper and lower bounds of the current DyKAN layer's input data distribution: if the upper bound of the data distribution range is greater than the upper bound of the grid range, the upper bound of the grid range is updated, and the newly added learnable basis function parameters are added to the forward parameter set; similarly, if the lower bound of the grid range is updated, the corresponding newly added learnable basis function parameters are incorporated into the backward parameter set. During the training and inference phases, the skill model concatenates and merges the forward parameter set, the main grid, and the backward parameter set to calculate the activation function. To simplify parameter management, parameters from the forward and backward parameter sets can be periodically merged into the main grid, thereby achieving a controllable and orderly growth in the size of the learnable basis function parameters. Specifically, when the distribution range of the input data shrinks, the skill model does not synchronously shrink the grid range or the number of learnable basis function parameters, effectively suppressing training instability caused by frequent increases and decreases in network parameters (i.e., learnable basis function parameters).

[0107] As shown in Figure 3, according to one embodiment of the present invention, the first skill model and the second skill model dynamically update the auxiliary skill model during reinforcement learning training through a constructed DyKAN layer update module. The DyKAN layer update module is a residual block that adopts the residual structure of the Kolmogorov-Arnold network (KAN) and is constructed jointly with the DyKAN layer by introducing a fully connected layer. It should be noted that, to specifically evaluate the learning ability of the basis function parameters, the parameters of the introduced fully connected layer are not updated during training.

[0108] In this embodiment, in the reinforcement learning training of the skill model, the step of using a low-rank decomposition method to assist in the dynamic updating of the skill model includes unifying the meshes of all B-spline activation functions within the same DyKAN layer into a shared mesh with the same size and range based on the low-rank decomposition method.

[0109] Organize all learnable basis function parameters of the DyKAN layer into a dimension of The parameter matrix; in this embodiment, it is assumed that the input dimension of a certain DyKAN layer is... Output dimension is Then the DyKAN layer contains a total of Each DyKAN layer has a B-spline activation function. By unifying the mesh of all B-spline activation functions within the same DyKAN layer to a single shared mesh, each DyKAN layer only needs to maintain one shared mesh, thus expanding the learnable basis function parameters at a lower cost. Therefore, all learnable basis function parameters of this DyKAN layer will be initialized to a dimension of... The parameter matrix, where, This indicates the dimension of learnable basis functions in a single B-spline activation function. This is the number of learnable basis functions.

[0110] The parameter matrix is ​​decomposed into the product of a first low-rank matrix and a second low-rank matrix, where the first low-rank matrix has dimensions of... The second low-rank matrix has the following dimensions. ,in, This indicates a low-rank dimension.

[0111] When the number of learnable basis functions in the B-spline activation function of the same DyKAN layer increases, only the second low-rank matrix is ​​expanded and updated, while the first low-rank matrix remains unchanged. This enables dynamic updating of the shared grid of the same DyKAN layer, which is used to match the update process of the dynamic Kolmogorov-Arnold network.

[0112] Through the above settings, the low-rank decomposition method effectively alleviates the problem of a surge in the number of learnable basis function parameters caused by grid expansion during the dynamic Kolmogorov-Arnold network update process, while achieving low-cost network parameter expansion. Furthermore, during both training and inference phases, the complete parameter matrix of the learnable basis function parameters is calculated and reconstructed before subsequent computations are executed. Therefore, the low-rank decomposition of the parameter matrix of the learnable basis function parameters does not affect the local influence mechanism of the DyKAN layer, effectively ensuring the performance of this scheme.

[0113] As shown in Figure 3, according to one embodiment of the present invention, the step of using a dynamic tanh function to assist in the dynamic updating of the skill model during reinforcement learning training of the skill model includes:

[0114] A dynamic tanh function layer is added before the DyKAN layer. The dynamic tanh function used includes learnable scaling and translation parameters. In this embodiment, the dynamic tanh function in the dynamic tanh function layer is represented as follows:

[0115] ;

[0116] in, This represents the dynamic tanh function. This indicates the output scaling parameter. This indicates the input scaling parameter. Indicates input data, This indicates the output translation parameters.

[0117] The constraint range of the DyKAN layer input data distribution is dynamically adjusted by scaling and translation parameters to stabilize the training process and maintain the scalability of the second low-rank matrix.

[0118] An iteration parameter is set, and the second low-rank matrix is ​​dynamically expanded based on the iteration parameter and the dynamic tanh function to match the update process of the dynamic Kolmogorov-Arnold network. In this embodiment, the iteration parameter includes: the current training step number. Merging parameter step threshold Step increment Upper limit of shared grid range within the same DyKAN layer Lower bound of shared mesh range within the same DyKAN layer Among them, the upper limit of the shared grid range is within the same DyKAN layer. The upper limit of the grid range in the initialization settings of the main grid's grid parameters. For consistency, the lower bound of the shared mesh range is maintained within the same DyKAN layer. The lower limit of the grid range in the initialization settings of the main grid's grid parameters. To ensure consistency; therefore, the update process includes the following steps:

[0119] S11. Construct the upper bound set for expanding the second low-rank matrix. and lower bound set ;

[0120] S12. Upon completion of the current training step, obtain the latest parameter matrix of the DyKAN layer and calculate the current observation based on the dynamic tanh function. ;

[0121] S13. If Then update The new network parameter matrix (i.e., the newly added learnable basis function parameters resulting from the mesh change) is added to the upper bound set. ,like Then update And add the new network parameter matrix to the lower bound set. ;in, Represents absolute value;

[0122] S14. If ,and Then the upper limit set and lower bound set Merge into the second low-rank matrix to generate a new second low-rank matrix, and update ;

[0123] S15. Update the merge parameter step threshold for ;

[0124] S16. Update current training steps Then repeat steps S12 to S15 to complete the dynamic expansion of the second low-rank matrix.

[0125] According to one embodiment of the present invention, the state information includes: robot body perception information, target object information, and environmental information; in this embodiment, the robot body perception information includes the robot's end effector speed and gripper state; the target object information covers the polar coordinates of all target objects (such as drawers, shelves, etc.) relative to the end effector; the environmental information includes information that is difficult to quantify by relative position, such as the pressing state of buttons, etc.

[0126] In this embodiment, during the training phase, environmental information is added to the state information. Gaussian noise.

[0127] According to one embodiment of the present invention, the first action and the second action include: the robot's end effector in The three accelerations along the axial direction and the opening and closing states of the end effector gripper can thus be expressed as: ,in, These represent the end effector at... Three accelerations in the axial direction, Indicates the open / closed state.

[0128] According to one embodiment of the present invention, during the training phase, the upper-level planning unit and the lower-level skill unit share the same reward function, and the reward function is composed of a weighted average of extrinsic and intrinsic rewards, and is expressed as follows:

[0129] ;

[0130] in, Represents the reward function, This represents an external reward, calculated using a custom reward function. This represents the intrinsic reward, which is generated by the Generative Adversarial Imitation Learning (GAIL) mechanism.

[0131] According to one embodiment of the present invention, the upper-layer planning unit is constructed based on an MLP network.

[0132] Referring to Figures 2 and 3, according to one embodiment of the present invention, in step S2, the step of generating the action weights for the weighted combination of the first action and the second action, the action weights output by the upper-level planning unit are: ,in, This indicates the action weight of the first action. This indicates the action weight of the second action.

[0133] Referring to Figures 2 and 3, in one embodiment of the present invention, in the step of generating the robot's execution action by weighting and combining the first action and the second action based on action weights, the Softmax function is used to weight the two actions. and Normalization is performed to obtain the corresponding weight coefficients. and The robot's actions are thus ultimately represented as ,in, Indicates the first action. Indicates the second action.

[0134] The above description is merely an example of a specific solution of the present invention. For any devices and structures not described in detail herein, it should be understood that they are implemented using common devices and methods already available in the art.

[0135] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A long-term autonomous operation method for robots based on hierarchical reinforcement learning, characterized in that, Includes the following steps: S1. Training Phase: Construct a hierarchical reinforcement learning model, which includes: a task chain decomposition unit, an upper-level planning unit, and a lower-level skill unit. The task chain decomposition unit acquires the robot's state information and outputs a long-term task chain, decomposing the long-term task chain into parallel and interleaved even-order and odd-order task chains. The lower-level skill unit includes: a first skill model and a second skill model, which respectively employ a dynamic Kolmogorov-Arnold network. The first skill model is trained using reinforcement learning based on even-order task chains to output the first action, and the second skill model is trained using reinforcement learning based on odd-order task chains to output the second action. In the reinforcement learning training of the first and second skill models, a dynamic tanh function and a low-rank decomposition method are used to assist the first skill model. The second skill model is dynamically updated; the upper-level planning unit performs reinforcement learning training based on the robot's state information to generate action weights that weight the first action and the second action; S2. Execution phase: Based on the robot's current state information, a long-term task chain is obtained and decomposed into even-order task chains and odd-order task chains, and the first action and the second action are generated respectively; action weights that weight the first action and the second action are generated; the first action and the second action are weighted and combined based on the action weights to generate the robot's execution action; In step S1, the first skill model and the second skill model are trained based on the policy gradient reinforcement learning method, and the first skill model and the second skill model are updated based on the skill model loss function, wherein the skill model loss function is expressed as: in, This represents the loss function of the skill model. Subtasks The loss function corresponding to the state, Subtasks The loss function corresponding to the state, This indicates the state of the subtask, i.e., the state information. Representing state and actions The advantage function, Representing state and actions The advantage function, Indicates in subtask The actions performed in the process, Indicates in subtask The actions performed in the process, This represents the mapping relationship from the advantage function to the loss function. Representing state The state value function under the given conditions This represents the value function of a state-action pair.

2. The long-term autonomous operation method for robots based on hierarchical reinforcement learning according to claim 1, characterized in that, The dynamic Kolmogorov-Arnold network used in the first skill model and the second skill model is consistent. The dynamic Kolmogorov-Arnold network is obtained by introducing a dynamic grid update mechanism based on the Kolmogorov-Arnold network. Under the action of the dynamic grid update mechanism, the dynamic Kolmogorov-Arnold network maintains the grid accuracy unchanged when updating the grid, and dynamically adjusts the grid size based on the change of the grid range and adaptively adjusts the number of learnable basis function parameters in the B-spline activation function according to the changed grid size.

3. The long-term autonomous operation method for robots based on hierarchical reinforcement learning according to claim 2, characterized in that, The dynamic Kolmogorov-Arnold network includes: an input layer for receiving state information corresponding to subtasks; at least one DyKAN layer, each DyKAN layer containing multiple B-spline activation functions for mapping inputs to outputs; an output layer for generating actions corresponding to subtasks; and a grid dynamic update mechanism module for dynamically expanding the grid of each DyKAN layer to update the number of learnable basis function parameters in the B-spline activation functions; the grid dynamic update mechanism module dynamically monitors the number of inputs to each DyKAN layer of the dynamic Kolmogorov-Arnold network after each update. The distribution range of the data is determined and compared with the grid range of the main grid to identify the newly added forward and backward grid points of the main grid. The newly added forward learnable basis function parameters corresponding to the newly added forward grid points are added to the preset forward parameter set, and the newly added backward learnable basis function parameters corresponding to the newly added backward grid points are added to the preset backward parameter set. In addition, the main learnable basis function parameters corresponding to the main grid, the newly added forward learnable basis function parameters, and the backward learnable basis function parameters are periodically merged to complete the adaptive update of the grid size and the number of learnable basis function parameters in the B-spline activation function.

4. The long-term autonomous robot operation method based on hierarchical reinforcement learning according to claim 3, characterized in that, In the reinforcement learning training of the first and second skill models, the step of using a low-rank decomposition method to assist in the dynamic updating of the first and second skill models involves unifying the meshes of all B-spline activation functions within the same DyKAN layer into a shared mesh with the same size and range based on the low-rank decomposition method. This includes: organizing all learnable basis function parameters of the DyKAN layer into a single mesh with the same dimension. The parameter matrix; where, This represents the input dimension of the DyKAN layer. This indicates the output dimension of the DyKAN layer. This indicates the dimension of learnable basis functions in a single B-spline activation function. This represents the number of learnable basis functions; the parameter matrix is ​​decomposed into the product of a first low-rank matrix and a second low-rank matrix, where the dimension of the first low-rank matrix is... The dimension of the second low-rank matrix is ;in, The value represents the low-rank dimension. When the number of learnable basis functions in the B-spline activation function of the same DyKAN layer increases, only the parameters of the second low-rank matrix are expanded and updated, while the first low-rank matrix remains unchanged. This enables dynamic updating of the shared grid of the same DyKAN layer, which is used to match the update process of the dynamic Kolmogorov-Arnold network.

5. The long-term autonomous robot operation method based on hierarchical reinforcement learning according to claim 4, characterized in that, In the reinforcement learning training of the first and second skill models, the step of using a dynamic tanh function to assist in the dynamic updating of the first and second skill models includes: adding a dynamic tanh function layer before the DyKAN layer, wherein the dynamic tanh function includes learnable scaling and translation parameters; dynamically adjusting the constraint range of the input data distribution of the DyKAN layer through the scaling and translation parameters to stabilize the training process and maintain the scalability of the second low-rank matrix; setting iteration parameters and dynamically expanding the second low-rank matrix based on the iteration parameters and the dynamic tanh function to match the update process of the dynamic Kolmogorov-Arnold network; the dynamic tanh function in the dynamic tanh function layer is expressed as: in, This represents the dynamic tanh function. This indicates the output scaling parameter. This indicates the input scaling parameter. Indicates input data, This indicates the output translation parameters.

6. The long-term autonomous operation method for robots based on hierarchical reinforcement learning according to claim 5, characterized in that, In the step of setting iteration parameters and dynamically updating the first and second skill models based on the iteration parameters and the dynamic tanh function, the iteration parameters include: the current training step number. Merging parameter step threshold Step increment Upper limit of shared grid range within the same DyKAN layer Lower bound of shared mesh range within the same DyKAN layer This includes: S11. Constructing the upper bound set for expanding the second low-rank matrix. and lower bound set S12. Upon completion of the current training step, obtain the latest parameter matrix of the DyKAN layer and calculate the current observation based on the dynamic tanh function. S13. If Then update And add the new network parameter matrix to the upper limit set. ,like Then update And add the new network parameter matrix to the lower bound set. ;in, Indicates absolute value; S14. If ,and Then the upper limit set and lower bound set Merge into the second low-rank matrix to generate a new second low-rank matrix, and update S15. Update the merging parameter step threshold. for S16. Update the current training steps. Then repeat steps S12 to S15 to complete the dynamic expansion of the second low-rank matrix.

7. The method for autonomous operation of a robot in the long time domain based on hierarchical reinforcement learning according to any one of claims 2 to 6, characterized in that, The state information includes: robot body perception information, target object information, and environmental information; the first action and the second action include: the robot's end effector in... The three accelerations in the axial direction and the opening and closing state of the end effector fixture.

8. The long-term autonomous operation method for robots based on hierarchical reinforcement learning according to claim 7, characterized in that, During the training phase, environmental information is added to the state information. Gaussian noise; during the training phase, the upper-level planning unit and the lower-level skill unit share the same reward function, which is composed of a weighted average of extrinsic and intrinsic rewards, and is expressed as: in, Represents the reward function, This represents an external reward, calculated using a custom reward function. It represents intrinsic reward and is generated by a generative adversarial imitation learning mechanism.

9. The long-term autonomous robot operation method based on hierarchical reinforcement learning according to claim 1, characterized in that, The upper-level planning unit is constructed based on an MLP network.

Images