Quadruped robot form optimization method based on Monte Carlo tree search space migration

Abstract

The invention discloses a quadruped robot form optimization method based on Monte Carlo tree search space migration. The quadruped robot form optimization method comprises a pre-learning stage and an optimization stage. In the pre-learning stage, the whole space is divided by directly using source task data; in the optimization stage, a subspace is iteratively selected for searching, a target task is sampled and evaluated in the space, and space division is adjusted according to observation values. According to the method, an effective solution is provided for efficient form optimization of the quadruped robot by adopting the method for carrying out search space migration based on the Monte Carlo tree. On one hand, hot start is provided for the morphological parameter optimization task of the quadruped robot, so that the optimization efficiency is improved; and on the other hand, the task weight can be dynamically adjusted according to the similarity between the tasks, and the division result of the search space is automatically adjusted, so that a plurality of potential compact subspaces are provided for the morphological parameter optimization of the quadruped robot, and the optimization speed and accuracy can be improved.

Description

Technical Field

[0001] The invention relates to a quadruped robot morphology optimization method, in particular to a quadruped robot morphology optimization method based on Monte Carlo tree search space migration, and relates to the technical field of artificial intelligence. Background Art

[0002] The quadruped robot morphology optimization task uses the parametric morphology of the MuJoCo Ant robot. This is an advanced 3D model consisting of a central torso and four legs, each of which is made up of two links. The core goal of this task is to achieve efficient crawling in a specific direction by adjusting the robot's morphological parameters. Quadruped robot morphology optimization technology has broad application prospects in the real world, especially in high-risk tasks such as search and rescue and terrain exploration. These optimized robots can show higher efficiency and better safety.

[0003] The process of quadruped robot morphology optimization usually involves complex system behaviors, and the effect of adjusting morphology parameters on the robot's walking performance is often not easy to predict through intuitive understanding or simple calculation. Therefore, the quadruped robot morphology optimization problem can be regarded as a black-box optimization problem (BBO), that is, the objective function has no analytical form and can only be evaluated through different inputs. Bayesian optimization (BO) is a widely used sample-efficient method to solve such problems. In each iteration, BO fits a surrogate model, usually a Gaussian process (GP), to approximate the objective function, and then maximizes the acquisition function to determine the next query point. However, in order to obtain an accurate evaluation of the morphology parameters, the quadruped robot morphology optimization task usually requires running hundreds of time steps in the simulator and performing multiple repeated experiments, which is very expensive to evaluate. Under the limited evaluation budget, the traditional BO method has only a small number of observations, which is not enough to build an accurate surrogate model and converges slowly. Therefore, the traditional BO method is difficult to effectively solve the expensive quadruped robot morphology optimization problem.

[0004] In order to solve the expensive quadruped robot morphology optimization problem, some studies have tried to combine transfer learning with BO. The BO method of transfer learning assumes that similar tasks may have similar characteristics, so the knowledge acquired from similar source tasks will help accelerate the optimization of the current target task. Current methods usually collect offline datasets from source tasks and use them to speed up the optimization of the target task. These methods can be divided into migration algorithms for acquisition functions, migration algorithms for initialization points, or migration algorithms for search spaces. Among them, the migration of search space has become a research hotspot due to its effectiveness and the advantage of being combined with other optimization methods. By learning and dividing the search space, this type of method can more effectively explore potential subspaces and significantly accelerate the optimization speed. Although the current space migration algorithms have shown great potential, there are still some limitations:

[0005] 1. When the learned search space is not suitable for the target task, it is difficult to flexibly adjust the search space.

[0006] 2. When the source task is significantly different from the target task, they tend to turn to full-space search.

[0007] 3. Simplistically evaluating the search space as “good” or “bad” without a more accurate numerical assessment.

[0008] In quadruped robot optimization tasks, it is sometimes difficult to obtain sufficient similar offline data sets for a specific robot or a specific optimization environment, and it is necessary to use offline optimization data of other robots in other environments. This will lead to low reliability and complexity of the source task data. However, the current spatial migration method is difficult to effectively identify and process similar data in the source task and effectively adjust the sub-search space, and the optimization efficiency is low in scenarios with complex source tasks. Summary of the invention

[0009] Purpose of the invention: In order to solve the problem of low efficiency of the current spatial transfer learning method in quadruped robot morphology optimization, the present invention provides a quadruped robot morphology optimization method based on Monte Carlo tree search spatial transfer.

[0010] Technical solution: A quadruped robot morphology optimization method based on Monte Carlo tree search space migration. First, considering the need to learn the relevant knowledge of the search space from the source task in advance, the method is divided into two main stages, the pre-learning stage and the optimization stage. In the pre-learning stage, the source task data is directly used to divide the entire space; in the optimization stage, a subspace is iteratively selected for search, and the target task is sampled and evaluated in the space, and the space division is adjusted according to the observed value.

[0011] The detailed algorithm flow of the pre-learning stage is as follows:

[0012] (101) Initialize an MCTS model T. The model contains only one root node, representing the entire search space Ω, and all data in the source task dataset fall into this node.

[0013] (102) When there is a leaf node that can be further divided, that is, the number of samples contained in the node exceeds θ, and these samples can be clustered into two clusters, the leaf node is expanded. For a group of source task samples in the leaf node, kMeans clustering is used to divide the samples into two groups, where the cluster with a higher sample average evaluation value is regarded as a "good" cluster, and the other cluster is regarded as a "bad" cluster. Finally, a binary classifier is used to divide the space where the two clusters are located. The space where the good cluster is located is the left child node of the node, and the space where the bad cluster is located is the right child node of the node.

[0014] (103) Repeat step (102) until all leaf nodes cannot be further divided.

[0015] (104) Returns the pre-trained MCTS model T.

[0016] The specific steps of the optimization phase are as follows.

[0017] (201) Collect source task datasets with the same dimension as the target quadruped robot morphology optimization task And pre-train an MCTS model.

[0018] (202) Initialize an empty set D T To store samples of the target task; initialize best_x = None, best_y = -inf to store the optimal morphological parameters and their evaluation values ​​during the optimization process.

[0019] (203) In each iteration, the UCB value of each node j is calculated to measure the degree of exploration and utilization of the node. Among them, n p is the sum of the number of source and target task samples in the parent node of node j, n j is the sum of the number of source and target task samples in node j, p j Ω j The potential value, C p It is a parameter to control the degree of utilization and exploration. Starting from the root node, the node with the largest UCB value is continuously selected from all the child nodes of the current node as the next exploration node until a leaf node m is found.

[0020] (204) If the target task dataset D T In the space Ω represented by node m m There are samples in Ω mOr samples in Ω to build a proxy model and optimize it with BO optimizer, from the sub-search space Ω m Get the new candidate morphological parameters x T,t Otherwise, in Ω m Randomly select candidate morphological parameters x from T,t .

[0021] (205) For the candidate morphological parameters x T,t Evaluate and get y T,t , and the sample (x T,t ,y T,t ) is added to the target task dataset D T When y T,t > best_y, the algorithm finds a new current optimal morphological parameter and updates best_x to x T,t , update best_y to y T,t Target task dataset D T Can be used to update and optimize models and task similarity.

[0022] (206) In order to make full use of the information of the source task, the similarity between the source task and the target task is considered, and a higher weight is assigned to the more similar source task. Here, the distance between tasks is used to represent the task similarity. The smaller the distance, the higher the similarity. As the target task dataset D T The addition of new samples in each source task dataset D i And the target task dataset D T The distance between them may change. According to the distance sorting, a linear weight allocation scheme is adopted to assign higher weights to source tasks with smaller distances to the target tasks.

[0023] (207) Based on the new weights, the potential value of each node in the MCTS model T is updated.

[0024] (208) If node m can be expanded, that is, contains more than θ target task samples and can be clustered into two clusters, then expand the node. For node expansion, only samples of the target task are considered.

[0025] (209) Perform a back-propagation step to update the information of each node on the path from the root node to the node m in the MCTS model T, including the number of visits and the samples included.

[0026] (210) Check whether the updated MCTS model T meets the requirement that the left child node has a greater potential value than the right child node. Any subtree that violates this property will be rebuilt.

[0027] Repeat the process (203)-(210) until the algorithm stops. The algorithm will use the morphological parameters stored in best_x as the initial morphological parameters of the quadruped robot for the final application.

[0028] Step (210) is called treeization, which is used to standardize the structure of the MCTS model T and reconstruct subtrees that violate the property that the left child node has a greater potential value than the right child node. The specific steps are as follows:

[0029] (2101) Initialize two empty queues Q and N, which are used to store the nodes to be visited and the nodes that need to be rebuilt in the subtree respectively.

[0030] (2102) Place the root node of the MCTS model T at the end of the queue Q.

[0031] (2103) When the queue Q is not an empty set, repeat the following: take a node node from Q in the order of first-in first-out. When the node node is not a leaf node, if the potential value of the left child of the node is less than the potential value of the right child, delete the subtree of the node node and put the node node at the end of the queue N; otherwise, put the left child and right child of the node node at the end of the queue Q in sequence.

[0032] (2104) Traverse the nodes in the queue N. If a node can be expanded, that is, the samples contained in the node exceed θ and can be clustered into two clusters, expand the node.

[0033] Beneficial effects: The present invention provides an effective solution for the efficient morphological optimization of quadruped robots by adopting a method for searching space migration based on Monte Carlo trees. On the one hand, the present invention provides a hot start for the morphological parameter optimization task of quadruped robots through the MCTS model in the pre-learning stage, thereby improving the optimization efficiency; on the other hand, the method can dynamically adjust the task weights according to the similarity between tasks, and automatically adjust the division results of the search space, thereby providing multiple potential compact subspaces for the optimization of the morphological parameters of quadruped robots, which can improve the speed and accuracy of optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of a method according to an embodiment of the present invention.

[0035] Figure 2 This is a rendering of the embodiment of the present invention, that is, the average ranking and variance of each optimization algorithm in the Mujuco Ant environment. DETAILED DESCRIPTION

[0036] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0037] Monte Carlo tree search (MCTS) space migration method, referred to as MCTS-transfer. This method is a search space learning method that can iteratively divide, select and optimize. In this method, each node of the MCTS model T represents a subspace, and the potential value of the node is calculated by calculating the weighted sum of the source sample value and the target sample value. In order to better identify similar source tasks and better utilize the information of the source tasks, the present invention dynamically assigns different weights to different source tasks according to the similarity between the source tasks and the target tasks. Based on reasonable weight allocation, the present invention will readjust the structure of the MCTS model T, that is, reconstruct the division of the search space, and provide multiple potential compact subspaces for the optimization of the morphological parameters of the target quadruped robot. This method can make full use of the information of the source task for hot start, and use similar source tasks to accelerate reasonable space division, thereby improving the efficiency of the quadruped robot morphological parameter search.

[0038] The purpose of the present invention is to solve the problem of low efficiency of quadruped robot morphology optimization. In order to implement the quadruped robot morphology optimization method based on Monte Carlo tree search space migration, the test environment selected in this embodiment is MuJoCoAnt, wherein the morphological parameters of the quadruped robot Ant are 60 continuous parameters including the size, direction and position of the limbs. In many real situations, it is difficult to obtain enough similar offline data sets when optimizing a specific robot or in a specific optimization environment, and it is necessary to use the offline optimization data of other robots in other environments. This will result in low reliability of source task data and certain complexity, making the optimization problem become an expensive black box optimization problem with complex source task data. Traditional space migration algorithms are difficult to handle such complex source tasks, and it is difficult to accurately evaluate spatial potential and adjust sub-search spaces. The search space migration algorithm proposed in this embodiment is committed to solving this problem, further improving the accurate use of source task data when the source task is complex and multivariate, and reasonably evaluating and adjusting the subspace potential.

[0039] The MCTS-transfer method is mainly divided into two main stages: pre-learning stage and optimization stage. The main process is to collect a source task dataset with the same dimension as the target quadruped robot morphology optimization task in the pre-learning stage. Directly use the source task data to divide the entire space; in the optimization stage, iteratively select the subspace with high UCB value as the search space, sample and evaluate the morphological parameters of the quadruped robot morphological parameter optimization task in this space, and adjust the search space division result according to the morphological parameters and their evaluation results. Let represent the entire search space, x i,j is a solution in space, i.e., the morphological parameters of the quadruped robot, y i,j is the evaluation value of the morphological parameter; the K source tasks are represented as The offline dataset of the i-th source task is represented as The target quadruped robot morphology parameter evaluation function can be expressed as f T (x), which is used to measure the average return of the quadruped robot in a specific time step under the morphological parameter x, and can be used to evaluate the quality of the morphological parameter x; the quadruped robot morphological parameter optimization task can be expressed as max x∈Ω f T (x), that is, to find the morphological parameters that can make the quadruped robot get the highest average reward; the search space division process is represented by the MCTS model T, each space division corresponds to the expansion of a leaf node, and each node m corresponds to a sub-search space Ω m .

[0040] In the pre-learning phase, T initially has only one root node, which represents the entire search space and contains the data of all source tasks. The expansion of node m corresponds to Ω m The node expansion process is: for Ω m For a set of samples of source tasks in the training set, the present invention uses kMeans clustering to divide the samples into two groups, where the cluster with a higher average evaluation value of the samples is regarded as a "good" cluster, and the other cluster is regarded as a "bad" cluster. Finally, a binary classifier is used to divide the space where the two clusters are located. In this method, the space where the good cluster is located is always called the left node, and the space where the bad cluster is located is called the right node. The division is performed recursively within each node. The division process involves a threshold hyperparameter θ, which is used to control the depth of the MCTS model T: if the number of samples contained in the node exceeds θ and these samples can be clustered into two clusters, the node will continue to be expanded. The MCTS model T generated in the pre-learning stage can give a suitable space division in advance according to the source task data, and provide a hot start for the subsequent optimization process. In the initial stage, the calculation formula of the spatial potential value is:

[0041]

[0042] Among them, p m represents the potential value of node m, D i ∩Ω m Denotes the dataset D iContained in the sub-search space Ω m The sample set in , K is the number of source tasks.

[0043] During the optimization process, the algorithm follows the four key steps of MCTS: selection, expansion, simulation, and backpropagation. In order to record the morphological parameters with the best performance, this method uses best_x to store the optimal morphological parameters during the optimization process, best_y to store the evaluation value corresponding to the optimal morphological parameters, and sets best_y to infinitesimal at the beginning. In each iteration, the algorithm selects the target node m by tracking the UCB value of the node, that is, starting from the root node, continuously selecting the node with the largest UCB value from all the child nodes of the current node as the next exploration node until the leaf node m is found. The sub-search space Ω corresponding to the node m m is regarded as a potential sub-search space. The calculation method of UCB is:

[0044]

[0045] Among them, n p is the sum of the number of source and target task samples in the parent node of node m, n m is the sum of the number of source and target task samples in node m, p m Ω m The potential value, C p is a parameter that controls the degree of exploitation and exploration. Next, use Ω m Or samples in Ω to build a proxy model and optimize it with BO optimizer, from the sub-search space Ω m Get the new candidate morphological parameters x T,t . Evaluate the morphological parameters and obtain y T,t . The sample (x T,t ,y T,t ) is added to the target task dataset D T When the evaluation value of the candidate morphological parameter is higher than the current optimal evaluation value, that is, y T,t >best_y, the algorithm will update the current optimal shape parameters and update best_x to x T,t , update best_y to y T,t Finally, this method will perform back propagation of the MCTS model T, that is, update the number of visits and potential value of each node. The potential value is calculated as follows:

[0046]

[0047] in, is the dataset D of the i-th task i In the sub-search space Ω mThe mean of the y values ​​of the samples in the equation, γ is the attenuation coefficient, usually set to 0.99, and w i is the weight of task i. In order to calculate w i , use D i and D T The distance between the i-th source task and the target task is measured by taking the average value of the best N sampling points in i , D T ), the distance between tasks can reflect the similarity of tasks. The smaller the distance, the higher the similarity. Then, the source tasks are sorted according to the distance between the source task and the target task, and the weights are updated. i The calculation is as follows:

[0048]

[0049] In this formula, a is used to control the proportion of source tasks with greater influence, and N m is the number of tasks with data points in node m, r i is the ranking value of the task. The smaller the ranking value, the smaller the distance between tasks and the higher the task similarity. This weight allocation method can assign higher weights to tasks with higher similarity.

[0050] After updating the potential values ​​of all nodes, the algorithm will expand the expandable nodes and recheck the current MCTS model T for treeification. First, apply breadth-first search to traverse all tree nodes. If there is such a node, the potential value of its right child node exceeds the potential value of its left child node, then the subtree of the node violates the tree building rules. This situation means that the current space division result is inaccurate, so the tree structure will be rebuilt. Specifically, the subtree of the node needs to be deleted and the node is added to the set of nodes to be reconstructed. After all nodes are traversed, check For each node in , if the node is splittable, that is, the samples contained in the node exceed θ and can be clustered into two clusters, then the node will be re-expanded. In this way, the process of subtree reconstruction can fine-tune the tree structure and retain some historical information, and make the tree structure more in line with the target task.

[0051] After the algorithm reaches the stopping criteria after multiple rounds of iterations, the morphological parameters stored in best_x will be used as the initial morphological parameters of the quadruped robot for the final application.

[0052] This algorithm has several significant advantages:

[0053] It can perform hot start for the optimization of the target task, that is, provide a better initial search space so that the quadruped robot has better initial morphological parameters.

[0054] The observed values ​​of the task can be optimized according to the morphological parameters of the target quadruped robot, the source tasks similar to the target task can be automatically identified, and the search space can be reconstructed to make the current search space structure more consistent with the target task.

[0055] The MCTS structure can provide multiple potential compact subspaces for BO optimization, thereby improving the speed and accuracy of quadruped robot morphological parameter optimization.

[0056] The specific steps of the main process of the method are as follows.

[0057] (1) Collect source task datasets with the same dimension as the target quadruped robot morphology optimization task And pre-train an MCTS model.

[0058] (2) Initialize an empty set D T To store samples of the target task; initialize best_x = None, best_y = -inf to store the optimal morphological parameters and their evaluation values ​​during the optimization process.

[0059] (3) In each iteration, the UCB value of each node j is calculated to measure the degree of exploration and utilization of the node. Among them, n p is the sum of the number of source and target task samples in the parent node of node j, n j is the sum of the number of source and target task samples in node j, p j Ω j The potential value, C p It is a parameter to control the degree of utilization and exploration. Starting from the root node, the node with the largest UCB value is continuously selected from all the child nodes of the current node as the next exploration node until a leaf node m is found.

[0060] (4) If the target task dataset D T In the space Ω represented by node m m There are samples in Ω m Or samples in Ω to build a proxy model and optimize it with BO optimizer, from the sub-search space Ω m Get the new candidate morphological parameters x T,t Otherwise, in Ω m Randomly select candidate morphological parameters x from T,t .

[0061] (5) For the candidate morphological parameters x T,t Evaluate and get y T,t , and the sample (x T,t ,y T,t ) is added to the target task dataset DT When y T,t > best_y, the algorithm finds a new current optimal morphological parameter and updates best_x to x T,t , update best_y to y T,t Target task dataset D T Can be used to update and optimize models and task similarity.

[0062] (6) In order to make full use of the information of the source task, the similarity between the source task and the target task is considered, and a higher weight is assigned to the more similar source task. Here, the distance between tasks is used to represent the task similarity. The smaller the distance, the higher the similarity. As the target task dataset D T The addition of new samples in each source task dataset D i And the target task dataset D T The distance between them may change. According to the distance sorting, a linear weight allocation scheme is adopted to assign higher weights to source tasks with smaller distances to the target tasks.

[0063] (7) Based on the new weights, update the potential value of each node in the MCTS model T.

[0064] (8) If node m can be expanded, that is, it contains more than θ samples of the target task and can be clustered into two clusters, then expand the node. For node expansion, only samples of the target task are considered.

[0065] (9) Perform the back-propagation step to update the information of each node on the path from the root node to node m in the MCTS model T, including the number of visits and the samples included.

[0066] (10) Check whether the updated MCTS model T meets the requirement that the left child node has a greater potential value than the right child node. Any subtree that violates this property will be rebuilt.

[0067] Repeat process (3)-(10) until the algorithm stops. The algorithm will use the morphological parameters stored in best_x as the initial morphological parameters of the quadruped robot for the final application.

[0068] Step (1) in the algorithm flow is the pre-training process, and the specific steps are as follows:

[0069] (1) Initialize an MCTS model T. The model contains only one root node, which represents the entire search space Ω. All data in the source task dataset fall into this node.

[0070] (2) When there is a leaf node that can be further divided, that is, the number of samples contained in the node exceeds θ, and these samples can be clustered into two clusters, the leaf node is expanded. For a group of source task samples in the leaf node, kMeans clustering is used to divide the samples into two groups, where the cluster with a higher average sample evaluation value is regarded as a "good" cluster, and the other cluster is regarded as a "bad" cluster. Finally, a binary classifier is used to divide the space where the two clusters are located. The space where the good cluster is located is the left child node of the node, and the space where the bad cluster is located is the right child node of the node.

[0071] (3) Repeat step (2) until all leaf nodes cannot be further divided.

[0072] (4) Return the pre-trained MCTS model T.

[0073] Step (10) in the algorithm flow is called treeization, which is used to standardize the structure of the MCTS model T and reconstruct subtrees that violate the property that the left child node has a greater potential value than the right child node. The specific steps are as follows:

[0074] (1) Initialize two empty queues Q and N, which are used to store the nodes to be visited and the nodes that need to be rebuilt.

[0075] (2) Place the root node of the MCTS model T at the end of the queue Q.

[0076] (3) When the queue Q is not an empty set, repeat the following steps. Take a node node from Q in first-in, first-out order. When the node node is not a leaf node, if the potential value of the left child node is less than the potential value of the right child node, delete the subtree of the node node and put the node node at the end of the queue N; otherwise, put the left child node and the right child node of the node node at the end of the queue Q in sequence.

[0077] (4) Traverse the nodes in the queue N. If a node can be expanded, that is, the number of samples contained in the node exceeds θ and can be clustered into two clusters, expand the node.

[0078] 1. Verification

[0079] The present invention provides a quadruped robot morphology optimization method based on Monte Carlo tree search space migration. The quadruped robot morphology optimization task is a complex black box optimization task. In order to verify and demonstrate the ability of MCTS-transfer in processing complex multivariate tasks and accurately evaluating and adjusting subspaces, the present invention generates a set of complex and multivariate source task data sets, and verifies the real effect of MCTS-transfer on this data set. The source task data set includes training data on the Ant Morphology task in design-bench, and some synthetic data. The AntMorphology task introduces Gaussian noise with a standard deviation of 0.03 on the basic morphology of the Ant robot; then, the task is optimized by randomly selecting initial morphological points and applying the covariance matrix adaptive evolution strategy (CMA-ES); during the optimization process, a Gaussian random perturbation with a standard deviation of 0.075 is applied to the default morphology of Ant, and the standard deviation of CMA-ES is set to 0.02. During the optimization and iteration process, a total of 25,009 offline data were collected. In addition, the present invention also mixes in a batch of synthetic data. Let x* be the optimal point found by all algorithms, lb and ub be the lower and upper bounds of the search space, and set two Sphere functions, f 1 (x)=(xx*) 2 and f 2 (x) = (xx*-ub+lb) 2 ; Use 7 optimization functions in f 1 and f 2 The sampling is Random Search, Shuffled Grid Search, Hill Climbing, Regularized Evolution Eagle Strategy, GP-EI, and a total of 14 data sets are obtained, each containing 300 data. For a set of morphological parameters x, the evaluation method adopted by the present invention is to conduct 100 time steps in the Mujoco Ant simulator environment, repeat the experiment 16 times, and take the average return under the morphological parameters as the evaluation result.

[0080] The present invention compares the following methods:

[0081] MCTS-transfer-GP: This invention.

[0082] Non-metastatic BO

[0083] ◆GP: The most basic BO algorithm.

[0084] ◆LA-MCTS: The non-transfer learning version of this invention.

[0085] ●Search space transfer learning algorithm

[0086] ◆Box-GP: A search space migration algorithm that restricts the search space to a box-shaped region containing the optimal values ​​of all source tasks.

[0087] ◆Ellipsoid-GP: A search space migration algorithm that restricts the search space to an ellipsoidal region that contains the optimal values ​​of all source tasks.

[0088] ◆Supervised-GP: A state-of-the-art search space transfer learning algorithm. It uses task similarity and voting mechanisms to combine and generate algorithms that guide promising search spaces.

[0089] ●Proxy model transfer learning algorithm

[0090] ◆PFN: A state-of-the-art proxy model transfer learning algorithm. An algorithm that uses a neural network based on prior knowledge as a proxy model to transfer knowledge.

[0091] For fair comparison, all GP and MCTS related parameters in the algorithm are the same. The sampling method is to sample the entire space 10,000 times, retain the candidate points that meet the conditions, and repeat this process 3 times. The experimental curve is the mean and standard deviation of the ranking results of the algorithm on all test problems, such as Figure 2 shown.

[0092] By comparing LA-MCTS and MCTS-transfer, it can be found that the effect of the present invention can be significantly improved after the search space migration, which proves the effectiveness of MCTS-transfer. After 25 iterations, MCTS-transfer can surpass the existing search space migration algorithm, proving that the present invention has obvious advantages in dealing with complex and diverse source tasks, and can better adjust the search space structure and find a better subspace. Combining all the results, the method of the present invention can alleviate the problems that the existing search space migration algorithm is difficult to deal with complex source tasks and difficult to accurately evaluate the spatial potential and adjust the sub-search space. It can iteratively divide the subspace through MCTS, dynamically adjust the source task weights, and deconstruct and reconstruct the tree, thereby extracting a more potential search space.

[0093] The present invention provides a hot start for the quadruped robot morphology parameter optimization task through the MCTS model in the pre-learning stage, thereby improving the optimization efficiency; and can dynamically adjust the task weight according to the similarity between tasks, and automatically adjust the division result of the search space, thereby providing multiple potential compact subspaces for the quadruped robot morphology parameter optimization, improving the speed and accuracy of optimization. Compared with the existing search space migration method, the present invention can identify similar source tasks in complex migration problems, has fast convergence speed, good performance, and is easy to implement, and has great practical application potential.

Claims

1. A quadruped robot morphology optimization method based on Monte Carlo tree search space migration, characterized in that: First, considering the need to learn relevant knowledge of the search space from the source task in advance, this method is divided into two main stages: pre-learning stage and optimization stage; The detailed algorithm flow of the pre-learning stage is as follows: (101) Initialize an MCTS model T; the model contains only one root node, representing the entire search space Ω, and all data in the source task dataset fall into this node; (102) When there is a leaf node that can be further divided, expand the leaf node; for a group of source task samples in the leaf node, use kMeans clustering to divide the samples into two groups, where the cluster with a higher sample average evaluation value is regarded as a "good" cluster, and the other cluster is regarded as a "bad" cluster. Finally, use a binary classifier to separate the space where the two clusters are located; where the space where the good cluster is located is the left child node of the node, and the space where the bad cluster is located is the right child node of the node; (103) Repeat step (102) until all leaf nodes cannot be further divided; (104) Return the pre-trained MCTS model T; The specific steps of the optimization phase are as follows: (201) Collect source task datasets with the same dimension as the target quadruped robot morphology optimization task And pre-train an MCTS model; (202) Initialize an empty set D T To store samples of the target task; initialize best_x = None, best_y = -inf to store the optimal morphological parameters and their evaluation values ​​during the optimization process; (203) In each iteration, the UCB value of each node i is calculated to measure the degree of exploration and utilization of the node. Starting from the root node, the node with the largest UCB value is continuously selected from all child nodes of the current node as the next exploration node until the leaf node m is found; (204) If the target task dataset D T In the space Ω represented by node m m There are samples in Ω m Or samples in Ω to build a proxy model and optimize it with BO optimizer, from the sub-search space Ω m Get the new candidate morphological parameters x T，t Otherwise, in Ω m Randomly select candidate morphological parameters x from T，t ; (205) For the candidate morphological parameters x T，t Evaluate and get y T,t , and the sample (x T,t ,y T,t ) is added to the target task dataset D T In; when y T,t > best_y, the algorithm finds a new current optimal morphological parameter and updates best_x to x T，t , update best_y to y T,t ; (206) assign higher weights to more similar source tasks; (207) Based on the new weights, update the potential value of each node in the MCTS model T; (208) If the node m can be expanded, expand the node; (209) Perform a back-propagation step to update the information of each node on the path from the root node to the node m in the MCTS model T, including the number of visits and the samples included; (210) Check whether the updated MCTS model T meets the requirement that the left child node has a larger potential value than the right child node; any subtree that violates this property will be rebuilt. Repeat the process (203)-(210) until the algorithm stopping criterion is reached; the morphological parameters stored in best_x are used as the initial morphological parameters of the quadruped robot for final application.

2. The quadruped robot morphology optimization method based on Monte Carlo tree search space migration according to claim 1 is characterized in that: In the step (206), the distance between tasks is used to represent the task similarity; With the target task dataset D T The addition of new samples in each source task dataset D i And the target task dataset D T The distance between them may change. According to the distance sorting, a linear weight allocation scheme is adopted to assign higher weights to source tasks with smaller distances to the target tasks.

3. The quadruped robot morphology optimization method based on Monte Carlo tree search space migration according to claim 1 is characterized in that: The step (210) is called treeization, which is used to standardize the structure of the MCTS model T and reconstruct subtrees that violate the property that the left child node has a greater potential value than the right child node. The specific steps are as follows: (2101) Initialize two empty queues Q and N, which are used to store nodes to be visited and nodes that need to be rebuilt; (2102) placing the root node of the MCTS model T at the end of the queue Q; (2103) When the queue Q is not an empty set, repeat the following: take a node node from Q in the order of first-in, first-out; when the node node is not a leaf node, if the potential value of the left child node of the node is less than the potential value of the right child node, delete the subtree of the node node and put the node node at the end of the queue N; otherwise, put the left child node and the right child node of the node node at the end of the queue Q in sequence; (2104) Traverse the nodes in the queue N. If a node can be expanded, that is, the samples contained in the node exceed θ and can be clustered into two clusters, expand the node.

Images