A robot morphological orientation evolution method based on environment perception and related device

Abstract

The application discloses a robot morphology directional evolution method based on environment perception and a related device, and relates to the technical field of robot morphology evolution. The method comprises the following steps: initializing a morphology gene vector and a control strategy of a robot; sampling and executing an action strategy according to the control strategy and environment perception information, and then calculating a system benefit value; judging whether the morphology gene vector of the robot is adapted to a current environment according to the environment perception information and the system benefit value; when the judgment result is yes, performing a mutation operation on the morphology gene vector of the robot based on a morphology mutation matrix; otherwise, jumping back to the step of sampling and executing the action strategy until a maximum iteration number is reached. The application quantifies the cumulative stimulation of environmental changes on the robot through an environment perception function to determine whether the robot needs to perform morphology evolution, and guides the direction of morphology evolution of the robot in different environments through the morphology mutation matrix, thereby accelerating the morphology evolution process of the robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot form evolution, and particularly relates to a robot form directional evolution method based on environment perception and a related device. BACKGROUND

[0002] Robots show significant advantages as substitutes for humans in dangerous environments. In the demand for diverse environments, robots of different forms have been developed to achieve greater adaptability. However, changing environmental conditions require the deployment of a large number of highly specialized robots, which must be redesigned and rebuilt each time. Creating robots with form intelligence to adapt to complex and changing environments is an effective solution to the above problems.

[0003] The problem of robot form evolution is considered crucial in the field of robotics, and the core goal is to find the optimal robot form through the evolution process to achieve simplified control in various complex tasks. However, form evolution faces two major challenges: 1) a large form search space, and 2) the computationally expensive training and testing of optimal control strategies required for each form evaluation.

[0004] Current robot form evolution methods mainly use evolutionary search algorithms. In each generation of evolution, robots with high fitness are selected and new forms are explored by introducing random mutations. In existing research, robot form generation involves evolution in a limited form search space or finding the optimal parameters for a given form. However, since each poorly performing robot form needs to be evaluated through lifelong learning, the large number of forms generated by random evolution results in a significant increase in computational time and cost. This challenging problem requires in-depth research to improve the efficiency and feasibility of robot form evolution. SUMMARY

[0005] The purpose of the present application is to provide a robot form directional evolution method based on environment perception and a related device, which can accelerate the robot form evolution process.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] In one aspect, the present application provides a robot form directional evolution method based on environment perception, comprising the following steps:

[0008] Initialize the form gene vector of the robot and the control strategy of the robot.

[0009] According to the control strategy of the robot and the environment perception information, sample and execute the action strategy.

[0010] According to the action strategy executed by the robot, calculate the system benefit value.

[0011] According to the environmental perception information and the system benefit value, it is judged whether the morphology gene vector of the robot is suitable for the current environment, and a first judgment result is obtained.

[0012] If the first judgment result is yes, a mutation operation is performed on the morphology gene vector of the robot based on a morphology mutation matrix, and the step of controlling the robot to sample and execute an action strategy based on a control strategy and environmental perception information is jumped to, until a maximum iteration number is reached.

[0013] If the first judgment result is no, the step of controlling the robot to sample and execute an action strategy based on a control strategy and environmental perception information is jumped to, until a maximum iteration number is reached.

[0014] Optionally, the method further comprises the following steps:

[0015] According to the plurality of evolution process sets of different robots and the optimal system benefit values corresponding to the robots, a contribution value of each mutation operation in the morphology mutation matrix is calculated.

[0016] For any mutation operation, a relative importance of the mutation operation is calculated according to the contribution values of the mutation operations.

[0017] According to the relative importance of the mutation operation, a probability of the mutation operation being selected in the morphology mutation matrix is updated.

[0018] Optionally, the contribution value of the mutation operation is calculated according to the following formula:

[0019] .

[0020] wherein, α i represents the i th mutation operation under the current environment, Ф(α i ) is the contribution value of the mutation operation α i , n represents the total number of mutation operations under the current environment, n! represents the factorial of n, m represents an evolution process set of any robot, M represents a set composed of the plurality of evolution process sets of different robots, r(m∪{α i}) represents a benefit value of the evolution process set obtained by merging the evolution process set m and the mutation operation α i , and r(m) represents a benefit value of the evolution process set m.

[0021] Optionally, the relative importance of the mutation operation is calculated according to the following formula:

[0022] .

[0023] wherein, I(α i) is the relative importance of the mutation operation a i , Ф(α i ) is the contribution value of the mutation operation a i , and n is the total number of mutation operations under the current environment.

[0024] Optionally, after sampling and executing the action policy according to the control policy of the robot and the environment perception information, the method further comprises the following steps:

[0025] Based on the proximal policy optimization algorithm, the model parameters of the control policy of the robot are optimized to maximize the target function.

[0026] Optionally, the target function is as shown in the following formula:

[0027] .

[0028] Wherein, L(θ) is the target function value, is the expected value at time step t, is the ratio of the new control policy to the old control policy, A t is the advantage function, clip(·,·,·) is the clipping function, the first item of the clipping function is the lower limit, the second item of the clipping function is the upper limit, ∈ is the hyperparameter, θ is the model parameter of the optimized control policy, θ old is the model parameter of the control policy before optimization, a t is the action policy, and s t is the environment perception information.

[0029] Optionally, according to the environment perception information and the system benefit value, it is judged whether the morphological gene vector of the robot is suitable for the current environment, and a first judgment result is obtained, which specifically comprises the following steps:

[0030] According to the environment perception information and the system benefit value, the environment perception function value is calculated.

[0031] If the environment perception function value is less than a preset threshold, the first judgment result is yes; otherwise, the first judgment result is no.

[0032] Optionally, the environment perception function value is calculated according to the following formula:

[0033] .

[0034] F(h, T) = ω1h + ω2T + b.

[0035] Wherein, D is the environment perception function value, k is the parameter of the control function slope, h is the terrain height, T is the change period, g is the current iteration number, is the optimal benefit value r bestCorresponding iteration number, omega1 and omega2 are the weight of terrain height and change period respectively, and b is bias.

[0036] Optionally, the mutation operation on the morphology gene vector of the robot specifically includes: adding a limb or a joint to the morphology gene vector of the robot, deleting a limb or a joint in the morphology gene vector of the robot, and modifying a physical attribute of a limb or a joint in the morphology gene vector of the robot.

[0037] The morphology gene vector of the robot is represented by the following formula:

[0038] G=(g0,g1,g2,g3,…,g l )。

[0039] Wherein, g0 represents the physical attribute of the head of the robot, g1-g l represent the physical attribute of the limb or joint of the robot, and any two limbs need to be connected by a joint.

[0040] On the other hand, the present application provides a computer device, comprising: a memory, a processor to store a computer program on the memory and run on the processor, and the processor executes the computer program to realize the steps of the robot morphology directional evolution method based on environment perception according to any one of the above.

[0041] On the other hand, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the robot morphology directional evolution method based on environment perception according to any one of the above.

[0042] On the other hand, the present application provides a computer program product, comprising a computer program, which is executed by a processor to realize the steps of the robot morphology directional evolution method based on environment perception according to any one of the above.

[0043] According to the specific embodiments of the present application, the following technical effects are disclosed:

[0044] The application provides a robot morphological directional evolution method based on environment perception and a related device, and the method comprises the following steps: initializing a morphological gene vector of a robot and a control strategy of the robot; sampling and executing an action strategy according to the control strategy of the robot and environment perception information; calculating a system benefit value according to the action strategy executed by the robot; judging whether the morphological gene vector of the robot is adaptive to a current environment according to the environment perception information and the system benefit value, and obtaining a first judgment result; if the first judgment result is yes, performing a mutation operation on the morphological gene vector of the robot based on a morphological mutation matrix; if the first judgment result is no, jumping to the step of controlling the robot to sample and execute an action strategy based on the control strategy and the environment perception information; and the process is repeated until a maximum iteration number is reached. The application quantifies the cumulative stimulation of environmental changes on the robot through an environment perception function to determine whether the robot needs to perform morphological evolution, and guides the direction of morphological evolution of the robot in different environments through a morphological mutation matrix to reduce additional calculation time, solve the defects of large search space and high calculation cost of traditional robot morphological evolution, and accelerate the robot morphological evolution process. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A flowchart of a robot morphological directional evolution method based on environment perception provided for Embodiment 1 of the present application;

[0047] Figure 2 A flowchart of steps S106-S108 in a robot morphological directional evolution method based on environment perception provided for Embodiment 1 of the present application;

[0048] Figure 3 A structural schematic diagram of a control strategy of a robot in a robot morphological directional evolution method based on environment perception provided for Embodiment 1 of the present application;

[0049] Figure 4 A schematic diagram of a morphological mutation operation of a robot in a robot morphological directional evolution method based on environment perception provided for Embodiment 1 of the present application;

[0050] Figure 5 An internal structure diagram of a computer device provided for Embodiment 4 of the present application. DETAILED DESCRIPTION

[0051] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] The present application aims to provide an environment perception based robot morphology directional evolution method and related device, aiming to improve the defects of large search space and high computation cost of traditional robot morphology evolution.

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0054] Embodiment 1

[0055] As shown in the flow chart, the environment perception based robot morphology directional evolution method in this embodiment includes the following steps: Figure 1

[0056] S101, initialize the morphology gene vector of the robot and the control strategy of the robot.

[0057] S102, sample and execute the action strategy according to the control strategy of the robot and the environment perception information. After sampling and executing the action strategy according to the control strategy of the robot and the environment perception information in step 102, the following steps are further included:

[0058] Based on the proximal policy optimization algorithm, the model parameters of the control strategy of the robot are optimized to maximize the objective function. The objective function is as follows:

[0059] .

[0060] Wherein, L(θ) is the objective function value, is the expected value at time step t, is the ratio of the new control strategy and the old control strategy, A t is the advantage function, clip(·,·,·) is the clipping function, the first item of the clipping function is the lower limit, the second item of the clipping function is the upper limit, ∈ is the hyperparameter, θ is the model parameter of the optimized control strategy, θ old is the model parameter of the control strategy before optimization, a t is the action strategy, s t is the environment perception information.

[0061] S103, calculate the system benefit value according to the action strategy executed by the robot. ​

[0062] S104, judging whether the morphology gene vector of the robot is adapted to the current environment according to the environment perception information and the system benefit value, and obtaining a first judgment result. Step S104 specifically includes the following steps:

[0063] S1041, calculating the environment perception function value according to the environment perception information and the system benefit value. The environment perception function value is calculated according to the following formula:

[0064] .

[0065] F(h, T) = ω1h + ω2T + b.

[0066] wherein D is the environment perception function value, k is a parameter of the slope of the control function, h is the terrain height, T is the change period, g is the current iteration number, is the optimal benefit value r before the current iteration number best corresponding iteration number, ω1 and ω2 are the weights of the terrain height and the change period respectively, and b is the bias.

[0067] If the environment perception function value is less than a preset threshold, step S1042 is executed, otherwise step S1043 is executed.

[0068] S1042, the first judgment result is assigned as yes.

[0069] S1043, the first judgment result is assigned as no.

[0070] If the first judgment result is yes, step S105 is executed; if the first judgment result is no, the process jumps to the step of controlling the robot to sample and execute the action strategy based on the control strategy and the environment perception information, until the maximum iteration number is reached.

[0071] S105, performing mutation operation on the morphology gene vector of the robot based on the morphology mutation matrix, and jumping to the step of controlling the robot to sample and execute the action strategy based on the control strategy and the environment perception information, until the maximum iteration number is reached.

[0072] The mutation operation on the morphology gene vector of the robot specifically includes adding a limb or a joint to the morphology gene vector of the robot, deleting a limb or a joint from the morphology gene vector of the robot, and modifying the physical properties of a limb or a joint in the morphology gene vector of the robot.

[0073] The morphology gene vector of the robot can be represented by the following formula:

[0074] G = (g0, g1, g2, g3, …, g l ).

[0075] Wherein, g0 represents the physical attribute of the head of the robot, g1-gn represent the physical attribute of the limbs or joints of the robot, and any two limbs need to be connected by a joint. l Wherein, g0 represents the physical attribute of the head of the robot, g1-gn represent the physical attribute of the limb or joint of the robot, and any two limbs need to be connected by a joint.

[0076] In the embodiment, the morphological mutation matrix includes a plurality of mutation operations and the probability of each mutation operation being selected, and the morphological mutation matrix is used to guide the robot to mutate. In order to make the morphological mutation matrix more reliable, the morphological mutation matrix also needs to be updated, such as Figure 2 The flowchart shown in the embodiment further includes the following steps after step S105:

[0077] S106, according to the plurality of evolution process sets of different robots and the optimal system benefit value corresponding to each robot, calculating the contribution value of each mutation operation in the morphological mutation matrix. Specifically, the contribution value of the mutation operation is calculated according to the following formula:

[0078] .

[0079] Wherein, α i represents the i-th mutation operation in the current environment, Ф(α i ) is the contribution value of the mutation operation α i , n represents the total number of mutation operations in the current environment, n! represents the factorial of n, m represents the evolution process set of any robot, M represents the set composed of the plurality of evolution process sets of different robots, r(m∪{α i}) represents the benefit value of the evolution process set after the combination of the evolution process set m and the mutation operation α i , and r(m) represents the benefit value of the evolution process set m.

[0080] S107, for any mutation operation, according to the contribution value of each mutation operation, calculating the relative importance of the mutation operation. Specifically, the relative importance of the mutation operation is calculated according to the following formula:

[0081] .

[0082] Wherein, I(α i ) is the relative importance of the mutation operation α i , Ф(α i ) is the contribution value of the mutation operation α i , and n is the total number of mutation operations in the current environment.

[0083] S108, according to the relative importance of the mutation operation, updating the probability of the mutation operation being selected in the morphological mutation matrix.

[0084] The above content of the embodiment illustrates a robot morphological directed evolution method based on environment perception in a step-by-step manner. Next, a robot morphological directed evolution method based on environment perception provided by the embodiment is described in a forward order, which includes the following steps:

[0085] Step one, initialize the morphological mutation matrix.

[0086] Robots can adapt to different environments or tasks through morphological evolution. Robots randomly select different mutation operations to produce diverse morphologies, including changing morphological structures and attributes. The morphologies generated by random evolution are searched in a wide morphological space, which requires a large computational cost. The present application sets four environments related to height, and robots will interact and learn in each environment to complete the task of forward movement. According to the robot morphology and the reward value obtained by the robot in the process of completing the task in a complex environment, a morphological mutation matrix is constructed, which has the following specific form:

[0087] .

[0088] Where each row represents a different environment, and the parameters in each row represent the probability of selecting each mutation operation in the current environment. Taking a ramp terrain parameterized by the amplitude of a sine wave as an example, when the robot cannot adapt to the current environment, the mutation operation with a higher probability is selected according to the morphological mutation matrix to achieve morphological directed evolution. Through the morphological mutation matrix, the direction of morphological evolution of the robot in the current environment can be guided to improve adaptability.

[0089] Step two, initialize the morphological gene vector and control strategy of the robot.

[0090] In the present embodiment, morphological directed evolution is composed of multiple robots. First, the number of limbs l of each morphology is randomly sampled, and the morphology G i of each robot is initialized to establish a gene vector to represent its morphological structure. The morphology of the robot is represented as a directed acyclic graph, with the head modeled as the starting point of the graph. The nodes in the graph represent the limbs or joints, and the node features include the physical attributes of the limbs. The joints connect the head and the limbs to form directed edges, and direct encoding is used to represent them as vectors. The gene vector has the form G=(g0, g1, g2, g3, …, g l ), where g represents the physical attribute value of the robot morphology, g0 is the physical attribute of the robot head, g1-g l are the physical attribute values of the limbs or joints when the morphology structure is traversed in breadth-first order. When the robot performs a mutation operation, the parameter values in the vector are adjusted to change the robot morphology.

[0091] Initialize the control strategy of each robot The robot interacts with the environment according to the current environment state s tperception information o t , by controlling policy π θ (a t |o t ) to take action a t , the network architecture is shown in Fig. Figure 3 The control policy contains two deep neural networks: the actor network outputs the probability distribution of actions, and the critic network evaluates the discounted future rewards of actions. The perception information of the robot in the environment (including terrain height, period, etc.) and self-perception information (including joint angles, angular velocity, and accelerometer, etc.) are taken as inputs, and each type of perception information is encoded into a vector by a two-layer MLP with hidden dimensions [64, 64], concatenated and further encoded into a 64-dimensional vector to pass to a linear layer, which outputs the Gaussian action policy parameters of the actor network and the discounted future rewards of the critic network. The size of the output layer depends on the number of driving joints.

[0092] Step three, the robot samples and performs actions in the environment, and optimizes the control policy parameters.

[0093] When the robot does not reach the maximum number of iterations, the robot randomly samples from the Gaussian action distribution policy in step two and performs actions, constantly interacting with the environment. At each time step, the robot selects an action by sampling the control policy π θ and interacts with the environment, and θ is the parameter of the two deep neural networks in the control policy in step two. The network parameters θ are optimized and learned through the proximal policy optimization algorithm (PPO) in the training process to maximize future rewards. The PPO algorithm maximizes the objective function, which includes two parts: the policy gradient term and the clipping term. The objective function is as follows:

[0094] .

[0095] where, is the expected value at time step t, is the ratio of the new policy to the old policy, A t is the advantage function, and clip(·,·,·) is a clipping function, where the first two items in the clip bracket are the upper and lower bounds, and the third item is the value to be clipped, and ∈ is a hyperparameter. Take the gradient of the objective function to get the gradient with respect to the network parameters θ, and use gradient ascent to perform parameter updates. By introducing the clipping term to limit the magnitude of each update, it ensures that the policy update is within a relatively small range, thereby improving the stability of the algorithm.

[0096] Step four, after performing the action, the terrain height and period change are perceived, and the reward value is recorded.

[0097] The task goal of the robot in the environment is to maximize forward movement in each round, and in step three, the control policy π θAfter sampling and performing actions, a reward value r can be obtained t The performance of the robot is measured by a reward function, which obtains a reward value at each time step:

[0098] r t = ω x v x - ω c ||a|| 2 .

[0099] where v x is the velocity component of the x-axis in the direction of motion, a is the input of the robot's effector, ω x and ω c are weights. The reward function encourages the robot to keep moving forward and obtain and record the reward value of the robot completing the task. Subsequently, based on the perception information in the environment and the reward value, it is determined whether the robot adapts to the current environment and whether it needs to evolve in form.

[0100] Step five, when the robot needs to change form to adapt to the environment, the form mutation matrix is used to guide the evolution direction of the robot's form, and the specific steps are as follows:

[0101] (1) At each iteration, the terrain height and periodic change information perceived in the environment are obtained, the reward value of the robot form after each iteration in step four is recorded, and the above information is input into the environment perception function; the robot perceives the change of the environment in the interaction process and monitors the terrain height and periodic change. The robot obtains a reward value when performing actions at each time step, and when the reward value stagnates under a given form, the robot cannot adapt to the current environment.

[0102] An environment perception function is established, which takes the terrain information perceived and the reward value as input to determine whether the robot evolves in form during the iterative learning process. The environment perception function is as follows:

[0103] .

[0104] F(h, T) = ω1h + ω2T + b.

[0105] where h and T represent terrain information, ω1 and ω2 are the weights of terrain height and change period respectively, h is the terrain height, T is the change period, b is the bias, and k is a parameter controlling the slope of the function; is the iteration number of the optimal reward value r best obtained in step four up to the current iteration number g. The function value D quantitatively represents the cumulative stimulation of environmental changes on the robot form.

[0106] (2) Compare the environment perception function value D in step (1) with a preset threshold to determine whether the robot evolves in form.

[0107] At the end of each iteration, the function value D is compared with a preset threshold value. If the function value D is less than the threshold value, it indicates that the robot in the current form cannot well adapt to the current environmental changes, and a mutation operation needs to be taken to evolve the form, step (3) is executed; if the function value D is greater than the threshold value, it indicates that the robot form and control strategy can learn to adapt to the environment, and the learning of the control strategy in the current form continues in steps three and four.

[0108] (3) According to the probability of each mutation operation in the form mutation matrix in step one, a mutation operation for evolving the current form is randomly selected, and the robot executes the mutation operation to generate a new robot form.

[0109] As shown in Figure 4 , it is a robot form mutation operation example of the embodiment of the application; for different environments, the robot can generate a new form through two types of mutation operations: 1) growing or deleting a limb 2) modifying the attributes of a limb or joint, including length, density and degrees of freedom, etc. When the robot needs to evolve the form in step S2, the mutation operation with a large probability value in the form mutation matrix in step one is preferentially selected. The robot selects and executes the mutation operation to change the parameter value of the gene vector G=(g0, g1, g2, g3, …, g l ) in step two, and then evolves to generate a new form, and repeats steps two to five to start adapting to the environment again.

[0110] Step six, according to the income value obtained by each robot in step four and the mutation operation selected by each robot in the evolution process in step five, the characteristic contribution value of each mutation operation is calculated.

[0111] The robot form space contains various forms, each form shows different performance when completing the task, and the robot form that can well adapt to the environment can be generated by executing different combinations of mutation operations from the initial form. When the robot evolves the form, there is a certain randomness in sampling the mutation operation according to the probability value in the above form mutation matrix.

[0112] When the above robot iterative learning ends, the mutation operation taken by the final robot form in step five in each evolution is recorded, the mutation operation combination selected by the final form of the first robot in the evolution process is represented as m1={α1, α2, α3, …, α k}, and the mutation operation combination selected by the final form of the i-th robot in the evolution process is represented as m i i={α1, α2, α3, …, α k}, the combination of the mutations selected by the robot in the evolution process forms a set M, M = {m1, m2, m3, …, m i}, where α k represents a mutation operation, and r(m) represents the optimal value of the robot form corresponding to each mutation operation combination m i obtained in step four.

[0113] Each mutation operation in the combination is a feature of the evolved form, and the contribution value of each feature in all combinations M is calculated using cooperative game theory:

[0114] .

[0115] where Ф(α i ) is the contribution value of the mutation operation α i , and r(m∪{α i})-r(m) represents the difference in the value of the combination m containing and not containing the mutation operation α i , and n is the total number of mutation operations. The parameter update of the subsequent form mutation matrix is based on the contribution value calculated for each mutation operation in the above process, and the necessary mutation operation for form evolution is identified.

[0116] Step seven, according to the contribution value of each mutation operation calculated in step six, the relative importance of each mutation operation is calculated, and the probability parameter of the form mutation matrix is updated.

[0117] In the above step one, a form mutation matrix is established and initialized to accelerate the process of robot form evolution, where the probability value in the matrix is adjusted based on the relative importance of each mutation operation. The contribution value calculated in the above step six can identify the relative importance of each mutation operation in the evolution of the robot form.

[0118] According to the contribution value of each mutation operation calculated in step six, the relative importance of each mutation operation is calculated by the following formula:

[0119] .

[0120] where I(α i ) is the relative importance of the mutation operation α i , and Ф(α i ) is the contribution value of the mutation operation α i calculated in the above step six. The higher the relative importance of the mutation operation, the greater the probability value of its selection. For the mutation operation, multiply the probability value of the mutation operation in the form mutation matrix by the corresponding relative importance, adjust the probability of each mutation operation being selected, and update the parameters of the form mutation matrix.

[0121] The robot morphological directional evolution method based on environment perception provided in the embodiment focuses on the morphological evolution of the robot in a complex environment, and introduces a morphological directional evolution method in the learning process, aiming to improve the defects of large search space and high computation cost of the traditional robot morphological evolution. In the scheme provided in the embodiment, the environment perception function and the morphological mutation matrix are combined to speed up the robot morphological evolution process. The environment perception function quantifies the cumulative stimulation of the environment change on the robot to determine whether the robot performs morphological evolution, and at the same time guides the direction of the robot morphological evolution to reduce the additional computation time and adapt to the environmental changes faster.

[0122] Embodiment 2

[0123] A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the steps of the robot morphological directional evolution method based on environment perception in embodiment 1.

[0124] Embodiment 3

[0125] A computer program product comprising a computer program, the computer program being executed by a processor to implement the steps of the robot morphological directional evolution method based on environment perception in embodiment 1.

[0126] Embodiment 4

[0127] A computer device, which can be a database, can have an internal structure as shown in Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store transactions to be processed. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the robot morphological directional evolution method based on environment perception in embodiment 1.

[0128] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0129] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Among them, any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0130] Each technical feature of the above embodiments can be combined arbitrarily, and in order to make the description concise, not all possible combinations of each technical feature in the above embodiments are described, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present application.

[0131] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. An environment-aware based robot morphological directed evolution method, characterized in that, The method comprises the following steps: initializing a morphology gene vector of a robot and a control strategy of the robot; sampling and executing an action strategy according to the control strategy of the robot and environmental perception information; calculating a system reward value according to the action strategy executed by the robot; judging whether the morphology gene vector of the robot is adaptive to a current environment according to the environmental perception information and the system reward value, and obtaining a first judgment result; if the first judgment result is yes, performing a mutation operation on the morphology gene vector of the robot based on a morphology mutation matrix, and jumping to the step of sampling and executing an action strategy according to the control strategy of the robot and environmental perception information until a maximum iteration number is reached; the morphology mutation matrix comprises a plurality of mutation operations and a probability of selection of each mutation operation, and the morphology mutation matrix is used to guide the robot to perform mutation; if the first judgment result is no, jumping to the step of sampling and executing an action strategy according to the control strategy of the robot and environmental perception information; until a maximum iteration number is reached; Based on the robot's control strategy and environmental perception information, the robot samples and executes action strategies, specifically including: the robot based on the current environmental state. s t Perceived information in o t Through robot control strategies π θ ( a t | o t Randomly sample and execute actions from a Gaussian action distribution policy. a t It interacts with the environment; the performance of the robot is measured by a reward function, and the reward function is as follows: ； wherein, r t is a system reward value, v x is a velocity component in the x-axis of the direction of motion, a is an input to the robot effector, ω x and ω c is a weight; the reward function encourages the robot to keep moving forward, to acquire and record the reward value for the robot completing the task; judging whether the morphology gene vector of the robot is adaptive to a current environment according to the environmental perception information and the system reward value, and obtaining a first judgment result, specifically comprising: calculating an environmental perception function value according to the environmental perception information and the system reward value; the environmental perception function value is calculated according to the following formula: ； ； wherein, D is an environmental perception function value, k is a parameter for controlling the slope of the function, h is a terrain height, T is a variation period, g is a current iteration number, is an optimal reward value before the current iteration number r best is a corresponding iteration number, ω 1 and ω 2 are weights for the terrain height and the variation period, respectively, b is a bias; if the environmental perception function value is less than a preset threshold, the first judgment result is yes; otherwise, the first judgment result is no.

2. The method of claim 1, wherein, The method further comprises the following steps: calculating a contribution value of each mutation operation in the morphology mutation matrix according to a plurality of evolution process sets of different robots and optimal system reward values corresponding to the robots; for any mutation operation, calculating a relative importance of the mutation operation according to the contribution values of the mutation operations; updating a probability of selection of the mutation operation in the morphology mutation matrix according to the relative importance of the mutation operation.

3. The method of claim 2, wherein, The contribution value of the mutation operation is calculated according to the following formula: ； wherein, α i represents the total number of mutation operations under the current environment, i an item mutation operation, Ф ( α i the contribution value of the mutation operation α i , n represents the total number of mutation operations under the current environment, n !represents n the factorial of m represents the evolution process set of any robot, M represents a set composed of multiple evolution process sets of different robots, r ( m ∪{ α i}) represents the evolution process set m merged with the mutation operation α i , r ( m ) represents the benefit value of the evolution process set m .

4. The method of claim 2, wherein, The relative importance of the mutation operation is calculated according to the following formula: ； wherein, I ( α i ) is the relative importance of the mutation operation α i , Ф ( α i ) is the contribution value of the mutation operation α i , n is the total number of mutation operations under the current environment.

5. The method of claim 1, wherein, After sampling and executing an action strategy according to the control strategy of the robot and environmental perception information, the method further comprises the following steps: optimizing model parameters of the control strategy of the robot based on a proximal policy optimization algorithm to maximize an objective function.

6. The method of claim 5, wherein, The objective function is as follows: ； wherein, L (·, ·, ·) is a clipping function, the first term of the clipping function is a lower limit, and the second term of the clipping function is an upper limit, θ ) is a target function value, is an expected value at a time step t , is a ratio of a new control policy and an old control policy, A t is an advantage function, clip (·, ·, ·) is a clipping function, the first term of the clipping function is a lower limit, and the second term of the clipping function is an upper limit, ϵ is a hyperparameter, θ is a model parameter of an optimized control policy, θ old is a model parameter of a control policy before optimization, a t is an action policy, s t is environment perception information.

7. The method of claim 1, wherein, The mutation operation on the morphology gene vector of the robot specifically comprises adding a limb or a joint to the morphology gene vector of the robot, deleting a limb or a joint in the morphology gene vector of the robot, or modifying a physical attribute of a limb or a joint in the morphology gene vector of the robot; The morphology gene vector of the robot is represented by the following formula: G =( g 0, g 1, g 2, g 3,…, g l ) ； wherein, g 0 represents a physical property of the head of the robot, g 1 g l 1 represents a physical property of a limb or joint of the robot, and any two limbs need to be connected with a joint.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the robot morphology directed evolution method based on environmental perception in any one of claims 1-7.

Images