Robot cerebellum reinforcement learning method and system based on adaptive environment change

Abstract

The application relates to the technical field of robot control, and discloses a robot cerebellum reinforcement learning method and system based on adaptive environmental changes, multi-modal sensing data in the running process of a robot is collected; based on the multi-modal sensing data, an environmental mutation increment and an environmental feature change amount are determined, and a main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount; a basic action vector of the robot is output based on the main strategy network, the state of the robot is valued, and a value evaluation amount is obtained; the basic action vector is updated and calculated based on the value evaluation amount, and a compensation action vector is obtained; the basic action vector and the compensation action vector are subjected to vector superposition and fusion, a final control vector is obtained, and the final control vector is taken as a cerebellum learning action instruction of the robot, dynamic adaptation of a reinforcement learning strategy and cerebellum rhythm control is realized, and the motion stability and learning convergence efficiency of the robot in an unstructured environment are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot cerebellum reinforcement learning method and system based on adaptive environmental changes. BACKGROUND

[0002] Robot motion control refers to a technical system for driving a robot body to complete a complex environment adaptation task by generating a command sequence that meets the kinematic constraints. This technology is in urgent demand in special operations, service interactions and other scenarios.

[0003] Existing robot reinforcement learning methods usually use model-free reinforcement learning methods (such as the PPO algorithm). Since they do not rely on an accurate environmental dynamics model, they can directly process high-dimensional state spaces and are widely used in the learning of complex skills such as robot walking and grasping. However, the existing technology has the following limitations:

[0004] (1) Insufficient environmental adaptation capability: the parameters of the trained policy network are fixed, making it difficult to cope with dynamic situations such as changes in ground material and sudden changes in load in the real environment;

[0005] (2) Low learning efficiency: when the environment changes, a large amount of data needs to be collected to retrain the entire network, which is time-consuming and inefficient in sample utilization;

[0006] (3) Model dependency contradiction: model-based methods are severely limited by model accuracy, while model-free methods lack the ability to quickly adapt to environmental changes;

[0007] (4) Cerebellum model potential not fully utilized: in existing solutions, CMAC is mostly used as a fixed parameter compensator, and its rapid online learning characteristics are not effectively utilized. SUMMARY

[0008] The robot cerebellum reinforcement learning method and system based on adaptive environmental changes provided by the embodiments of the present application solve the problems of response delay to environmental mutations, rigid policy network updates, and cerebellum control gain mismatch in existing robot motion control, achieve dynamic adaptation of reinforcement learning strategies and cerebellum rhythm control, and optimize the motion stability and learning convergence efficiency of robots in unstructured environments.

[0009] To achieve the above purpose, the present application provides a robot cerebellum reinforcement learning method based on adaptive environmental changes, comprising:

[0010] In the current learning cycle, multi-modal sensor data in the robot running process is collected, the multi-modal sensor data being the state response of the robot in the environmental change, and the multi-modal sensor data at least including visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data;

[0011] determine an environment mutation increment and an environment feature change amount based on the multi-modal sensor data, and obtain a main strategy network according to the environment mutation increment and the environment feature change amount;

[0012] output a basic action vector of the robot based on the main strategy network, perform value evaluation on the robot state to obtain a value evaluation amount;

[0013] update and calculate the basic action vector based on the value evaluation amount to obtain a compensation action vector;

[0014] perform vector superposition and fusion on the basic action vector and the compensation action vector to obtain a final control vector, and take the final control vector as a cerebellum learning action instruction of the robot.

[0015] Further, when determining an environment mutation increment and an environment feature change amount based on the multi-modal sensor data, and obtaining a main strategy network according to the environment mutation increment and the environment feature change amount, it comprises:

[0016] determine an environment mutation increment based on the multi-modal sensor data;

[0017] adopt a joint encoding mechanism of hierarchical perception and attention to extract features of the environment mutation increment to obtain an environment feature change amount;

[0018] fuse the environment feature change amount and a first control parameter to obtain a second control parameter, the first control parameter being a control reference value of a full amount state in a previous learning period, and the second control parameter being a control reference value of a full amount state in a current learning period;

[0019] obtain the main strategy network according to the first control parameter and the second control parameter.

[0020] Further, when determining an environment mutation increment based on the multi-modal sensor data, it comprises:

[0021] calculate an environment change degree of each spatial grid based on the multi-modal sensor data and a spatio-temporal convolutional neural network, the spatio-temporal convolutional neural network comprising a cascaded structure of a one-dimensional convolutional layer in the time dimension and a two-dimensional convolutional layer in the space dimension;

[0022] determine a mutation region and a state node associated with the mutation region based on the environment change degree and the multi-modal sensor data, the mutation region being a continuous spatial region with an environment change degree greater than a first threshold and a connectivity index greater than a second threshold, and the state node being a joint angle vector, a center of mass position coordinate, and an end effector pose matrix of the robot;

[0023] Based on the mutation region and the state node associated with the mutation region, an environment mutation increment is obtained through a graph structure construction algorithm, wherein the graph structure construction algorithm takes the mutation region as a parent node, takes the state node as a child node, and establishes a bidirectional connection edge, and the bidirectional connection edge contains a spatial distance weight and a time sequence related weight.

[0024] Further, when calculating the environment change degree of each spatial grid based on the multi-modal sensor data and the spatio-temporal convolutional neural network, it includes:

[0025] Obtain the first environment parameter distribution of the robot in the last learning period, which is obtained by probability density estimation of the environment parameters collected in the last learning period through a Gaussian mixture model;

[0026] Based on the multi-modal sensor data, obtain the second environment parameter distribution of the robot in the current learning period;

[0027] The first environment parameter distribution and the second environment parameter distribution are normalized, and the average of the KL divergence value and the JS divergence value is determined as the environment change degree.

[0028] Further, when obtaining the main policy network according to the first control parameter and the second control parameter, it includes:

[0029] Based on the first control parameter and the second control parameter, obtain the first policy value corresponding to the first control parameter and the second policy value corresponding to the second control parameter;

[0030] Calculate the root mean square error of the first policy value and the second policy value, and calculate the absolute value of the ratio of the root mean square error to the first policy value as the value difference degree;

[0031] Based on the adaptive learning rate adjustment formula, obtain the adjustment amplitude of the initial policy network, and determine the main policy network based on the adjustment amplitude.

[0032] Further, when the value evaluation of the robot state is obtained, it includes:

[0033] Obtain the state perception data of the robot;

[0034] Obtain the historical state data of the robot in the last learning period, and the historical state data is the joint motion trajectory of the robot at the historical time;

[0035] Based on the historical state data and the state perception data, calculate the state transition probability distribution;

[0036] constructing a value evaluation function based on the state transition probability distribution, and iteratively solving the value evaluation function to obtain the value evaluation quantity.

[0037] Further, in the updating and calculating of the base action vector based on the value evaluation quantity, the compensation action vector is obtained, comprising:

[0038] calculating an action advantage degree based on the value evaluation quantity, wherein the action advantage degree is a deviation degree of the base action vector from an average action value;

[0039] obtaining a historical action vector of the robot in a previous learning period;

[0040] constructing an action update gradient based on the historical action vector and the action advantage degree;

[0041] performing clipping processing on the action update gradient to obtain a gradient clipping value, and updating the base action vector based on the gradient clipping value to obtain the compensation action vector.

[0042] Further, in the updating of the base action vector based on the gradient clipping value, the compensation action vector is obtained, comprising:

[0043] when the action advantage degree is greater than a first advantage threshold and less than a first gradient threshold, a first compensation action vector is obtained, the first compensation action vector positively reinforces and corrects the base action vector;

[0044] when the action advantage degree is less than or equal to a second advantage threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensation action vector is obtained, the second compensation action vector conservatively restricts and corrects the base action vector; wherein the first advantage threshold is greater than the second advantage threshold.

[0045] Further, in the vector superposition and fusion of the base action vector and the compensation action vector, the final control vector is obtained, comprising:

[0046] performing a length normalization processing on the base action vector to obtain a unit base vector;

[0047] performing an amplitude scaling processing on the compensation action vector to obtain a scaled compensation vector;

[0048] calculating a vector inner product of the unit base vector and the scaled compensation vector to obtain a fusion weight value;

[0049] Based on the fusion weight value, the unit basis vector and the scaling compensation vector are weighted and summed to obtain the final control vector.

[0050] To achieve the above object, the application further provides a robot cerebellum reinforcement learning system based on adaptive environmental changes, comprising:

[0051] A data acquisition module is configured to acquire multi-modal sensor data of a robot during operation in a current learning period, wherein the multi-modal sensor data is a state response of the robot in environmental changes, and the multi-modal sensor data at least includes visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data.

[0052] A network determination module is configured to determine an environmental mutation increment and an environmental feature change amount based on the multi-modal sensor data, and obtain a main policy network according to the environmental mutation increment and the environmental feature change amount.

[0053] A value evaluation module is configured to evaluate a value of a state of the robot based on a basic action vector of the robot output by the main policy network, and obtain a value evaluation amount.

[0054] An action update module is configured to update and calculate the basic action vector based on the value evaluation amount, and obtain a compensation action vector.

[0055] A cerebellum learning module is configured to superimpose and fuse the basic action vector and the compensation action vector to obtain a final control vector, and use the final control vector as a cerebellum learning action instruction of the robot.

[0056] Compared with the prior art, the application has the following beneficial effects:

[0057] The application discloses a robot cerebellum reinforcement learning method and system based on adaptive environmental changes, multi-modal sensor data of a robot during operation is acquired, a main policy network is obtained based on an environmental mutation increment and an environmental feature change amount, a value of a state of the robot is evaluated based on a basic action vector of the robot output by the main policy network, a value evaluation amount is obtained, the basic action vector is updated and calculated based on the value evaluation amount, a compensation action vector is obtained, the basic action vector and the compensation action vector are superimposed and fused to obtain a final control vector, and the final control vector is used as a cerebellum learning action instruction of the robot, dynamic adaptation of a reinforcement learning strategy and cerebellum rhythm control is realized, and motion stability and learning convergence efficiency of the robot in an unstructured environment are optimized. BRIEF DESCRIPTION OF DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not intended to limit the application thereto. The same reference numerals in different drawings denote the same or similar components. In the drawings:

[0059] Figure 1 A flowchart of a robot cerebellum reinforcement learning method based on adaptive environment changes in an embodiment of the application is shown.

[0060] Figure 2 A structural diagram of a robot cerebellum reinforcement learning system based on adaptive environment changes in an embodiment of the application is shown. DETAILED DESCRIPTION

[0061] The specific embodiments of the present application will be further described with reference to the drawings and examples. The following examples are used to illustrate the present application but are not used to limit the scope of the present application.

[0062] In the description of the present application, it should be understood that the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0063] The terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0064] In the description of the present application, it should be noted that unless otherwise specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through intermediate medium, or the communication between the two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0065] The following is a description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

[0066] As Figure 1As shown, the embodiments of the present application disclose a robot cerebellum reinforcement learning method based on adaptive environmental changes, comprising:

[0067] S110: In the current learning period, multi-modal sensor data in the running process of the robot is collected, the multi-modal sensor data is the state response of the robot in the environmental change, and the multi-modal sensor data at least includes visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data;

[0068] In the embodiment, the current learning period refers to a time window for the robot to perform a single task, for example, a 10-minute inspection task is a learning period.

[0069] In the embodiment, the multi-modal sensor data is synchronously collected by the RGB-D camera, the six-axis IMU, the six-dimensional moment sensor, and the temperature and humidity sensor integrated in the robot body, and the data sampling frequency is unified to 50 Hz. The visual depth information is 640*480 pixel point cloud data, the inertial measurement unit data is three-axis acceleration and three-axis angular velocity, and the contact force feedback data is three-dimensional force and three-dimensional moment at the end of the foot.

[0070] S120: Based on the multi-modal sensor data, an environmental mutation increment and an environmental feature change amount are determined, and a main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount;

[0071] In some embodiments of the present application, when the environmental mutation increment and the environmental feature change amount are determined based on the multi-modal sensor data, and the main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount, it includes:

[0072] Based on the multi-modal sensor data, the environmental mutation increment is determined;

[0073] A hierarchical perception and attention joint encoding mechanism is adopted to extract features of the environmental mutation increment, and an environmental feature change amount is obtained;

[0074] The environmental feature change amount and a first control parameter are fused to obtain a second control parameter, the first control parameter is a control reference value of the full state in the last learning period, and the second control parameter is a control reference value of the full state in the current learning period;

[0075] The main strategy network is obtained according to the first control parameter and the second control parameter.

[0076] In the embodiment, the environmental mutation increment is first constructed into a heterogeneous graph (the mutation area is the parent node and the state node is the child node), then the current environmental feature is extracted by the hierarchical perception encoder, and finally the vector difference is made with the historical feature of the last period, that is, the environmental feature change amount is obtained.

[0077] In some embodiments of the present application, when determining the environmental mutation increment based on the multi-modal sensor data, comprising:

[0078] Based on the multi-modal sensor data and the spatio-temporal convolutional neural network, the environmental change degree of each spatial grid is calculated, the spatio-temporal convolutional neural network comprising a cascaded structure of a time dimension one-dimensional convolutional layer and a space dimension two-dimensional convolutional layer;

[0079] Based on the environmental change degree and the multi-modal sensor data, the mutation region and the state node associated with the mutation region are determined, the mutation region being a continuous spatial region with the environmental change degree greater than a first threshold and the connectivity index greater than a second threshold, and the state node being a joint angle vector, a center of mass position coordinate and an end effector pose matrix of the robot;

[0080] Based on the mutation region and the state node associated with the mutation region, the environmental mutation increment is obtained by a graph structure construction algorithm, wherein the graph structure construction algorithm takes the mutation region as a parent node, takes the state node as a child node and establishes a bidirectional connection edge, and the bidirectional connection edge contains a spatial distance weight and a time sequence correlation weight.

[0081] In the present embodiment, all multi-modal sensor data is normalized before processing the multi-modal sensor data. For example, a maximum-minimum normalization method.

[0082] In the present embodiment, the time dimension one-dimensional convolution kernel size of the spatio-temporal convolutional neural network is 5, the space dimension two-dimensional convolution kernel size is 3x3, and a batch normalization layer and a ReLU activation layer are inserted between the two convolutional layers. The calculation output of the environmental change degree is a scalar value for each spatial grid, and the spatial grid size is set to 0.5m x 0.5m. The first threshold is set to 0.6, the second threshold is set to 8, and the connectivity index is obtained by calculating the number of 8-neighbor connections of the grid region. The mutation region screening uses morphological opening operation to remove noise points and uses region growing algorithm to cluster connected domains, and only the region blocks with an area greater than 0.5 square meters are retained. The state node data is calculated in real time through the robot forward kinematics model, and the update frequency is 100 Hz. The graph structure construction uses a bidirectional linked list structure, the spatial distance weight is a negative exponential function of the Euclidean distance between two nodes, and the time sequence correlation weight is a negative power function of the time stamp difference between two nodes, and the power index is set to 0.5.

[0083] In this embodiment, the geometric center of the mutation region is obtained by calculating the arithmetic mean of the coordinates of all points in the region, and the region volume is obtained by calculating the convex hull volume. The joint angle vector of the state node contains 12 joint values, the centroid position coordinates are calculated by the robot dynamics model, and the end effector pose matrix is a 4x4 homogeneous transformation matrix. The spatial distance weight of the bidirectional connection edge is normalized between 0 and 1, and the time sequence correlation weight is set to 1 when the time interval is less than 0.1 seconds and 0 when the time interval is greater than 0.5 seconds.

[0084] In this embodiment, the environmental mutation increment is represented by a spatio-temporal graph structure, and the graph nodes are divided into two categories: environmental mutation region nodes and robot state nodes. The environmental mutation increment represents the region where the environmental topology has changed significantly and its associated robot state, such as a 30cm high obstacle suddenly appearing on the ground, a slope angle changing more than 15 degrees within 5 seconds, etc.

[0085] The beneficial effects of the above technical solutions are: the spatio-temporal convolutional neural network is used to quickly calculate the environmental change degree, improving the real-time perception; the graph structure construction algorithm is used to establish the association between the environment and the state, enhancing the causal reasoning ability; the bidirectional connection edge design supports the bidirectional propagation of information, improving the accuracy of state estimation.

[0086] In some embodiments of the present application, when calculating the environmental change degree of each spatial grid based on the multi-modal sensor data and the spatio-temporal convolutional neural network, it includes:

[0087] Obtain the first environmental parameter distribution of the robot in the last learning period, which is obtained by probabilistic density estimation of the environmental parameters collected in the last learning period by a Gaussian mixture model;

[0088] Based on the multi-modal sensor data, obtain the second environmental parameter distribution of the robot in the current learning period;

[0089] The first and second environmental parameter distributions are normalized, the KL divergence value and the JS divergence value are calculated respectively, and the average of the KL divergence value and the JS divergence value is determined as the environmental change degree.

[0090] In this embodiment, the Gaussian mixture model contains 5 Gaussian components, each component contains a mean vector, a covariance matrix, and a weight coefficient. The environmental parameters include terrain roughness, obstacle density, ground friction coefficient, light intensity, and noise level, with a total of 5 dimensions. The probabilistic density estimation is iteratively optimized using the expectation maximization algorithm, with an iteration number of 100 and a convergence threshold of 0.001.

[0091] In this embodiment, the second environment parameter distribution is estimated by a probability density estimation through the same Gaussian mixture model structure as the first environment parameter distribution.

[0092] In this embodiment, the Gaussian mixture model structure of the first environment parameter distribution is the same as that of the second environment parameter distribution, but the model parameters are different. The environment parameter collection obtains terrain point cloud data through a laser radar carried by a robot, counts the number and distribution of obstacles through a visual recognition algorithm, calculates the friction coefficient through a force sensor to measure the ground reaction force, measures the light intensity through a light-sensitive sensor, and measures the environmental noise through a microphone array. The modeling data of the first environment parameter distribution comes from the historical data of the last 30 seconds of the previous learning period, and the modeling data of the second environment parameter distribution comes from the real-time data of the last 5 seconds of the current learning period. The calculation of the KL divergence and the JS divergence is performed in parallel on the embedded GPU of the robot, and the calculation delay is less than 10 milliseconds.

[0093] In this embodiment, the KL divergence calculation adopts Monte Carlo sampling approximation, and the number of sampling points is 1000 points. The JS divergence calculation is obtained through symmetrization based on the KL divergence value. The normalization method adopts logarithmic normalization, that is, the natural logarithm of the KL divergence value and the JS divergence value is taken, and then divided by a preset maximum logarithmic value to ensure that the result falls within the interval of 0 to 1, and the maximum logarithmic value is set to 10.

[0094] The above technical scheme has the beneficial effects that: the Gaussian mixture model is used to probabilistically represent the environment parameters, improving the completeness of the environment description; the divergence value is calculated by comparing the double distributions, quantifying the statistical significance of the environmental changes; the average value fuses the KL and JS divergences, taking into account the directional and symmetric evaluation of the changes, making the environmental change degree calculation more robust.

[0095] In some embodiments of the present application, when the main policy network is obtained according to the first control parameter and the second control parameter, it includes:

[0096] Based on the first control parameter and the second control parameter, the first policy value corresponding to the first control parameter and the second policy value corresponding to the second control parameter are obtained.

[0097] The root mean square error of the first policy value and the second policy value is calculated, and the absolute value of the ratio of the root mean square error to the first policy value is calculated as the value difference degree.

[0098] The adjustment amplitude of the initial policy network is obtained based on the adaptive learning rate adjustment formula, and the main policy network is determined based on the adjustment amplitude.

[0099] In this embodiment, the policy value is output by a value evaluator subnetwork of the policy network. The value evaluator is a double-layer fully connected network, with a hidden layer dimension of 128 and a single-node linear output layer. The value evaluation uses a time-difference algorithm, with a discount factor γ set to 0.95. The TD error is calculated by the difference between the current reward and the future value estimate. The root mean square error is obtained by calculating the square mean of the difference between the first policy value and the second policy value, and then taking the square root. This calculation is performed within a rolling time window, with a window length set to 20 time steps.

[0100] In this embodiment, the value difference degree is a dimensionless ratio, and the first threshold is set to 0.3. When the value difference degree exceeds the threshold, it is determined that the environment has changed significantly. The adaptive learning rate adjustment formula is a piecewise function: when the value difference degree is less than 0.1, the adjustment amplitude is 0; when the value difference degree is between 0.1 and 0.3, the adjustment amplitude increases linearly; when the value difference degree is greater than 0.3, the adjustment amplitude reaches a saturation value, which is set to 5 times the baseline learning rate 0.001.

[0101] In this embodiment, the first policy value is calculated by the old policy network corresponding to the frozen first control parameter, and the second policy value is calculated by the new policy network corresponding to the learnable second control parameter. The calculation of the root mean square error uses an exponential moving average method, with a forgetting factor set to 0.9. A small constant is added to the denominator of the value difference degree to prevent division by zero errors. The adjustment amplitude is updated every 5 learning periods, and a sliding smoothing process is used for updating, with a smoothing coefficient set to 0.8.

[0102] In this embodiment, the main policy network is a three-layer fully connected neural network, with an input dimension of 128-dimensional environment feature vectors and an output dimension of 12-dimensional basic action vectors, corresponding to the desired angles of the hip joints and knee joints of the robot's limbs.

[0103] The above technical solution has the following beneficial effects: by comparing the double policy values to calculate the value difference degree, the influence of environmental changes on policy performance is objectively quantified; by using an adaptive learning rate adjustment formula to dynamically control the policy network update strength, the update speed is avoided to be too fast to cause oscillation or too slow to cause lag; by using a piecewise function design, stability and agility are considered, and the adaptive ability of the policy network to environmental changes is improved.

[0104] S130: performing value evaluation on the robot state based on the basic action vector of the robot output by the main policy network, to obtain a value evaluation quantity;

[0105] In this embodiment, the basic action vector is the original control instruction generated by the main policy network, which is the initial value before local correction.

[0106] In some embodiments of the present application, when the value evaluation quantity is obtained by performing value evaluation on the robot state, it includes:

[0107] obtain state perception data of the robot;

[0108] obtain historical state data of the robot in a previous learning period, the historical state data being a joint motion trajectory of the robot at a historical time;

[0109] calculate a state transition probability distribution based on the historical state data and the state perception data;

[0110] construct a value evaluation function based on the state transition probability distribution, and iteratively solve the value evaluation function to obtain the value evaluation quantity.

[0111] In this embodiment, the state perception data includes 12 joint angles, 12 joint angular velocities, three-axis center of mass positions, three-axis center of mass velocities, and three-dimensional contact forces of four points on the foot bottom, totaling 42-dimensional vectors. The historical state data is cached in an experience replay pool, and the cache capacity is set to 10,000 trajectories, each with a length of 50 time steps.

[0112] In this embodiment, the state transition probability distribution is approximated by constructing a state transition matrix, which has a dimension of 100x100. The state space is discretized by a K-means clustering algorithm, and the number of cluster centers is set to 100. The Euclidean distance is used as the distance metric for clustering.

[0113] In this embodiment, the value evaluation function adopts the form of Bellman equation, and the Reward function is composed of four parts: forward speed reward, energy consumption penalty, posture stability reward, and collision avoidance penalty. The weights of the four parts are 0.3, 0.2, 0.3, and 0.2, respectively. The iterative solution of the value evaluation function uses the value iteration algorithm, and the number of iterations is set to 50, and the convergence threshold is set to 0.01.

[0114] In this embodiment, the value evaluation function is: V(s) = r(s) + γ x Σ[P(s'|s,a) x V(s')], where V(s) is the value evaluation function, γ is the discount factor, r(s) is the immediate reward (the weighted sum of energy consumption penalty, stability reward, task progress reward, and collision avoidance penalty), P(s'|s,a) is the state transition probability distribution, and V(s') is the value of the next state. Here, the values of multiple next states can be determined, and then the product values of each next state value and the state transition probability distribution are determined, and finally the sum value is determined.

[0115] The beneficial effects of the above technical solutions are: by fusing historical state data and current perception data, the time sequence consistency of state evaluation is improved; by modeling the environmental dynamic characteristics through state transition probability distribution, the prediction accuracy of value evaluation is enhanced; by iterative solving through the value evaluation function, offline precalculation is realized, and the online calculation load is reduced.

[0116] S140: updating and calculating the base action vector based on the value evaluation quantity to obtain a compensation action vector;

[0117] In some embodiments of the present application, when the base action vector is updated and calculated based on the value evaluation quantity to obtain a compensation action vector, it includes:

[0118] Based on the value evaluation quantity, the action advantage degree is calculated, wherein the action advantage degree is the deviation degree of the base action vector from the average action value;

[0119] The historical action vector of the robot in the last learning cycle is obtained;

[0120] Based on the historical action vector and the action advantage degree, an action update gradient is constructed;

[0121] The action update gradient is clipped to obtain a gradient clipping value, and the base action vector is updated based on the gradient clipping value to obtain the compensation action vector.

[0122] In this embodiment, the action advantage degree is obtained by calculating the difference between the value evaluation quantity obtained by executing the base action vector in the current state and the average value evaluation quantity of executing all possible actions. The difference value is calculated under the assumption of Gaussian distribution in the action space, the mean of the Gaussian distribution is set as the base action vector, and the covariance matrix is initialized as the unit matrix.

[0123] In this embodiment, the historical action vector is stored in an action cache queue, the queue length is set to 100, and the cached actions are the last 100 base action vectors and their corresponding value evaluation quantities. The action update gradient is constructed by the policy gradient theorem, the gradient direction points to the direction of increasing action advantage degree, and the gradient amplitude is determined by the action advantage degree and the similarity of the historical action vector. The similarity is calculated by the cosine similarity, and when the similarity is higher than 0.8, the gradient amplitude is attenuated to 0.5 times. The gradient clipping threshold is set to 0.5, and when the gradient norm exceeds the threshold, it is scaled to the threshold size. The update of the compensation action vector adopts the momentum method, the momentum coefficient is set to 0.9, the update step is dynamically related to the learning rate, and the learning rate reference value is set to 0.01.

[0124] In this embodiment, the gradient clipping adopts a global clipping manner, that is, the L2 norm of all action dimension gradients is calculated, and when it exceeds the threshold, it is uniformly scaled.

[0125] The beneficial effects of the above technical solutions are: the improvement potential of the basis action vector is quantified by the action advantage, guiding the compensation direction; the historical action vector information is used to avoid repeated exploration of inefficient actions, improving the learning sample utilization; the gradient clipping is used to prevent the update step from being too large to cause policy collapse, ensuring the learning stability; the compensation action vector is used to realize local fine correction, enhancing the adaptability of the policy to the environment details.

[0126] In some embodiments of the present application, when the basis action vector is updated based on the gradient clipping value to obtain the compensation action vector, it includes:

[0127] When the action advantage is greater than a first advantage threshold and less than a first gradient threshold, a first compensation action vector is obtained, which positively reinforces and corrects the basis action vector;

[0128] When the action advantage is less than or equal to a second advantage threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensation action vector is obtained, which conservatively constrains and corrects the basis action vector; wherein the first advantage threshold is greater than the second advantage threshold.

[0129] In this embodiment, the first advantage threshold is set to 0.2, the second advantage threshold is set to 0.05, and the first gradient threshold is set to 0.5. When the action advantage is greater than 0.2 and the gradient clipping value is less than 0.5, the positive reinforcement correction mode is enabled. The positive reinforcement correction is realized by adding the gradient clipping value multiplied by the amplification coefficient 1.5 to the basis action vector, the addition method is vector addition, the amplification coefficient is dynamically adjusted according to the size of the action advantage, the larger the advantage, the larger the amplification coefficient, and the maximum does not exceed 2. When the action advantage is less than or equal to 0.05, or when the gradient clipping value is greater than or equal to 0.5, it is determined that there is a risk of update shock, and the conservative constraint correction mode is enabled. The conservative constraint correction is realized by adding the gradient clipping value multiplied by the attenuation coefficient 0.5 to the basis action vector, and the attenuation coefficient is not less than 0.2 at the lowest, ensuring that the correction direction is correct but the amplitude is limited. The difference between the first advantage threshold and the second advantage threshold is set to 0.15 to form a buffer interval, avoiding frequent switching of the correction mode causing jitter.

[0130] S150: Vector addition and fusion of the basis action vector and the compensation action vector are performed to obtain a final control vector, and the final control vector is used as the cerebellar learning action instruction of the robot.

[0131] In some embodiments of the present application, when the basis action vector and the compensation action vector are vector added and fused to obtain a final control vector, it includes:

[0132] length normalization processing is performed on the basis action vector to obtain a unit basis vector;

[0133] amplitude scaling processing is performed on the compensatory action vector to obtain a scaled compensatory vector;

[0134] vector inner product of the unit basis vector and the scaled compensatory vector is calculated to obtain a fusion weight value;

[0135] based on the fusion weight value, the unit basis vector and the scaled compensatory vector are weighted and summed to obtain the final control vector.

[0136] In this embodiment, the length normalization of the basis action vector adopts L2 norm normalization, that is, the vector is divided by its Euclidean length, and if the vector length is zero, it is kept as a zero vector. The amplitude scaling of the compensatory action vector is realized by a scaling coefficient, which is set to 0.3. The coefficient is dynamically adjusted according to the confidence of the compensatory action vector, and the confidence is obtained by calculating the average cosine similarity of the compensatory action vector and the historical successful compensatory vector. When the similarity is higher than 0.8, the scaling coefficient is increased to 0.5, and when the similarity is lower than 0.3, the scaling coefficient is reduced to 0.1. The vector inner product calculation of the unit basis vector and the scaled compensatory vector is performed in a 12-dimensional action space, and the inner product result ranges from -1 to 1. The fusion weight value is converted to the interval of 0 to 1 by linear mapping, and the mapping formula is (inner product value + 1) / 2. The weighted sum adopts dynamic weight distribution. When the fusion weight value is greater than 0.7, the weight of the unit basis vector is 0.3 and the weight of the scaled compensatory vector is 0.7; when the fusion weight value is less than 0.3, the weight of the unit basis vector is 0.8 and the weight of the scaled compensatory vector is 0.2; and when the fusion weight value is between 0.3 and 0.7, the weight is linearly interpolated.

[0137] The beneficial effects of the above technical solution are: the length normalization preserves the global planning information of the basis action, preventing the compensatory action from distorting the overall motion direction; the amplitude scaling controls the correction strength of the compensatory action, avoiding local over-adjustment; the vector inner product dynamically evaluates the compensatory confidence, realizing adaptive weight distribution; the weighted sum integrates global planning and local correction to generate a final control vector with foresight and adaptability, improving the overall performance and robustness of robot motion control.

[0138] In order to further illustrate the technical idea of the application, the technical solution of the application will be described in conjunction with specific application scenarios.

[0139] Correspondingly, as shown in Figure 2 The application also provides a robot cerebellum reinforcement learning system based on adaptive environmental changes, which comprises:

[0140] The data acquisition module is configured to acquire multi-modal sensing data of the robot during a current learning period, the multi-modal sensing data being state responses of the robot in an environment change, and the multi-modal sensing data including at least visual depth information, inertial measurement unit data, contact force feedback data, and environment temperature and humidity data.

[0141] The network determination module is configured to determine an environment mutation increment and an environment feature change amount based on the multi-modal sensing data, and obtain a main strategy network according to the environment mutation increment and the environment feature change amount.

[0142] The value evaluation module is configured to perform value evaluation on a state of the robot based on the main strategy network outputting a basic action vector of the robot, and obtain a value evaluation amount.

[0143] The action update module is configured to perform update calculation on the basic action vector based on the value evaluation amount, and obtain a compensation action vector.

[0144] The cerebellum learning module is configured to perform vector superposition and fusion on the basic action vector and the compensation action vector, obtain a final control vector, and take the final control vector as a cerebellum learning action instruction of the robot.

[0145] In the description of the above-described embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0146] Although the present application has been described with reference to the embodiments above, various modifications can be made to it without departing from the scope of the present application and equivalent components can be substituted therefor. In particular, each feature in the embodiments disclosed in the present specification can be combined with any other features in any manner, provided that there is no structural conflict. The combinations of these features are not all described in the present specification only for the purpose of saving space and resources.

[0147] It should be understood by those skilled in the art that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some technical features. Any modification, equivalent substitution, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A robot cerebellum reinforcement learning method based on adaptive environmental changes, characterized in that, include: During the current learning cycle, multimodal sensing data is collected during the robot's operation. The multimodal sensing data represents the robot's state response to environmental changes. The multimodal sensing data includes at least visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data. Based on the multimodal sensing data, the environmental mutation increment and the environmental feature change amount are determined, and the main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount. Based on the basic motion vector of the robot output by the main policy network, the value of the robot's state is evaluated to obtain the value evaluation quantity. The basic action vector is updated and calculated based on the value assessment to obtain the compensation action vector; The basic motion vector and the compensated motion vector are superimposed and fused to obtain the final control vector, which is then used as the cerebellum learning motion command for the robot.

2. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 1, characterized in that, When determining the environmental abrupt change increment and the environmental feature change amount based on the multimodal sensing data, and obtaining the main policy network based on the environmental abrupt change increment and the environmental feature change amount, the process includes: Based on the multimodal sensing data, determine the incremental changes in environmental abrupt changes; A joint encoding mechanism of hierarchical perception and attention is adopted to extract features from the environmental mutation increment to obtain the environmental feature change quantity; The environmental characteristic change and the first control parameter are fused to obtain the second control parameter. The first control parameter is the control reference value of the full state in the previous learning cycle, and the second control parameter is the control reference value of the full state in the current learning cycle. The main strategy network is obtained based on the first control parameter and the second control parameter.

3. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 2, characterized in that, When determining the environmental abrupt change increment based on the multimodal sensing data, the following are included: The environmental variability of each spatial grid is calculated based on the multimodal sensing data and the spatiotemporal convolutional neural network, which includes a cascaded structure of a one-dimensional convolutional layer in the time dimension and a two-dimensional convolutional layer in the spatial dimension. Based on the environmental variability and the multimodal sensing data, abrupt change regions and state nodes associated with the abrupt change regions are determined. The abrupt change regions are continuous spatial regions where the environmental variability is greater than a first threshold and the connectivity index is greater than a second threshold. The state nodes are the robot's joint angle vectors, centroid position coordinates, and end effector pose matrix. Based on the mutation region and the state nodes associated with the mutation region, the environmental mutation increment is obtained through a graph structure construction algorithm. The graph structure construction algorithm takes the mutation region as the parent node, the state node as the child node, and establishes bidirectional connecting edges. The bidirectional connecting edges include spatial distance weights and temporal correlation weights.

4. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 3, characterized in that, When calculating the environmental variability of each spatial grid based on the multimodal sensing data and the spatiotemporal convolutional neural network, the following steps are included: The robot obtains a first environmental parameter distribution during the previous learning cycle. The first environmental parameter distribution is obtained by estimating the probability density of the environmental parameters collected during the previous learning cycle using a Gaussian mixture model. Based on the multimodal sensing data, the distribution of the second environmental parameters of the robot during the current learning cycle is obtained; The distributions of the first and second environmental parameters are normalized, and the KL divergence and JS divergence values ​​are calculated respectively. The average value of the KL divergence and JS divergence values ​​is determined as the degree of environmental variability.

5. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 3, characterized in that, When obtaining the main policy network based on the first control parameter and the second control parameter, the process includes: Based on the first control parameter and the second control parameter, obtain the first strategy value corresponding to the first control parameter and the second strategy value corresponding to the second control parameter; Calculate the root mean square error between the value of the first strategy and the value of the second strategy, and calculate the absolute value of the ratio of the root mean square error to the value of the first strategy as the value difference. The adjustment range of the initial policy network is obtained based on the adaptive learning rate adjustment formula, and the main policy network is determined based on the adjustment range.

6. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 1, characterized in that, When assessing the value of the robot's state to obtain the assessed value, the following steps are included: Acquire the robot's state perception data; Obtain the historical state data of the robot in the previous learning cycle, wherein the historical state data is the joint motion trajectory of the robot at a historical moment; Based on the historical state data and the state perception data, calculate the state transition probability distribution; Based on the state transition probability distribution, a value assessment function is constructed, and the value assessment function is iteratively solved to obtain the value assessment quantity.

7. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 1, characterized in that, When updating the basic action vector based on the value assessment to obtain the compensated action vector, the following steps are included: Based on the value assessment, the action advantage is calculated, wherein the action advantage is the degree of deviation between the basic action vector and the average action value; Obtain the robot's historical action vectors from the previous learning cycle; Based on the historical action vectors and the action dominance, an action update gradient is constructed; The action update gradient is clipped to obtain a gradient clipping value, and the base action vector is updated based on the gradient clipping value to obtain the compensated action vector.

8. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 7, characterized in that, When updating the base action vector based on the gradient clipping value to obtain the compensated action vector, the process includes: When the action dominance is greater than the first dominance threshold and less than the first gradient threshold, a first compensated action vector is obtained, and the first compensated action vector positively strengthens and corrects the base action vector. When the action dominance is less than or equal to the second dominance threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensated action vector is obtained, and the second compensated action vector performs conservative constraint correction on the base action vector; wherein, the first dominance threshold is greater than the second dominance threshold.

9. The robot cerebellum reinforcement learning method based on adaptive environmental changes according to claim 1, characterized in that, When performing vector superposition and fusion of the basic motion vector and the compensated motion vector to obtain the final control vector, the following steps are included: The basic action vector is normalized to obtain a unit basic vector; The compensation action vector is subjected to amplitude scaling to obtain a scaled compensation vector; Calculate the dot product of the unit base vector and the scaling compensation vector to obtain the fusion weight value; Based on the fusion weight value, the unit base vector and the scaling compensation vector are weighted and summed to obtain the final control vector.

10. A robot cerebellum reinforcement learning system based on adaptive environmental changes, applied to the robot cerebellum reinforcement learning method based on adaptive environmental changes as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to collect multimodal sensing data during the robot's operation within the current learning cycle. The multimodal sensing data is the robot's state response to environmental changes. The multimodal sensing data includes at least visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data. The network determination module is used to determine the environmental mutation increment and the environmental feature change amount based on the multimodal sensing data, and to obtain the main strategy network according to the environmental mutation increment and the environmental feature change amount. The value assessment module is used to assess the value of the robot's state based on the robot's basic motion vector output by the main policy network, and obtain the value assessment quantity. The action update module is used to update the basic action vector based on the value assessment quantity to obtain the compensation action vector; The cerebellum learning module is used to perform vector superposition and fusion of the basic action vector and the compensated action vector to obtain the final control vector, and to use the final control vector as the cerebellum learning action command of the robot.

Images