Robot cerebellum reinforcement learning method and system based on adaptive environment change

Abstract

The invention relates to the technical field of robot control, and discloses a robot cerebellum reinforcement learning method and system based on adaptive environment change, and the method comprises the steps: collecting multi-modal sensing data in the operation process of a robot; based on the multi-modal sensing data, determining an environment sudden change increment and an environment characteristic variable quantity, and obtaining a main strategy network according to the environment sudden change increment and the environment characteristic variable quantity; outputting a basic action vector of the robot based on the main strategy network, and performing value evaluation on the state of the robot to obtain a value evaluation quantity; performing updating calculation on the basic action vector based on the value evaluation quantity to obtain a compensation action vector; and performing vector superposition fusion on the basic action vector and the compensation action vector to obtain a final control vector, and taking the final control vector as a cerebellum learning action instruction of the robot, thereby realizing dynamic adaptation of a reinforcement learning strategy and cerebellum rhythm control. And the motion stability and the learning convergence efficiency of the robot in the unstructured environment are optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot cerebellum reinforcement learning method and system based on adaptive environmental changes. BACKGROUND

[0002] Robot motion control refers to a technical system for driving a robot body to complete a complex environment adaptation task by generating a command sequence that meets the kinematic constraints. This technology is urgently needed in special operations, service interactions and other scenarios.

[0003] Existing robot reinforcement learning methods usually use model-free reinforcement learning methods (such as PPO algorithm). Because they do not rely on accurate environmental dynamics models, they can directly process high-dimensional state spaces and are widely used in learning complex skills such as robot walking and grasping. However, the existing technology has the following limitations: (1) Insufficient environmental adaptation: the trained policy network parameters are fixed, making it difficult to cope with dynamic situations such as changes in ground material and sudden changes in load in real environments; (2) Low learning efficiency: when the environment changes, a large amount of data needs to be collected to retrain the entire network, which is time-consuming and inefficient in sample utilization; (3) Model dependency contradiction: model-based methods are severely limited by model accuracy, while model-free methods lack the ability to quickly adapt to environmental changes; (4) Cerebellum model potential not fully utilized: in existing solutions, CMAC is mostly used as a fixed parameter compensator, and its fast online learning characteristics are not effectively utilized. SUMMARY

[0004] The robot cerebellum reinforcement learning method and system based on adaptive environmental changes provided by the embodiments of the present application solve the problems of response delay to environmental mutations, rigid policy network updates, and cerebellum control gain mismatch in existing robot motion control, achieve dynamic adaptation of reinforcement learning strategies and cerebellum rhythm control, and optimize the motion stability and learning convergence efficiency of robots in unstructured environments.

[0005] To achieve the above purpose, the present application provides a robot cerebellum reinforcement learning method based on adaptive environmental changes, comprising: In the current learning cycle, multi-modal sensor data during robot operation is collected, the multi-modal sensor data being the state response of the robot in the environmental change, and the multi-modal sensor data at least including visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data; Based on the multi-modal sensor data, an environmental mutation increment and an environmental feature change amount are determined, and a main policy network is obtained according to the environmental mutation increment and the environmental feature change amount. based on the base action vector output by the main policy network, value evaluation is performed on the robot state to obtain a value evaluation quantity; based on the value evaluation quantity, the base action vector is updated and calculated to obtain a compensation action vector; the base action vector and the compensation action vector are vector superimposed and fused to obtain a final control vector, and the final control vector is taken as the cerebellum learning action instruction of the robot.

[0006] Further, when determining the environment mutation increment and the environment feature change quantity based on the multi-modal sensor data, and obtaining the main policy network according to the environment mutation increment and the environment feature change quantity, comprising: based on the multi-modal sensor data, an environment mutation increment is determined; a hierarchical perception and attention joint encoding mechanism is adopted to extract features of the environment mutation increment to obtain an environment feature change quantity; the environment feature change quantity and a first control parameter are fused to obtain a second control parameter, the first control parameter being a control reference value of a full-quantity state in a previous learning period, and the second control parameter being a control reference value of a full-quantity state in a current learning period; the main policy network is obtained according to the first control parameter and the second control parameter.

[0007] Further, when determining the environment mutation increment based on the multi-modal sensor data, comprising: based on the multi-modal sensor data and a spatio-temporal convolutional neural network, an environment change degree of each spatial grid is calculated, the spatio-temporal convolutional neural network including a cascaded structure of a time dimension one-dimensional convolutional layer and a space dimension two-dimensional convolutional layer; based on the environment change degree and the multi-modal sensor data, a mutation region and a state node associated with the mutation region are determined, the mutation region being a continuous spatial region with an environment change degree greater than a first threshold and a connectivity index greater than a second threshold, and the state node being a joint angle vector, a center of mass position coordinate and an end effector pose matrix of the robot; based on the mutation region and the state node associated with the mutation region, the environment mutation increment is obtained by a graph structure construction algorithm, wherein the graph structure construction algorithm takes the mutation region as a parent node, takes the state node as a child node and establishes a bidirectional connection edge, and the bidirectional connection edge includes a spatial distance weight and a time sequence related weight.

[0008] Further, when calculating the environment change degree of each spatial grid based on the multi-modal sensor data and the spatio-temporal convolutional neural network, comprising: obtain a first environment parameter distribution of the robot in the previous learning period, the first environment parameter distribution being obtained by a Gaussian mixture model for probabilistic density estimation on environment parameters collected in the previous learning period; obtain a second environment parameter distribution of the robot in the current learning period based on the multi-modal sensing data; normalize the first environment parameter distribution and the second environment parameter distribution, calculate a KL divergence value and a JS divergence value respectively, and determine an average value of the KL divergence value and the JS divergence value as the environment change degree.

[0009] Further, when the main policy network is obtained according to the first control parameter and the second control parameter, the method comprises: obtain a first policy value corresponding to the first control parameter and a second policy value corresponding to the second control parameter based on the first control parameter and the second control parameter; calculate a root mean square error of the first policy value and the second policy value, and calculate an absolute value of a ratio of the root mean square error to the first policy value as a value difference degree; obtain an adjustment amplitude of an initial policy network based on an adaptive learning rate adjustment formula, and determine the main policy network based on the adjustment amplitude.

[0010] Further, when the value evaluation quantity is obtained by performing value evaluation on the robot state, the method comprises: obtain state perception data of the robot; obtain historical state data of the robot in a previous learning period, the historical state data being a joint motion trajectory of the robot at a historical time; calculate a state transition probability distribution based on the historical state data and the state perception data; construct a value evaluation function based on the state transition probability distribution, and iteratively solve the value evaluation function to obtain the value evaluation quantity.

[0011] Further, when the compensation action vector is obtained by updating and calculating the basic action vector based on the value evaluation quantity, the method comprises: calculate an action advantage degree based on the value evaluation quantity, wherein the action advantage degree is a degree of deviation of the basic action vector from an average action value; obtain a historical action vector of the robot in a previous learning period; construct an action update gradient based on the historical action vector and the action advantage degree; The gradient of the action is clipped to obtain a gradient clipping value, and the base action vector is updated based on the gradient clipping value to obtain the compensation action vector.

[0012] Further, when the base action vector is updated based on the gradient clipping value to obtain the compensation action vector, the following steps are included: When the action dominance is greater than a first dominance threshold and less than a first gradient threshold, a first compensation action vector is obtained, which positively reinforces the base action vector; When the action dominance is less than or equal to a second dominance threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensation action vector is obtained, which conservatively constrains the base action vector; wherein the first dominance threshold is greater than the second dominance threshold.

[0013] Further, when the base action vector and the compensation action vector are vector superimposed and fused to obtain the final control vector, the following steps are included: The base action vector is length normalized to obtain a unit base vector; The compensation action vector is amplitude scaled to obtain a scaled compensation vector; The vector inner product of the unit base vector and the scaled compensation vector is calculated to obtain a fusion weight value; Based on the fusion weight value, the unit base vector and the scaled compensation vector are weighted and summed to obtain the final control vector.

[0014] In order to achieve the above purpose, the application also provides a robot cerebellum reinforcement learning system based on adaptive environmental changes, comprising: A data acquisition module is configured to acquire multi-modal sensor data during the operation of a robot in a current learning period, wherein the multi-modal sensor data is the state response of the robot in the environmental change, and the multi-modal sensor data at least includes visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data. A network determination module is configured to determine an environmental mutation increment and an environmental feature change amount based on the multi-modal sensor data, and obtain a main policy network according to the environmental mutation increment and the environmental feature change amount. A value evaluation module is configured to output a base action vector of the robot based on the main policy network, evaluate the state of the robot, and obtain a value evaluation amount. An action update module is configured to update and calculate the base action vector based on the value evaluation amount to obtain a compensation action vector. The cerebellum learning module is used to perform vector superposition and fusion of the basic action vector and the compensated action vector to obtain the final control vector, and to use the final control vector as the cerebellum learning action command of the robot.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention discloses a robot cerebellar reinforcement learning method and system based on adaptive environmental changes. The method involves collecting multimodal sensor data during robot operation; determining environmental mutation increments and environmental feature changes based on the multimodal sensor data, and obtaining a master policy network based on these increments and changes; outputting the robot's basic action vectors from the master policy network to evaluate the robot's state and obtain a value assessment value; updating the basic action vectors based on the value assessment value to obtain a compensation action vector; and then superimposing and fusing the basic action vectors and compensation action vectors to obtain the final control vector. This final control vector is used as the robot's cerebellar learning action command, achieving dynamic adaptation between the reinforcement learning strategy and cerebellar rhythm control, and optimizing the robot's motion stability and learning convergence efficiency in unstructured environments. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the robot cerebellum reinforcement learning method based on adaptive environmental changes in an embodiment of the present invention is shown. Figure 2 A schematic diagram of the structure of a robot cerebellum reinforcement learning system based on adaptive environmental changes is shown in an embodiment of the present invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0018] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0019] The terms "first", "second", "third", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or an indicated number of features. Thus, a feature defined with "first", "second", etc. can include one or more of that feature. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0020] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0021] The following is a description of the preferred embodiments of the present application in conjunction with the accompanying drawings.

[0022] As shown in Figure 1 The embodiments of the present application disclose a robot cerebellum reinforcement learning method based on adaptive environmental changes, comprising: S110: In the current learning period, multi-modal sensor data in the running process of the robot is collected, the multi-modal sensor data is the state response of the robot in the environmental change, and the multi-modal sensor data at least includes visual depth information, inertial measurement unit data, contact force feedback data and environmental temperature and humidity data; In the present embodiment, the current learning period refers to the time window of a single task performed by the robot, for example, a 10-minute inspection task is a learning period.

[0023] In the present embodiment, the multi-modal sensor data is synchronously collected by the RGB-D camera, six-axis IMU, six-dimensional torque sensor and temperature and humidity sensor integrated in the robot body, and the data sampling frequency is unified to 50 Hz. The visual depth information is 640x480 pixel point cloud data, the inertial measurement unit data is three-axis acceleration and three-axis angular velocity, and the contact force feedback data is three-dimensional force and three-dimensional torque at the end of the foot.

[0024] S120: Based on the multi-modal sensor data, the environmental mutation increment and the environmental feature change amount are determined, and the main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount; In some embodiments of the present application, when the environmental mutation increment and the environmental feature change amount are determined based on the multi-modal sensor data, and the main strategy network is obtained according to the environmental mutation increment and the environmental feature change amount, it comprises: determine an environment mutation increment based on the multi-modal sensor data; extract features of the environment mutation increment by adopting a joint encoding mechanism of hierarchical perception and attention, to obtain an environment feature change amount; fuse the environment feature change amount and a first control parameter to obtain a second control parameter, the first control parameter being a control reference value of a full amount state in a previous learning period, and the second control parameter being a control reference value of a full amount state in a current learning period; obtain the main policy network according to the first control parameter and the second control parameter.

[0025] In this embodiment, the environment mutation increment is first constructed into a heterogeneous graph (mutation region as parent node and state node as child node), then input into a hierarchical perception encoder to extract current environment features, and finally vector difference is performed with historical features in the previous period, to obtain the environment feature change amount.

[0026] In some embodiments of the present application, when determining the environment mutation increment based on the multi-modal sensor data, the following steps are included: calculate an environment change degree of each spatial grid based on the multi-modal sensor data and a spatio-temporal convolutional neural network, the spatio-temporal convolutional neural network including a cascaded structure of a one-dimensional convolutional layer in time dimension and a two-dimensional convolutional layer in space dimension; determine a mutation region and state nodes associated with the mutation region based on the environment change degree and the multi-modal sensor data, the mutation region being a continuous spatial region with an environment change degree greater than a first threshold and a connectivity index greater than a second threshold, and the state nodes being joint angle vectors, mass center position coordinates and end effector pose matrices of the robot; obtain the environment mutation increment based on the mutation region and the state nodes associated with the mutation region by a graph structure construction algorithm, wherein the graph structure construction algorithm takes the mutation region as a parent node, takes the state nodes as child nodes and establishes bidirectional connection edges, and the bidirectional connection edges include spatial distance weights and time sequence related weights.

[0027] In this embodiment, all multi-modal sensor data is normalized before being processed. For example, a maximum-minimum normalization method.

[0028] In this embodiment, the time dimension one-dimensional convolution kernel size of the spatio-temporal convolutional neural network is 5, the spatial dimension two-dimensional convolution kernel size is 3x3, and a batch normalization layer and a ReLU activation layer are inserted between the two convolution layers. The calculation output of the environmental change degree is a scalar value for each spatial grid, and the spatial grid size is set to 0.5m x 0.5m. The first threshold is set to 0.6, the second threshold is set to 8, and the connectivity index is obtained by calculating the number of 8-neighbor connections of the grid region. The mutation region screening uses morphological opening operation to remove noise points, and uses region growing algorithm to cluster connected domains, and retains the region block with an area greater than 0.5 square meters. The state node data is calculated in real time through the robot forward kinematics model, and the update frequency is 100 Hz. The graph structure construction adopts a double-linked list structure, the spatial distance weight is a negative exponential function of the Euclidean distance between two nodes, and the time sequence correlation weight is a negative power function of the time stamp difference between two nodes, and the power index is set to 0.5.

[0029] In this embodiment, the geometric center of the mutation region is obtained by calculating the arithmetic mean of the coordinates of all points in the region, and the region volume is obtained by calculating the convex hull volume. The joint angle vector of the state node contains 12 joint values, the centroid position coordinates are calculated through the robot dynamics model, and the end effector pose matrix is a 4x4 homogeneous transformation matrix. The spatial distance weight of the bidirectional connection edge is normalized between 0 and 1, and the time sequence correlation weight is set to 1 when the time interval is less than 0.1 seconds and 0 when the time interval is greater than 0.5 seconds.

[0030] In this embodiment, the environmental mutation increment is represented by a spatio-temporal graph structure, and the graph nodes are divided into two categories: environmental mutation region nodes and robot state nodes. The environmental mutation increment represents the region where the environmental topology has changed significantly and its associated robot state, such as a 30cm high obstacle suddenly appearing on the ground, a slope angle changing more than 15 degrees within 5 seconds, etc.

[0031] The beneficial effects of the above technical solutions are: the spatio-temporal convolutional neural network is used to realize the rapid calculation of the environmental change degree, which improves the real-time perception; the graph structure construction algorithm is used to establish the association between the environment and the state, which enhances the causal reasoning ability; the bidirectional connection edge design supports the bidirectional propagation of information, which improves the accuracy of state estimation.

[0032] In some embodiments of the present application, when calculating the environmental change degree of each spatial grid based on the multi-modal sensor data and the spatio-temporal convolutional neural network, it includes: obtaining a first environmental parameter distribution of the robot in the last learning period, the first environmental parameter distribution being obtained by estimating the probability density of the environmental parameters collected in the last learning period through a Gaussian mixture model; based on the multi-modal sensor data, obtaining a second environmental parameter distribution of the robot in the current learning period; The first environment parameter distribution and the second environment parameter distribution are normalized, a KL divergence value and a JS divergence value are respectively calculated, and an average value of the KL divergence value and the JS divergence value is determined as the environment change degree.

[0033] In this embodiment, the Gaussian mixture model includes 5 Gaussian components, each component includes a mean vector, a covariance matrix, and a weight coefficient. The environment parameters include terrain roughness, obstacle density, ground friction coefficient, light intensity, and noise level, with a total of 5 dimensions. The probability density estimation is iteratively optimized using the expectation maximization algorithm, with the number of iterations set to 100 and the convergence threshold set to 0.001.

[0034] In this embodiment, the second environment parameter distribution is estimated by the same Gaussian mixture model structure as the first environment parameter distribution.

[0035] In this embodiment, the Gaussian mixture model structure of the first environment parameter distribution and the second environment parameter distribution is the same, but the model parameters are different. The environment parameter acquisition obtains the terrain point cloud data through the laser radar carried by the robot, counts the number and distribution of obstacles through the visual recognition algorithm, calculates the friction coefficient through the ground reaction force measured by the force sensor, measures the light intensity through the light sensor, and measures the environmental noise through the microphone array. The modeling data of the first environment parameter distribution comes from the historical data of the last 30 seconds of the previous learning period, and the modeling data of the second environment parameter distribution comes from the real-time data of the last 5 seconds of the current learning period. The calculation of the KL divergence and the JS divergence is performed in parallel on the embedded GPU of the robot, and the calculation delay is less than 10 milliseconds.

[0036] In this embodiment, the KL divergence calculation uses Monte Carlo sampling approximation with 1000 sampling points. The JS divergence calculation is based on the KL divergence value through symmetrization processing. The normalization method uses logarithmic normalization, that is, the natural logarithm of the KL divergence value and the JS divergence value is taken, and then divided by a preset maximum logarithm value to ensure that the result falls within the interval of 0 to 1, and the maximum logarithm value is set to 10.

[0037] The beneficial effects of the above technical solutions are: the Gaussian mixture model is used to probabilistically represent the environment parameters, improving the completeness of the environment description; the divergence value is calculated by comparing the double distribution, quantifying the statistical significance of the environment change; the average value of the KL and JS divergence is fused, considering the change direction and symmetry evaluation, making the environment change degree calculation more robust.

[0038] In some embodiments of the present application, when the main policy network is obtained according to the first control parameter and the second control parameter, it includes: Based on the first control parameter and the second control parameter, a first strategy value corresponding to the first control parameter and a second strategy value corresponding to the second control parameter are obtained; The root mean square error of the first strategy value and the second strategy value is calculated, and the absolute value of the ratio of the root mean square error to the first strategy value is taken as a value difference degree; Based on an adaptive learning rate adjustment formula, an adjustment amplitude of an initial strategy network is obtained, and the main strategy network is determined based on the adjustment amplitude.

[0039] In this embodiment, the strategy value is output by a value evaluator subnetwork of the strategy network. The value evaluator is a double-layer fully connected network, the dimension of the hidden layer is 128, and the output layer is a single-node linear output. The value evaluation adopts a time difference algorithm, the discount factor γ is set to 0.95, and the TD error is calculated by the difference between the current reward and the future value estimation. The root mean square error is obtained by calculating the square mean of the difference between the first strategy value and the second strategy value, and then taking the square root. This calculation is performed in a rolling time window, and the window length is set to 20 time steps.

[0040] In this embodiment, the value difference degree is a dimensionless ratio, and the first threshold is set to 0.3. When the value difference degree exceeds the threshold, it is determined that the environment has changed significantly. The adaptive learning rate adjustment formula is a piecewise function: when the value difference degree is less than 0.1, the adjustment amplitude is 0; when the value difference degree is between 0.1 and 0.3, the adjustment amplitude increases linearly; when the value difference degree is greater than 0.3, the adjustment amplitude reaches a saturation value, and the saturation value is set to 5 times the baseline learning rate 0.001.

[0041] In this embodiment, the first strategy value is calculated by an old strategy network corresponding to the first control parameter, and the second strategy value is calculated by a new strategy network corresponding to the second control parameter. The calculation of the root mean square error uses an exponential moving average method, and the forgetting factor is set to 0.9. A small constant is added to the denominator of the value difference degree to prevent division by zero error. The adjustment amplitude is updated every 5 learning periods, and a sliding smoothing process is used for updating, and the smoothing coefficient is set to 0.8.

[0042] In this embodiment, the main strategy network is a three-layer fully connected neural network, the input dimension is a 128-dimensional environment feature vector, and the output dimension is a 12-dimensional basic action vector, which corresponds to the desired angles of the hip joints and knee joints of the robot limbs.

[0043] The beneficial effects of the above technical solutions are: the influence degree of environmental changes on the performance of the strategy is objectively quantified by calculating the value difference degree through double-strategy value comparison; the updating strength of the strategy network is dynamically controlled through the self-adaptive learning rate adjustment formula to avoid shaking caused by too fast updating or lag caused by too slow updating; the adaptive ability of the strategy network to environmental changes is improved by considering stability and agility through the design of a segmented function.

[0044] S130: value evaluation is performed on the robot state based on the base action vector of the robot output by the main strategy network, and a value evaluation quantity is obtained; In this embodiment, the base action vector is the original control instruction generated by the main strategy network, which is the initial value before local correction.

[0045] In some embodiments of the present application, when the value evaluation is performed on the robot state to obtain the value evaluation quantity, it includes: acquiring state perception data of the robot; acquiring historical state data of the robot in the last learning period, the historical state data being joint motion trajectories of the robot at historical time points; calculating a state transition probability distribution based on the historical state data and the state perception data; constructing a value evaluation function based on the state transition probability distribution, and iteratively solving the value evaluation function to obtain the value evaluation quantity.

[0046] In this embodiment, the state perception data includes 12 joint angles, 12 joint angular velocities, three-axis centroid positions, three-axis centroid velocities, and three-dimensional contact forces of four points on the foot bottom, which are 42-dimensional vectors. The historical state data is cached in an experience replay pool, and the cache capacity is set to 10,000 trajectories, each with a length of 50 time steps.

[0047] In this embodiment, the state transition probability distribution is approximated by constructing a state transition matrix, the matrix dimension is 100x100, the state space is discretized by a K-means clustering algorithm, the number of cluster centers is set to 100, and the Euclidean distance is used as the cluster distance metric.

[0048] In this embodiment, the value evaluation function adopts the form of Bellman equation, and the Reward function is composed of four parts: forward speed reward, energy consumption penalty, attitude stability reward, and collision avoidance penalty, and the weights of the four parts are 0.3, 0.2, 0.3, and 0.2 respectively. The value iteration algorithm is used for iterative solution of the value evaluation function, the number of iterations is set to 50, and the convergence threshold is set to 0.01.

[0049] In this embodiment, the value evaluation function is: V(s) = r(s) + γ x Σ[P(s'|s, a) x V(s')], where V(s) is the value evaluation function, γ is the discount factor, r(s) is the immediate reward (the weighted sum of the energy consumption penalty, the stability reward, the task progress reward and the collision avoidance penalty), P(s'|s, a) is the state transition probability distribution, V(s') is the value of the next state, where the values of multiple next states can be determined, and then the product values of each next state value and the state transition probability distribution are determined, and then the total value is determined.

[0050] The beneficial effects of the above technical solutions are: by fusing historical state data and current perception data, the time sequence consistency of state evaluation is improved; by modeling the environmental dynamic characteristics through the state transition probability distribution, the prediction accuracy of value evaluation is enhanced; by iterative solving through the value evaluation function, offline pre-computation is realized, and online computation load is reduced.

[0051] S140: updating and calculating the base action vector based on the value evaluation quantity to obtain a compensation action vector; In some embodiments of the present application, when the base action vector is updated and calculated based on the value evaluation quantity to obtain a compensation action vector, it includes: Based on the value evaluation quantity, calculate the action advantage degree, wherein the action advantage degree is the deviation degree of the base action vector from the average action value; Obtain the historical action vector of the robot in the last learning cycle; Based on the historical action vector and the action advantage degree, construct an action update gradient; Clip the action update gradient to obtain a gradient clipping value, and update the base action vector based on the gradient clipping value to obtain the compensation action vector.

[0052] In this embodiment, the action advantage degree is obtained by calculating the difference between the value evaluation quantity obtained by executing the base action vector in the current state and the average value evaluation quantity of executing all possible actions, and the difference value is calculated under the assumption of Gaussian distribution in the action space. The mean of the Gaussian distribution is set as the base action vector, and the covariance matrix is initialized as the unit matrix.

[0053] In this embodiment, the historical action vectors are stored in the action cache queue, the queue length is set to 100, and the cached actions are the last 100 basic action vectors and their corresponding value evaluation quantities. The action update gradient is constructed by the policy gradient theorem, the gradient direction points to the direction of increasing action advantage, and the gradient amplitude is jointly determined by the action advantage and the similarity of the historical action vector. The similarity is calculated by the cosine similarity, and when the similarity is higher than 0.8, the gradient amplitude is attenuated to 0.5 times. The gradient clipping threshold is set to 0.5, and when the gradient norm exceeds the threshold, it is scaled to the threshold size in proportion. The update of the compensation action vector adopts the momentum method, the momentum coefficient is set to 0.9, the update step is dynamically related to the learning rate, and the learning rate reference value is set to 0.01.

[0054] In this embodiment, the gradient clipping adopts a global clipping method, that is, the L2 norm of all action dimension gradients is calculated, and when it exceeds the threshold, it is uniformly scaled.

[0055] The beneficial effects of the above technical solutions are: the action advantage quantifies the improvement potential of the basic action vector, guiding the compensation direction; the historical action vector information avoids repeated exploration of inefficient actions, improving the utilization rate of learning samples; the gradient clipping prevents the update step from being too large, causing policy collapse, ensuring learning stability; the compensation action vector realizes local fine correction, enhancing the adaptability of the policy to environmental details.

[0056] In some embodiments of the present application, when the basic action vector is updated based on the gradient clipping value to obtain the compensation action vector, it includes: When the action advantage is greater than a first advantage threshold and less than a first gradient threshold, a first compensation action vector is obtained, which positively reinforces and corrects the basic action vector; When the action advantage is less than or equal to a second advantage threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensation action vector is obtained, which conservatively constrains and corrects the basic action vector; wherein the first advantage threshold is greater than the second advantage threshold.

[0057] In this embodiment, the first advantage threshold is set to 0.2, the second advantage threshold is set to 0.05, and the first gradient threshold is set to 0.5. When the action advantage is greater than 0.2 and the gradient clipping value is less than 0.5, the positive reinforcement correction mode is enabled. The positive reinforcement correction is implemented by adding the gradient clipping value multiplied by the amplification factor 1.5 to the basic action vector, and the addition method is vector addition. The amplification factor is dynamically adjusted according to the size of the action advantage, and the larger the advantage, the larger the amplification factor, and the maximum does not exceed 2. When the action advantage is less than or equal to 0.05, or when the gradient clipping value is greater than or equal to 0.5, it is determined that there is a risk of updating the oscillation, and the conservative constraint correction mode is enabled. The conservative constraint correction is implemented by adding the gradient clipping value multiplied by the attenuation factor 0.5 to the basic action vector, and the attenuation factor is not less than 0.2 to ensure that the correction direction is correct but the amplitude is limited. The difference between the first advantage threshold and the second advantage threshold is set to 0.15 to form a buffer interval to avoid frequent switching of the correction mode causing jitter.

[0058] S150: Vector superposition fusion is performed on the basic action vector and the compensation action vector to obtain a final control vector, and the final control vector is taken as a cerebellum learning action instruction of the robot.

[0059] In some embodiments of the present application, when the basic action vector and the compensation action vector are vector superposition fused to obtain a final control vector, the following steps are included: The basic action vector is subjected to length normalization processing to obtain a unit basic vector; The compensation action vector is subjected to amplitude scaling processing to obtain a scaled compensation vector; The vector inner product of the unit basic vector and the scaled compensation vector is calculated to obtain a fusion weight value; Based on the fusion weight value, the unit basic vector and the scaled compensation vector are weighted and summed to obtain the final control vector.

[0060] In this embodiment, the length normalization of the basic action vector adopts L2 norm normalization, that is, the vector is divided by its Euclidean length, and if the vector length is zero, it is kept as a zero vector. The amplitude scaling of the compensation action vector is realized by a scaling coefficient, which is set to 0.3. The coefficient is dynamically adjusted according to the confidence of the compensation action vector, and the confidence is obtained by calculating the average cosine similarity between the compensation action vector and the historical successful compensation vector. When the similarity is higher than 0.8, the scaling coefficient is increased to 0.5, and when the similarity is lower than 0.3, the scaling coefficient is reduced to 0.1. The vector inner product of the unit basic vector and the scaled compensation vector is calculated in the 12-dimensional action space, and the inner product result ranges from -1 to 1. The fusion weight value in the interval of 0 to 1 is converted by linear mapping, and the mapping formula is (inner product value + 1) / 2. The weighted sum adopts dynamic weight distribution. When the fusion weight value is greater than 0.7, the weight of the unit basic vector is 0.3 and the weight of the scaled compensation vector is 0.7; when the fusion weight value is less than 0.3, the weight of the unit basic vector is 0.8 and the weight of the scaled compensation vector is 0.2; and when the fusion weight value is between 0.3 and 0.7, the weight is linearly interpolated.

[0061] The beneficial effects of the above technical solution are: the length normalization retains the global planning information of the basic action, preventing the compensation action from distorting the overall motion direction; the amplitude scaling controls the correction strength of the compensation action, avoiding local over-adjustment; the vector inner product dynamically evaluates the compensation reliability, realizing adaptive weight distribution; the weighted sum integrates the global planning and local correction, generating the final control vector with forward-looking and adaptive characteristics, improving the overall performance and robustness of robot motion control.

[0062] In order to further illustrate the technical idea of the present application, the technical solution of the present application will be described in conjunction with specific application scenarios.

[0063] Correspondingly, as shown in Figure 2 The present application also provides a robot cerebellum reinforcement learning system based on adaptive environmental changes, which comprises: A data acquisition module is configured to acquire multi-modal sensing data during the operation of the robot in a current learning period, wherein the multi-modal sensing data is the state response of the robot in the environmental change, and the multi-modal sensing data at least includes visual depth information, inertial measurement unit data, contact force feedback data, and environmental temperature and humidity data. A network determination module is configured to determine an environmental mutation increment and an environmental feature change amount based on the multi-modal sensing data, and obtain a main policy network according to the environmental mutation increment and the environmental feature change amount. A value evaluation module is configured to output a basic action vector of the robot based on the main policy network, and evaluate the value of the robot state to obtain a value evaluation amount. An action updating module is configured to perform an updating calculation on the base action vector based on the value evaluation quantity, to obtain a compensation action vector; A cerebellum learning module is configured to perform vector superposition fusion on the base action vector and the compensation action vector, to obtain a final control vector, and to take the final control vector as a cerebellum learning action instruction of the robot.

[0064] In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0065] Although the present application has been described with reference to the embodiments above, various modifications can be made to the present application and equivalents thereof without departing from the scope of the present application. In particular, the features of the disclosed embodiments can be used in any combination without departing from the scope of the present application, and the combinations of these features are not described in the present specification only for the purpose of saving space and resources.

[0066] It should be understood by those of ordinary skill in the art that the above are only preferred embodiments of the present application, and are not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art can still modify the technical solutions recorded in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A robot cerebellum reinforcement learning method based on adaptive environmental changes, characterized in that, The method comprises the following steps: In the current learning cycle, collect multi-modal sensor data during the operation of the robot, wherein the multi-modal sensor data is the state response of the robot in the environment change, and the multi-modal sensor data at least includes visual depth information, inertial measurement unit data, contact force feedback data and environment temperature and humidity data; Based on the multi-modal sensor data, determine the environment mutation increment and the environment feature change amount, and obtain the main strategy network according to the environment mutation increment and the environment feature change amount; Output the basic action vector of the robot based on the main strategy network, perform value evaluation on the state of the robot, and obtain a value evaluation amount; Update and calculate the basic action vector based on the value evaluation amount to obtain a compensation action vector; Perform vector superposition and fusion on the basic action vector and the compensation action vector to obtain a final control vector, and take the final control vector as the cerebellar learning action instruction of the robot.

2. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 1, characterized in that, When determining the environment mutation increment and the environment feature change amount based on the multi-modal sensor data, and obtaining the main strategy network according to the environment mutation increment and the environment feature change amount, the method comprises the following steps: Determine the environment mutation increment based on the multi-modal sensor data; Adopt a joint encoding mechanism of hierarchical perception and attention to extract features of the environment mutation increment to obtain the environment feature change amount; Fuse the environment feature change amount and a first control parameter to obtain a second control parameter, wherein the first control parameter is a control reference value of a full state in a previous learning cycle, and the second control parameter is a control reference value of a full state in the current learning cycle; Obtain the main strategy network according to the first control parameter and the second control parameter.

3. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 2, characterized in that, When determining the environment mutation increment based on the multi-modal sensor data, the method comprises the following steps: Calculate the environment change degree of each spatial grid based on the multi-modal sensor data and a space-time convolutional neural network, wherein the space-time convolutional neural network comprises a cascaded structure of a time dimension one-dimensional convolutional layer and a space dimension two-dimensional convolutional layer; Determine a mutation region and a state node associated with the mutation region based on the environment change degree and the multi-modal sensor data, wherein the mutation region is a continuous spatial region with an environment change degree greater than a first threshold and a connectivity index greater than a second threshold, and the state node is a joint angle vector, a center of mass position coordinate and an end effector pose matrix of the robot; Obtain the environment mutation increment by a graph structure construction algorithm based on the mutation region and the state node associated with the mutation region, wherein the graph structure construction algorithm takes the mutation region as a parent node, takes the state node as a child node and establishes a bidirectional connection edge, and the bidirectional connection edge includes a spatial distance weight and a time sequence related weight.

4. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 3, characterized in that, When calculating the environment change degree of each spatial grid based on the multi-modal sensor data and the space-time convolutional neural network, the method comprises the following steps: Obtain a first environment parameter distribution of the robot in the previous learning cycle, wherein the first environment parameter distribution is obtained by performing probability density estimation on the environment parameters collected in the previous learning cycle through a Gaussian mixture model; obtain a second environment parameter distribution of the robot in the current learning period based on the multi-modal sensor data; normalize the first environment parameter distribution and the second environment parameter distribution, calculate a KL divergence value and a JS divergence value respectively, and determine an average value of the KL divergence value and the JS divergence value as the environment change degree.

5. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 3, characterized in that, In the process of obtaining the main policy network based on the first control parameter and the second control parameter, the following steps are included: obtain a first policy value corresponding to the first control parameter and a second policy value corresponding to the second control parameter based on the first control parameter and the second control parameter; calculate a root mean square error of the first policy value and the second policy value, and calculate an absolute value of a ratio of the root mean square error to the first policy value as a value difference degree; obtain an adjustment amplitude of an initial policy network based on an adaptive learning rate adjustment formula, and determine the main policy network based on the adjustment amplitude.

6. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 1, characterized in that, In the process of obtaining a value evaluation quantity by evaluating the state of the robot, the following steps are included: obtain state perception data of the robot; obtain historical state data of the robot in the last learning period, the historical state data being a joint motion trajectory of the robot at a historical time; calculate a state transition probability distribution based on the historical state data and the state perception data; construct a value evaluation function based on the state transition probability distribution, and iteratively solve the value evaluation function to obtain the value evaluation quantity.

7. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 1, characterized in that, In the process of obtaining a compensation action vector by updating and calculating the basic action vector based on the value evaluation quantity, the following steps are included: calculate an action advantage degree based on the value evaluation quantity, wherein the action advantage degree is a deviation degree of the basic action vector from an average action value; obtain a historical action vector of the robot in the last learning period; construct an action update gradient based on the historical action vector and the action advantage degree; perform clipping processing on the action update gradient to obtain a gradient clipping value, and update the basic action vector based on the gradient clipping value to obtain the compensation action vector.

8. The robot cerebellum reinforcement learning method based on adaptive environment changes according to claim 7, characterized in that, In the process of obtaining the compensation action vector by updating the basic action vector based on the gradient clipping value, the following steps are included: when the action advantage degree is greater than a first advantage threshold and less than a first gradient threshold, a first compensation action vector is obtained, which positively reinforces and corrects the basic action vector; when the action advantage degree is less than or equal to a second advantage threshold, or when the gradient clipping value is greater than or equal to the first gradient threshold, a second compensation action vector is obtained, which conservatively constrains and corrects the basic action vector; wherein the first advantage threshold is greater than the second advantage threshold. 9.The robot cerebellum reinforcement learning method based on adaptive environment change according to claim 1, wherein, In the process of obtaining a final control vector by vector superposition and fusion of the basic action vector and the compensation action vector, the following steps are included: perform length normalization processing on the basic action vector to obtain a unit basic vector; perform amplitude scaling processing on the compensation action vector to obtain a scaled compensation vector; calculating a vector inner product of the unit basis vector and the scaling compensation vector to obtain a fusion weight value; performing weighted summation on the unit basis vector and the scaling compensation vector based on the fusion weight value to obtain the final control vector.

10. A robot cerebellar reinforcement learning system based on adaptive environmental changes, applied to the robot cerebellar reinforcement learning method based on adaptive environmental changes according to any one of claims 1-9, characterized in that, Comprise: a data acquisition module configured to acquire multi-modal sensing data of a robot during operation in a current learning period, the multi-modal sensing data being a state response of the robot in an environment change, and the multi-modal sensing data comprising at least visual depth information, inertial measurement unit data, contact force feedback data, and environment temperature and humidity data; a network determination module configured to determine an environment mutation increment and an environment feature change amount based on the multi-modal sensing data, and obtain a main policy network according to the environment mutation increment and the environment feature change amount; a value evaluation module configured to output a basic action vector of the robot based on the main policy network, and evaluate a value of a state of the robot to obtain a value evaluation amount; an action update module configured to update and calculate the basic action vector based on the value evaluation amount to obtain a compensation action vector; a cerebellum learning module configured to perform vector superposition fusion on the basic action vector and the compensation action vector to obtain a final control vector, and use the final control vector as a cerebellum learning action instruction of the robot.

Images