Robot reinforcement learning motion control method based on data dimensionality reduction technology

Abstract

The present invention relates to the field of safety enhancement technology and discloses a robot reinforcement learning motion control method based on data dimensionality reduction technology. The method comprises the following steps: sampling robot motion control trajectory samples with a high-dimensional state space at an unbalanced position, and performing dimensionality reduction processing on the state space sample data; estimating the safety probability of the robot state in a low-dimensional space by a grid method, and training the motion strategy of the high-dimensional space robot by a reinforcement learning method. The present invention can be directly applied to real physical systems without the need for simulation modeling training, thus avoiding the inherent gap problem between simulation and real machines and reducing modeling costs.

Description

Technical Field

[0001] The present invention relates to the field of security enhancement technology, and in particular to a robot reinforcement learning motion control method based on data dimensionality reduction technology. Background Art

[0002] While reinforcement learning and deep learning methods have achieved remarkable success in robotic system control in recent years, existing technologies largely rely on simulation, limiting their application in the real world. This is due in part to the inherent trial-and-error nature of reinforcement learning methods, which can lead robots to explore dangerous intermediate strategies. Furthermore, due to the difficulty in measuring the differences between physical simulation engines and the real world, well-trained strategies in simulation may not perform well when transferred to the real world. These two situations inevitably raise the practical challenge of safety assurance. Because safety assurance can reduce wear and tear on the robot itself, physical damage to the surrounding environment, and reduce costs, safe reinforcement learning techniques are essential to ensure the safety of robot motion control.

[0003] Existing safety reinforcement learning methods implement safety constraints by adjusting the reinforcement learning algorithm, adding additional risk terms, or restricting the exploration process. However, this approach requires a complete prior understanding of the robot model and environment. Furthermore, current techniques for setting safe regions include combining control theory to design Lyapunov barrier functions to obtain precise safe regions. However, this approach requires a known and accurate model and is less applicable to robotic systems with high-dimensional state spaces, strong nonlinearities, inaccurate modeling, and uncertain dynamics. Although models can be learned through some probabilistic modeling techniques, safety performance clearly depends on the accuracy of the learned model. Furthermore, modeling safety-critical control as a differential game is also a common technique. However, solving the Hamilton-Jacobi-Bellman equations (HJB equations) in high-dimensional state spaces requires high computing power and suffers from the curse of dimensionality. Therefore, directly calculating and solving the safe regions of high-dimensional robotic systems is difficult.

[0004] Currently, physics-inspired dimensionality reduction techniques have been proposed for robotic systems with high-dimensional state spaces. These techniques, to some extent, address the safety issues inherent in reinforcement learning for these systems. However, physics-inspired dimensionality reduction requires a high level of prior knowledge about the robot model and the task, is not universally applicable, and lacks a unified dimensionality reduction standard. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a robot reinforcement learning motion control method based on data dimensionality reduction technology, which is used for reducing the dimensionality of data driven in the safety area of ​​complex high-dimensional state space during the robot reinforcement learning motion control process, and based on this, achieving safety protection in the learning process. The present invention can be directly applied to real physical systems without the need for simulation modeling training, avoiding the inherent gap problem between simulation and real machines, and reducing modeling costs. The present invention can handle high-dimensional robot systems without prior information, and through real data sampling, it reduces the dimensionality of the safety area without obtaining low-dimensional dynamic equations. It is a model-free, data-based method that significantly improves the applicable scenarios and practicality of safety reinforcement learning. Two control strategies are used, including a corrective control strategy and a reinforcement learning strategy, to ensure that the safety probability of the robot's state is always higher than the safety threshold, thereby ensuring the safety of the reinforcement learning process in a probabilistic sense.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A robot reinforcement learning motion control method based on data dimensionality reduction technology includes the following steps:

[0008] Step A: Collect the initial state x of the robot kt non-equilibrium points init , get the robot's initial motion trajectory sample set D with high-dimensional space init ={D1,D2,…,D i ,…,D kt}, where D i is the motion trajectory sample of the i-th robot, D i ={x i init ,Φ(t,x i init ),l(x i init )}, kt is D init The size of Φ(t,x i init ) represents the correction controller K(x) used to adjust the robot’s initial state x i init The robot motion trajectory after control, l(x i init ) is a security label;

[0009] Step B: Use the dynamic time warping algorithm to merge the data set D t The distance measure ω between the motion trajectories in ij Calculation is performed, and no feedback motion trajectory sample set D is obtained feedback Merge data set D t There is only D init; distance metric ω ij The calculation method of ω is: ij =ω(x i ,x j )=DTW(Φ(x i ),Φ(x j )), and normalize the distance metric to obtain the distance matrix Ω; x i 、x j Represent the merged data set D t The initial state in D is the initial state of the robot's i-th and j-th trajectories. init x in the middle i init 、x j init , in D feedback x in the middle i feedback 、x j feedback DTW(·) represents the dynamic time warping algorithm;

[0010] Step C: By using the distance matrix Ω and the metric learning method, the data set D is merged while maintaining the distance metric relationship between the high-dimensional space and the low-dimensional space. t The initial state x in i That is, the state in the original high-dimensional state space Perform dimensionality reduction and map to a low-dimensional state Get the high-dimensional state point set X={x 1 ,x 2 ,…} one-to-one corresponding low-dimensional state point set n is the dimension of high-dimensional space, m is the dimension of low-dimensional space;

[0011] Step D: Use the polynomial regression method to train the regression learner to obtain the mapping relationship f between high-dimensional space and low-dimensional space l : x s =f l (x), where the superscript l represents the number of times the mapping is constructed; the regression error ε follows the empirical distribution, and the confidence interval of ε is solved in turn to obtain the predicted interval estimate Where 1-α represents the confidence level; represents the point estimate obtained from the regression, Unified as x s , x 1 ,x 2 ,…are uniformly denoted as x;

[0012] Step E: The safety confidence is obtained based on the safety probability estimated in the current state interval, which can predict whether the current state of the robot is safe and obtain the estimated safety area D. If the current state is judged to be safe, the robot's reinforcement learning motion control strategy π(x) is trained; if the current state is judged to be unsafe, the correction controller K(x) is enabled to control the robot state back to the safe area, and the trajectory information is collected to obtain the feedback motion trajectory sample set D. feedback ={D1,D2,…,D i ,…,D kf}, where kf is the feedback motion trajectory sample set D feedback Size, D i ={x i feedback ,Φ(t,x i feedback ),l(x i feedback )},x i feedback is the current state of the robot in the real training environment, Φ(t,x i feedback ) represents the correction controller K(x) used to control x i feedback The robot motion trajectory after control, l(x i feedback ) is a security label;

[0013] Step F: Determine the feedback motion trajectory sample set D feedback When kf=k*ku, the motion trajectory sample set D will be fed back feedback and the initial motion trajectory sample set D init Merge to get the merged data set D t ={D init ,D feedback}={D1,D2,…,D i ,…,D kt+kf}, D i ={x i ,Φ(t,x i ),l(x i )}, where x i is the initial state, in D init x in the middle i init , in D feedback x in the middle i feedback , merge the data set D t As the data source for calculating the motion trajectory distance matrix in step B, where k represents the multiple and ku is a specified set size;

[0014] Repeat steps B to F to obtain a robot reinforcement learning motion control strategy π(x) with safety assurance.

[0015] Furthermore, step A specifically includes:

[0016] According to the initial asymptotically stable equilibrium point x = 0 under the known correction controller K(x) and the attraction domain with x = 0 Define the safe region D based on the positive invariant set: The purpose of collecting samples of each motion trajectory is to estimate the robot's safe area; where x0 is the current state of the robot, represents the original high-dimensional state space, Φ(t;x0) represents the motion trajectory of the robot starting from the current state x0 under the given correction controller K(x); specifically, it includes:

[0017] Collect the robot's initial non-equilibrium point x i init Information, the safe area D must include the initial equilibrium point x = 0, that is, x i init ∈X\{0}, judge whether the current state of the robot is safe by the end point of the trajectory; use l(x i init ) represents the safety status label, l(x i init )=1 means “safe”, l(x i init )=0 means “unsafe”;

[0018] For the motion trajectory sample set D init Perform data enhancement, including:

[0019] The states of all points on the same motion trajectory have the same safety label. The states of the points on the motion trajectory are uniformly sampled to obtain a set of enhanced motion trajectory sample sets D e ={D1,D2,…,D j ,…}, where the jth enhanced motion trajectory sample is D j ={x j e ,Φ(t,x j e ),l(x i imit )},x j e is the state of the sampling point; Φ(t,x j e ) represents the original motion trajectory Φ(t,x i init ) intercepted from state xj e The subsequent trajectory, safety label and original trajectory Φ(t,x i init )same.

[0020] Furthermore, when calculating the distance matrix Ω in step B:

[0021]

[0022] Among them, ω max =maxω ij , which means normalizing the distance matrix; δ is a constant.

[0023] Furthermore, in step D, when the polynomial regression method is used to train the regression learner to obtain the mapping relationship between the high-dimensional space and the low-dimensional space, the t-SNE dimensionality reduction method is used to reduce the dimension multiple times and construct the regression mapping until the set number of dimensionality reductions is reached or the combined data set D is obtained. t The number of samples in kt+kf exceeds the threshold.

[0024] Furthermore, in step E, the safety confidence is obtained based on the safety probability estimated in the interval of the current state, and the safety of the robot is predicted to be safe, and the estimated safety area D is obtained:

[0025] Use the grid to divide the low-dimensional space and determine the safety probability of the current high-dimensional state x Among them, C p C represents the number of points in the high-dimensional space where the high-dimensional state x is mapped to the low-dimensional space and falls on the grid with the security label “safe”. n Represents the number of points in the high-dimensional space where the high-dimensional state x is mapped to the low-dimensional space and falls on the grid with the security label “unsafe”; when P(x∈D)>p t When P(x∈D)≤p t When the current high-dimensional state x is mapped to the low-dimensional space, the grid where the point estimate is located is considered unsafe; t is the safety threshold; the safety probability is calculated for all grids within the range of the predicted interval estimate, and the ratio of safe grids to all grids within the predicted interval estimate is used as the safety confidence. When the safety confidence reaches the confidence threshold p s When , the interval estimation of the current high-dimensional state x mapping is determined to be safe; finally, the estimated safe area D is obtained.

[0026] Compared with the prior art, the beneficial technical effects of the present invention are:

[0027] 1. This paper proposes a robot safety reinforcement learning framework that is directly applied to the real physical world. It avoids the cost of physical simulation modeling and the difference between the model and the real machine. There is no need to train in a simulation environment and then migrate to the real machine. The robot can be trained directly in the real physical environment without any modeling errors and without the need to migrate the motion control strategy. At the same time, the safety of the training process is guaranteed in a probabilistic sense.

[0028] 2. This paper proposes a data-based safe zone dimensionality reduction technology, which is suitable for robot systems with high-dimensional complex state spaces. It does not require prior model information and performs dimensionality reduction processing on the high-dimensional state space data used to train robots in the real physical world, reducing the complexity of the problem and the amount of computation, which helps to implement reinforcement learning in the field of high-dimensional robot motion control.

[0029] 3. The present invention proposes a low-dimensional security probability representation method that takes into account dimensionality reduction errors, performs error analysis on the low-dimensional security probability, considers the mapping error between high-dimensional space and low-dimensional space, and the probability error caused by information loss in the process of low-dimensional space representing high-dimensional space, making the security probability estimation more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the flow of the robot reinforcement learning motion control method based on data dimensionality reduction technology provided by the present invention;

[0031] Figure 2 This is an algorithm structure diagram of the robot reinforcement learning motion control method based on data dimensionality reduction technology of the present invention. DETAILED DESCRIPTION

[0032] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] This paper uses data-driven dimensionality reduction technology to extend the safety reinforcement learning framework to model-free high-dimensional robotic systems and provides error analysis. In this embodiment of the present invention, a robot dog is selected as the high-dimensional robotic system, and the safety region is defined based on the region of attraction (ROA): Where x0 is the robot dog's current state, and Φ(t; x0) represents the robot dog's trajectory starting from the current state x0 under a given corrective controller K(x). Given the corrective controller K(x), the state is stabilized, and a PID controller is used for stability design. A safe region D is defined based on the initial asymptotically stable equilibrium point x = 0 under the known corrective controller K(x) and the positive invariant set of the attraction domain based on x = 0. When the robot dog's current state is within the safe region D, it is considered safe, and the reinforcement learning control strategy π(x) continues to be trained. Otherwise, the current state is considered dangerous, and the corrective controller K(x) is used to control the current state back to the safe region. The safe reinforcement learning framework is as follows:

[0034]

[0035] Here, u represents the control strategy, p represents the safety probability threshold, and P(x∈D) represents the safety probability of the current high-dimensional state x. High-dimensional safety regions cannot be directly solved due to the curse of dimensionality. Therefore, the safety region is unknown and can only be estimated. Here, the safety region of the high-dimensional robot dog is described using safety probabilities.

[0036] For the robot dog system with high-dimensional state space, due to the dimensionality curse, in order to obtain the safety probability of the current state, it is necessary to reduce the dimensionality of the state space and use a low-dimensional state space. Representing a high-dimensional state space To this end, the present invention proposes to use data-driven dimensionality reduction technology to perform unified dimensionality reduction processing on the model-free high-dimensional robot dog system.

[0037] Now define the robot dog's initial motion trajectory sample set D init ={D1,D2,…,D kt}, kt is the initial motion trajectory sample set D init The size of the i-th initial motion trajectory sample D i ={x i init ,Φ(t,x i init ),l(x i init )},x init represents the initial state of the non-equilibrium point, Φ(t,x i init ) represents the correction controller K(x) used to control x i init After the robot dog is controlled, according to the definition of the safe area, the non-equilibrium point state inside the safe area will return to the equilibrium point under the action of the correction controller K(x). Therefore, the end state of the trajectory can be used as the feedback state to judge whether the robot dog's current state is safe. i init ) indicates the safety status label. l(x i init )=1 means safe, otherwise l(x i init )=0 means unsafe. Define the robot dog feedback motion trajectory sample set D feedback ={D1,D2,…,D kf}, kf is the feedback motion trajectory sample set D feedback The size of the feedback trajectory sample D i ={xi feedback ,Φ(t,x i feedback ),l(x i feedback )}, the meanings of each item are similar to those in the initial motion trajectory sample set, and will not be repeated here. When kf = k*ku, the motion trajectory sample set D is fed back feedback and the initial motion trajectory sample set D init merged into a data set D t ={D init ,D feedback}={D1,D2,…,D i ,…,D kt+kf}, D i ={x i ,Φ(t,x i ),l(x i )}, where x i is the initial state, in D init x in the middle i init , in D feedback x in the middle i feedback , merge the data set D t It serves as the data source for the subsequent motion trajectory distance matrix calculation, where k represents the multiple and ku is a specified set size.

[0038] The present invention introduces data enhancement technology to expand the original data to obtain a more accurate dimensionality reduction space. Given a correction controller K(x), the states on all trajectories in the data set are uniformly sampled to obtain a set of enhanced motion trajectory sample points D e ={D1,D2,…,D j ,…}, where the jth enhanced motion trajectory sample is D j ={x j e ,Φ(t,x j e ),l(x i init )},x j e is the state of the sampling point; Φ(t,x j e ) represents the original motion trajectory Φ(t,x i init ) intercepted from state x j e The subsequent trajectory, safety label and original trajectory Φ(t,x i init ) to achieve data enhancement.

[0039] The dimensionality reduction technology used in this invention is a data-driven dimensionality reduction method, which essentially aims to achieve the division of the safe area. t Medium x i ,x j , we can consider their trajectory Φ(x i ),Φ(x j ) and express x as the distance between trajectories. i ,x j The distance between them. Such a metric is called the distance metric of the trajectory ω ij :ω ij =ω(x i ,x j )=d(Φ(x i ),Φ(x j )). According to the above, the present invention adopts Dynamic Time Warping (DTW) to merge the data set D t The distance measurement ω is performed on the trajectory of the movement ij =DTW(Φ(x i ),Φ(x j )), and normalize the distance metric at the same time, and get the elements of the distance matrix Ω as follows:

[0040]

[0041] Among them, δ is a constant set according to the safety label to increase the discrimination between the safety and unsafe motion trajectories. max =maxω ij , normalize the distance matrix. If the security labels are different, add δ to the normalized distance.

[0042] The present invention performs data dimensionality reduction processing based on the distance matrix Ω. The Euclidean distance distribution of each point in the low-dimensional space is the same as the motion trajectory distance distribution of each initial point in the high-dimensional space. The present invention uses t-SNE (t-distributed stochastic neighbor embedding) to reduce the dimensionality of the high-dimensional Mapping to low dimension Where n is the dimension of the original state space, m is the dimension of the reduced space, and in this embodiment, m=2 is selected. By using the t-SNE dimensionality reduction method, a point set X in the low-dimensional space corresponding to the high-dimensional original state space set X is obtained. s For each feedback data generated, t-SNE is used to reduce the dimension again according to the above method to obtain a more accurate low-dimensional space, and a regression mapping is constructed to obtain the dimension reduction function f(x): x s =fl (x), where the superscript l represents the number of times the mapping is constructed. Regression learning requires a certain amount of samples to achieve good results. To this end, consider re-performing regression learning every time a certain number of samples are obtained to improve the effect of f, thereby gradually reducing the error. In addition, when the combined data set D is obtained t If the number of samples kt+kf in exceeds the threshold, the obtained low-dimensional space is considered to be determined and the regression mapping is determined. The specific regression method uses polynomial regression, and the polynomial regression order is selected through cross-validation.

[0043] Considering the error factor, assuming that the regression error ε obeys the empirical distribution, sort the regression error by value, determine the confidence level according to the specific task requirements, and obtain the confidence interval of ε based on the quantile, and further estimate the prediction interval of the current state This embodiment adopts α=0.05, that is, the confidence level is 1-α=0.95, where is the point estimate obtained by regression. Thus, we obtain the low-dimensional space and the regression error ε. As mentioned above, it is difficult to calculate the safe region in the high-dimensional original state space. Therefore, we calculate the safe probability in the reduced low-dimensional space to obtain a low-dimensional representation of the high-dimensional safe region. The safe probability in the low-dimensional space is used to represent the safe probability in the high-dimensional space.

[0044] The present invention adopts the idea of ​​frequency instead of probability and uses the grid method to divide the low-dimensional space. Determine the safety probability of the current high-dimensional state x, where C p C represents the count of points in the grid with the security label safe after the points in the high-dimensional space are mapped to the low-dimensional space. n Represents the count of points in the high-dimensional space that fall into the grid with the security label unsafe after being mapped to the low-dimensional space. The specific grid size is selected according to the actual task and experience. In this embodiment, the low-dimensional space can be divided into 10*10 grid spaces. Given a security threshold p t When P(x∈D)>p t When the current high-dimensional state x is mapped to the low-dimensional space, the grid where the estimated point is located is considered safe. Considering the regression error, the low-dimensional point is predicted to fall within the confidence region of 1-α=0.95. The safety probability is calculated for all grids within the range, and the ratio of safe grids to all mapped grids is the safety confidence level. When the safety confidence level reaches p s When , the interval estimation of the current state mapping is considered safe. Therefore, a safety probability estimation with error analysis is given to make the safety estimation more accurate. In addition, due to the small amount of early data, large regression error, and inaccurate low-dimensional space, the safety threshold p t and security confidence p sSet to dynamically change, with a low value in the early stages and a gradually increasing value as the amount of data increases. These two thresholds are used to balance the performance of reinforcement learning exploration and the safety of the robot dog.

[0045] The safety probability of the current state is estimated based on the interval. If it is judged to be safe, the robot dog motion control strategy π(x) is trained according to the aforementioned standard safety reinforcement learning framework. Once it is judged to be unsafe, the correction controller K(x) is enabled to implement safety-critical control of the robot dog state, collect trajectory information, and obtain the feedback motion trajectory sample set D feedback ={D1,D2,…,D kf Until the reinforcement learning strategy is trained under the safety guarantee mechanism described in the present invention.

[0046] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0047] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A robot reinforcement learning motion control method based on data dimensionality reduction technology, comprising the following steps: Step A: Acquisition Robot Non-equilibrium initial state , get the robot's initial motion trajectory sample set with high-dimensional space ,in is the motion trajectory sample of the i-th robot, , for The size of Indicates the use of correction controller For a randomly given robot initial state The robot's motion trajectory after control, It is a safety label; Step B: Use the dynamic time warping algorithm to merge the data set The distance measure between motion trajectories in Calculation is performed, and no feedback motion trajectory sample set is obtained When merging data sets Only exists in ; Distance metric The calculation method is: , and normalize the distance metric to obtain the distance matrix ; 、 Represents the merged data sets The initial state of the i-th and j-th trajectories in Zhongwei 、 ,exist Zhongwei 、 ; represents the dynamic time warping algorithm; Step C: Through the distance matrix And metric learning methods, which merge data sets while maintaining the distance metric relationship between high-dimensional space and low-dimensional space The initial state in That is, the state in the original high-dimensional state space Perform dimensionality reduction and map to a low-dimensional state , and get the high-dimensional state point set One-to-one corresponding low-dimensional state point set ; It is a high-dimensional space dimension, It is a low-dimensional space dimension; Step D: Use the polynomial regression method to train the regression learner to obtain the mapping relationship between high-dimensional space and low-dimensional space : , , where the superscript Represents the number of times the mapping is constructed; regression error Obey the empirical distribution and solve it in sequence , and obtain the predicted interval estimate ,in Indicates the confidence level; represents the point estimate obtained from the regression, Unified record , Unified record ; Step E: Obtain the safety confidence level based on the safety probability estimated in the current state interval, which can predict whether the robot's current state is safe and obtain the estimated safety area D; If the current state is judged to be safe, the robot's reinforcement learning motion control strategy is Conduct training; If the current state is judged to be unsafe, the correction controller is activated , control the robot state back to the safe area, collect trajectory information, and obtain the feedback motion trajectory sample set ,in, is the feedback motion trajectory sample set The size of , is the current state of the robot in the real training environment, Indicates the use of correction controller right The robot's motion trajectory after control, Is a safety label; judge the feedback motion trajectory sample set The size of When the motion trajectory sample set is fed back and the initial motion trajectory sample set Merge to get the merged data set , ,in As the initial state, the data set will be merged As the data source for calculating the distance matrix in step B, Indicates multiples, is a specified set size; Repeat steps B to E to obtain a robot reinforcement learning motion control strategy with safety assurance. .

2. The robot reinforcement learning motion control method based on data dimensionality reduction technology according to claim 1 is characterized in that: Step A specifically includes: According to the known correction controller The initial asymptotically stable equilibrium point under and The attraction domain Define the safe region D based on the positive invariant set: ,in, is the current state of the robot, represents the original high-dimensional state space, Indicates that under a given correction controller The robot's current state The starting motion trajectory; specifically including: Collect the robot's initial non-equilibrium point Information, according to the definition of the safe area, the safe area D must include the initial equilibrium point ,Right now , judge whether the current state of the robot is safe by the end point of the trajectory; use Indicates the security status label, It means "safety", Indicates "unsafe"; For the motion trajectory sample set Perform data enhancement, including: The states of all points on the same motion trajectory have the same safety label. The states of the points on the motion trajectory are uniformly sampled to obtain a set of enhanced motion trajectory samples. , where the jth enhanced motion trajectory sample is , is the state of the sampling point; Indicates from Intercepted from the state Future motion trajectories, safety tags and same.

3. The robot reinforcement learning motion control method based on data dimensionality reduction technology according to claim 1 is characterized in that: Calculate the distance matrix in step B hour: ； in, , which means normalizing the distance matrix; is a constant.

4. The robot reinforcement learning motion control method based on data dimensionality reduction technology according to claim 1 is characterized in that: When the polynomial regression method is used to train the regression learner in step D to obtain the mapping relationship between the high-dimensional space and the low-dimensional space, the t-SNE dimensionality reduction method is used to reduce the dimension multiple times and construct the regression mapping until the set number of dimensionality reductions is reached or the merged data set is obtained. The number of samples in Exceeds the threshold.

5. The robot reinforcement learning motion control method based on data dimensionality reduction technology according to claim 1 is characterized in that: In step E, the safety confidence is obtained based on the safety probability estimated in the current state interval, and the safety of the robot is predicted to be safe, and the estimated safety area D is obtained: Use the grid to divide the low-dimensional space and determine the current high-dimensional state Safety probability ,in, Represents high-dimensional states in high-dimensional space After the points are mapped to the low-dimensional space, the number of points that fall into the grid with the security label "safe" is Represents high-dimensional states in high-dimensional space The number of points that fall into the grid with the security label "unsafe" after the points are mapped to the low-dimensional space; When the current high-dimensional state The grid where the mapped point is estimated to be located is safe; when When the current high-dimensional state The grid where the point estimate is located in the low-dimensional space mapping is not safe; is the safety threshold; the safety probability is calculated for all grids within the predicted interval estimate, and the ratio of safe grids to all grids within the predicted interval estimate is used as the safety confidence. When the safety confidence reaches the confidence threshold When the current high-dimensional state The mapped interval is estimated to be safe; ultimately, an estimated safe region D is obtained.

Images