A robot search and rescue method based on offline reinforcement learning

Abstract

The application discloses a kind of robot search and rescue methods based on offline reinforcement learning, first, the robot control system consisting of server node and robot node is constructed, server node is equipped with first storage module, first model module, first trajectory module, first strategy module, second trajectory module, second strategy module;Robot node upper deployment has detection module, motion module, calculation module.First model module uses D4RL data set to train deep probabilistic neural network and is integrated into environment dynamics model.First trajectory module, second trajectory module respectively uses environment dynamics model to construct O-MDP and P-MDP.Under O-MDP, first strategy module is trained, and second strategy module is trained under P-MDP, and the robot search and rescue strategy is obtained.Robot node executes search and rescue task according to search and rescue strategy.The application can complete search and rescue task under the condition that the number of offline data set is limited and the quality is not high.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control and offline reinforcement learning technology, and particularly relates to a robot search and rescue method based on offline reinforcement learning. Especially, when the strategy contained in the offline data set cannot meet the search and rescue task in a complex scene, the method for improving the generalization of the robot strategy to successfully complete the search and rescue task. BACKGROUND

[0002] In today's society, the application of robot technology is becoming more and more widespread, among which the robot search and rescue technology is an important direction of robot application. The development of robot search and rescue technology can greatly improve the search and rescue efficiency and reduce the risk in the process of personnel search and rescue, so it is concerned. However, in complex scenes such as earthquake disasters, building collapse and other situations, the search and rescue task of the robot becomes extremely difficult. At present, the traditional robot search and rescue method mostly adopts manual programming and online learning method, and these methods have the problems of low precision, slow response, easy to be disturbed by environment and the like.

[0003] Manual programming is the earliest and most basic method of robot control, which requires writing control programs manually and uploading them to the robot controller for execution. This method requires highly professional skills and experience, and a large amount of programming work is required for complex tasks; online learning refers to the method that the robot learns how to perform a specific task by interacting with the environment in real time. The advantage of this method is that it can quickly adapt to different environments and tasks, but it also has problems such as long learning time, low precision, and being easily affected by environmental changes; in recent years, people have begun to try to use offline reinforcement learning technology to control robots. Reinforcement learning (RL) is a learning paradigm in which an agent learns how to maximize its cumulative reward in a specific environment by trying to execute action instructions. The representative work of online reinforcement learning is the SAC algorithm (principle can refer to the literature "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor", translated as "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor", published by PMLR publishing agency, published in 2018), which can be used to train the network parameters of the reinforcement learning strategy. Offline reinforcement learning is a data-driven learning paradigm that learns only from static datasets of previously collected experiences. In this setting, the behavior policy (the policy that collects the dataset) interacts with the environment to collect static offline datasets, which can be used to learn robot control strategies without further interaction with the environment. This mode is very valuable in environments where online interaction is not practical, because data collection is likely to be expensive, or it may be dangerous to collect these datasets (for example, robots, education, medical care, and autonomous driving). Even in cases where online interaction with the environment is feasible, people may still prefer to use previously collected data to improve the versatility of complex fields.

[0004] To address these issues, in recent years, people have begun to try to use machine learning techniques such as offline reinforcement learning for robot control. According to whether the dynamics model of the robot control system and the environment is used in the control process of the robot, the current robot control method based on offline reinforcement learning method is usually divided into model-free and model-based methods. Many model-free control methods limit the generation of action instructions outside the distribution of the offline data set (Out Of Distribution, OOD) to reduce extrapolation errors. Existing methods usually constrain the policy directly, typical algorithms include CQL (principle can be referred to the literature “A minimalist approach to offline reinforcement learning”, translated as “A minimalist approach to offline reinforcement learning”, published in NIPS, published in 2021), conservative estimation of value function (principle can be referred to the literature “Conservative q-learning for offline reinforcement learning”, translated as “Conservative q-learning for offline reinforcement learning”, published in NIPS, published in 2020), IQL (principle can be referred to the literature “Offline reinforcement learning with implicit q-learning”, translated as “Offline reinforcement learning with implicit q-learning”, published in ICLR, published in 2022), and TD3+BC (principle can be referred to the literature “A minimalist approach to offline reinforcement learning”, translated as “A minimalist approach to offline reinforcement learning”, published in NIPS, published in 2021) and other methods to make the robot learn the search and rescue strategy in the offline data set as much as possible, but it is difficult for the robot to learn the search and rescue strategy not contained in the data set. Model-based robot control methods use dynamics models to learn or generate robot control strategies, which have been introduced into the offline reinforcement learning setting in recent years and have achieved better generalization performance (ability to generate robot control strategies not present in the data set). Model-based methods use recorded data sets to train the environment dynamics model in a supervised manner.Dyna-style algorithm (principle can refer to the literature "Offline reinforcement learning from images with latent space models", translated as "Learning, planning and reaction integration architecture based on approximate dynamic programming", published by Elsevier Press, published in 1990) can be used for policy optimization, and then widely used by model-based reinforcement learning algorithms, such as MOBO algorithm (principle can refer to the literature "When to Trust Your Model: Model-Based Policy Optimization", translated as "When to Trust Your Model: Model-Based Policy Optimization", published in NIPS, published in 2019). Based on the model of the robot control method has the potential of generalization ability, which can make the robot have the ability to obtain the search and rescue strategy that does not appear in the data set. At the same time, due to the limited amount of offline data set, it is impossible to cover all samples in the sample space, so it is essential to quantify the credibility of the model to avoid overestimation. MOPO algorithm (principle can refer to the literature "MOPO: Model-based offline policy optimization", translated as "MOPO: Model-based offline policy optimization", published in NIPS, published in 2020) constructs a pessimistic Markov decision process (on the basis of the original Markov decision process, the reward generated by the learned environmental dynamics model is subtracted from the predicted uncertainty to obtain a pessimistic reward, which is used to replace the reward in the original Markov decision process), and uses uncertainty estimation as a penalty when evaluating the strategy, so that the robot is more likely to obtain the search and rescue strategy in the data set, but at the same time, it also causes the problem of difficult generalization to produce new strategies.

[0005] However, in complex scenarios, the terrain and topography faced by the robot search and rescue task may be very complex. In the search and rescue scene, various complex terrains and topographies may be encountered, such as mountains, valleys, plateaus, jungles, etc., which will bring great difficulties to the movement and search and rescue task of the robot, and put forward higher requirements for the search and rescue strategy of the robot. However, due to the limited quantity and quality of the offline data set, it is impossible to contain all the strategies of the robot adapting to complex scenarios, so new robot search and rescue methods are needed to improve the adaptability of the robot.

[0006] Based on the above analysis, the traditional robot search and rescue method needs a large amount of manual pre-programming, and these manual pre-programming often only applies to specific scenarios and cannot adapt to the needs of complex environments. The current search and rescue method based on offline reinforcement learning can enable the robot to obtain the existing strategy in the offline data set, but these strategies cannot cover the needs of most complex scenarios. Therefore, it is necessary to study a new robot search and rescue method to enable the robot to adapt flexibly in different complex scenarios and to learn new search and rescue strategies autonomously, thereby improving the search and rescue efficiency and accuracy of the robot. SUMMARY

[0007] The technical problem to be solved by the present application is: under the premise that the amount of data in the offline data set is limited (not more than 1000 demonstration trajectories) and the data quality is not high (the data set may not contain a search and rescue method suitable for the current complex scenario), how to control the robot to complete the search and rescue task in the complex scenario.

[0008] The present application provides a robot search and rescue method based on offline reinforcement learning encouraging strategy generalization, which trains a more optimistic and bold trajectory generation strategy model by constructing an optimistic Markov decision process (i.e. Optimistic Markov Decision Process, O-MDP) (on the basis of the original Markov decision process, the optimistic reward is obtained by adding the predicted uncertainty to the reward generated by the learned environment dynamics model, which is used to replace the reward in the original Markov decision process), uses the trajectory generation strategy model and relies on the good generalization performance of the environment dynamics model to generate some new trajectories (including optimistic trajectories and pessimistic trajectories), the strategies contained in these trajectories do not appear in the original offline data set, and then uses these trajectories to train the control strategy model of the robot offline to control the robot motion to adapt to the search and rescue task in the complex scenario.

[0009] The technical scheme of the present application is to first construct a robot control system composed of a server node and a robot node, the server node is installed with a first storage module, a first model module, a first trajectory module, a first strategy module, a second trajectory module and a second strategy module; the robot node is deployed with a detection module, a motion module and a calculation module. The first model module uses the D4RL dataset to train a deep probabilistic neural network and integrates it into an environment dynamics model. The first trajectory module and the second trajectory module respectively use the environment dynamics model to construct O-MDP and P-MDP. The first strategy module is trained under O-MDP, and the second strategy module is trained under P-MDP to obtain a robot search and rescue strategy. The robot node executes the search and rescue task according to the search and rescue strategy. The present application uses the uncertainty of the environment dynamics model to construct O-MDP (optimistic MDP), so that the robot can fully utilize the uncertainty to enable the trajectory generation strategy to have the ability to explore and generalize unknown strategies when only a limited offline dataset is available. Then the trajectory generation strategy is used to optimize the strategy offline. The optimized strategy also has the ability to explore unknown strategies in the trajectory generation strategy, and can better execute the search and rescue task in real complex scenarios, solving the problem of completing the search and rescue task under the condition of limited number and low quality of offline dataset. The meaning of "optimistic" is that for the uncertainty in the prediction process of the environment dynamics model, the best case is taken, that is, the maximum reward.

[0010] The present application comprises the following steps:

[0011] First step, construct a robot control system. The system is composed of a server node and a robot node.

[0012] The server node can be a PC or a server, which is installed with an Ubuntu (requires 16.04 or above version) operating system (for X86 architecture processor) or an Ubuntu Mate (requires 16.04 or above version) operating system (for ARM architecture processor), and is installed with a ROS (Robot Operating System) and a TensorFlow deep learning framework (requires 1.10.0 or above version) matched with the operating system. It is also installed with a first storage module, a first model module, a first trajectory module, a first strategy module, a second trajectory module and a second strategy module.

[0013] A robot node refers to a robot capable of perception, movement, and software execution. Suitable robot nodes include, but are not limited to, ground-based robot nodes such as the Boston Dynamics robot dog, the Jamoca robot dog from Tencent Robotics X Lab, and the Half-Cheetah robot. Each robot node, in addition to its built-in detection and motion modules, also has a computing module. The detection module consists of sensors capable of sensing its own and its environment's state and collecting environmental data from the search and rescue scenario, including odometry, speedometers, scanning radar, infrared cameras, and depth cameras. The detection module is connected to the computing module and periodically photographs or scans the search and rescue scenario within its sensor's visible range and the other M-1 robot nodes (excluding itself) every α seconds to obtain the current search and rescue scenario state, which is then sent to the computing module. The suggested value for α is 0.3 to 1.

[0014] The computation module is connected to the motion module, detection module, and second strategy module, and is responsible for generating action commands for the motion module to execute the search and rescue strategy. The computation module runs the same operating system as the server node, and also uses the same ROS and TensorFlow deep learning framework. The computation module contains a third strategy network created using the TensorFlow deep learning framework. This third strategy network consists of three fully connected neural network layers, each containing 256 neurons.

[0015] The first storage module is a memory module with more than 1GB of available space, connected to the first model module and the second strategy module. It stores the offline dataset, which contains D offline trajectories (D is a positive integer, with a suggested range of 1000-1200). Each offline trajectory is a sequence of L consecutive quadruplets (L is a positive integer, with a suggested range of 800-1000). The stored L quadruplets, i.e., one offline trajectory, are represented as follows: Where, r t This is a score evaluating the task completion rate based on the search and rescue scenario observed at time t. t+1 This represents the state of the search and rescue scenario observed at time point t+1. Logically, s t+1 This can be interpreted as being caused by s t After behavior a t The new search and rescue scenario state obtained afterward. t=1 represents the initial time, t=L represents the Lth time. This represents an offline trajectory consisting of L quadruples.

[0016] The first model module is connected with the first storage module, the first trajectory module and the second trajectory module. The first model module is responsible for generating different environment dynamics models according to different kinds of offline data sets. The first model module uses the TensorFlow deep learning framework to create N (N is a positive integer, and the recommended value range of N is 5-7) deep probabilistic neural networks (the principle can refer to the literature “Estimating the mean and variance of the target probability distribution”, which is published by IEEE in 1994) composed of full connection layers. The output result of each deep probabilistic neural network is the mean and variance of a certain Gaussian distribution. The N deep probabilistic neural networks in the first model module read the offline data set from the first storage module and are trained in a supervised learning manner. In order to further measure the model uncertainty, the first model module generally trains N deep probabilistic neural networks at the same time, and the N trained deep probabilistic neural networks are integrated into an environment dynamics model. The first model module sends the environment dynamics model to the first trajectory module and the second trajectory module.

[0017] The environment dynamics model is a model with the ability to predict environment dynamics and evaluate uncertainty. In model-based reinforcement learning, the environment dynamics model refers to the abstraction of learning the dynamics of the environment in which the robot interacts. The environment dynamics is generally modeled as a Markov decision process, which can be represented as a five-tuple where S represents the state space, A represents the action instruction space, and γ represents the discount factor of future rewards, The environment dynamics model is represented as which can be represented as SxA→S, that is, an action instruction in the action instruction space A acts on a state in the state space S, so that a new state in the state space S is transferred; R represents a reward function for scoring the current action instruction, and R can be represented as SxA→R, that is, an action instruction in the action instruction space A acts on a state in the state space S, and an immediate reward is obtained by using the reward function R. Generally, training the environment dynamics model corresponds to predicting S and R under the given state S, action instruction space A and discount factor γ, and the training process can be recorded as the function θ is the parameter of the environment dynamics model, s t represents the observed scene state at the tth time point; a t represents the action instruction taken by the robot between the tth time point and the t+1th time point; represents the scene state of the robot predicted by the environment dynamics model at time t+1. is the reward obtained by the robot at time t predicted by the environmental dynamics.

[0018] The first strategy module is connected with the first trajectory module and the second trajectory module. The first strategy module is responsible for generating action instructions for exploring the environmental dynamics model, and sending the action instructions to the first trajectory module and the second trajectory module. The trajectories generated by the action instructions contain more trajectories that do not appear in the offline data set; the first strategy module is composed of a first strategy network created using the TensorFlow deep learning framework, and the first strategy network is composed of three fully connected layers, each of which contains 256 neurons.

[0019] The "strategy" is specifically manifested in the parameters of the first strategy network in the first strategy module (i.e. the parameter values of the weight matrix and the bias vector between the fully connected layers): the first strategy module reads the optimistic trajectory from the first trajectory module, extracts s t from the optimistic trajectory, and the first strategy network multiplies s t with the weight matrix between each layer of neurons and adds the bias vector between each layer of neurons, and passes layer by layer to obtain the action instruction a t , and sends a t to the first trajectory module and the second trajectory module. Therefore, the network parameters of the first strategy network embody the available action instructions that should be output according to the current state, that is, embody the "strategy" of the present application.

[0020] The first trajectory module is connected with the first model module and the first strategy module, receives the environmental dynamics model from the first model module, and receives the action instruction a t from the first strategy module, constructs an O-MDP (optimistic MDP, the meaning of "optimistic" is that for the uncertainty in the process of predicting the environmental dynamics model, the best case is taken, that is, the maximum reward) based on a t and the environmental dynamics model, generates and stores the optimistic trajectory for the first strategy module to read. The first trajectory module has an experience data replay pool with a space of more than 200MB, which stores M1 optimistic trajectories generated by the first trajectory module (M1 is the size of the experience data replay pool, and is a positive integer, and the recommended value range is 5000-10000). The mth (1≤m≤M1) (corresponding to the mth time point) optimistic trajectory is represented as s m represents the environmental state at the mth time point of the first trajectory module. a m represents the action instruction generated by the first strategy module between the mth and m+1th time points. is the feedback score of the O-MDP for the completion of the task at the mth time point. s m+1 is the environmental state at the m+1th time point predicted by the O-MDP. is the feedback score of the O-MDP for the completion of the task at the mth time point. s m+1 is the environmental state at the m+1th time point predicted by the O-MDP.

[0021] The second trajectory module is connected with the first model module, the first strategy module and the second strategy module. The second trajectory module receives the environment dynamics model from the first model module and receives the action instruction a t from the first strategy module. Based on a t and the environment dynamics model, the second trajectory module constructs a P-MDP (Pessimistic MDP, the meaning of "pessimistic" is that for the uncertainty in prediction, the worst case is taken, i.e. the smallest reward), generates and stores a pessimistic trajectory for the second strategy module to read. The second trajectory module has an experience data replay pool with a space of more than 200MB, which stores M2 pessimistic trajectories (M2 is the size of the experience replay pool, which is a positive integer, and the recommended value range is 5000-10000). The mm(th, 1≤mm≤M2) trajectory in the pessimistic trajectory can be represented by a four-tuple, which is s mm , which represents the environment state at the mm(th time point of the second trajectory module. a mm , which represents the action instruction generated by the first strategy module between the mm(th and mm+1(th time points. is the feedback score of the P-MDP for the completion of the task at the mm(th time point. s mm+1 is the environment state at the mm+1(th time point predicted by the P-MDP.

[0022] The second strategy module is connected with the second trajectory module, the first storage module and the calculation module, and is composed of a second strategy network created using the TensorFlow deep learning framework. The second strategy network is the same as the first strategy network of the first strategy module and is also composed of three fully connected layers. Each fully connected layer contains 256 neurons. The second strategy module reads the offline trajectory in the offline data set from the first storage module, reads the pessimistic trajectory from the second trajectory module, generates the final search and rescue strategy of the robot using the offline strategy optimization method, and sends the search and rescue strategy to the calculation module of the robot node.

[0023] Secondly, an offline data set for training the N deep probabilistic neural networks in the first model module is prepared, and the method is as follows:

[0024] 2.1 Select the D4RL dataset (see the paper “Fu J, Kumar A, Nachum O, et al. D4rl: Datasets for deep data-driven reinforcement learning [J]. arXiv preprint arXiv:2004.07219, 2020.” by Fu J, et al.) as the offline dataset for training N deep probabilistic neural networks. The D4RL offline dataset contains various different task scenarios, such as robot control, game play, etc., and provides high-quality trajectory data. The D4RL dataset can provide different task environments and evaluation metrics, including continuous control tasks and discrete control tasks.

[0025] The D4RL dataset includes 4 types of datasets: “Random”, “Medium”, “Medium-replay”, and “Medium-expert”. The “Random” dataset is composed of trajectories collected by a randomly initialized policy network. The “Medium” includes trajectories collected from early-stopping policies trained by the SAC algorithm (see the paper “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor” published by PMLR in 2018). The “Medium-replay” includes all trajectories collected during the process of training the SAC algorithm to the performance level of “Medium”. The “Medium-expert” is composed of a 1:1 combination of “Medium” data and expert data collected using the SAC algorithm. Each type of dataset contains 1000 trajectories in total. Since these 4 types of datasets are well-known trajectories used before robot search and rescue, these trajectories are collectively referred to as offline trajectories. An offline trajectory is a sequence of L consecutive quadruples (L is a positive integer, and the recommended value range of L is 800-1000), and the L quadruples stored in a trajectory are represented as The quadruple is <s t ,a t ,r t ,s t+1 > where s t represents the environment state of the robot node at the t-th time point. a t represents the action command generated by the robot node between the t-th and t+1-th time points. r ts is the feedback score of the environment to the robot node at time point t. t+1 This represents the search and rescue scenario state observed by the robot node at time point t+1. The policies contained in the D4RL dataset are the strategies by which the robot dog "Halfcheetah" learns to walk in the simulation environment. Each class in the D4RL dataset contains 1000 trajectories, which is equivalent to 1000×L quadruples, for a total of 4×1000×L quadruples.

[0026] 2.2 Store the D4RL dataset used for offline training of N deep probabilistic neural networks into the first storage module.

[0027] The third step involves the first model module training N deep probabilistic neural networks using the D4RL dataset in the first storage module. These N trained deep probabilistic neural networks are then integrated into a trained environmental dynamics model, which is sent to the first trajectory module and the second trajectory module. The method is as follows:

[0028] 3.1 The first model module reads the offline dataset (i.e., the D4RL dataset) from the first storage module.

[0029] 3.1.2 Initialize weight parameters: Initialize the weight parameters of the N neural network models in the first model module to random numbers between [0,1].

[0030] 3.1.3 Set network training parameters, setting the learning rate to 2.5 × 10⁻⁶. -4 By setting the batch size to 32 and the number of training iterations E to 100, the best environmental dynamics model can be obtained.

[0031] 3.1.4 The first model module uses supervised learning to simultaneously train N deep probabilistic neural networks, and the environmental dynamics model after E iterations is represented as follows: This formula indicates that each deep probabilistic neural network is used to fit a Gaussian distribution. Indicates the basis (s) t ,a t )right The function used for prediction is θ, where θ represents the parameters of the environmental dynamics model. There are N Gaussian distributions in total, where... Denotes a Gaussian distribution, where It is the reward predicted by the environmental dynamics model at time point t. This is the state predicted by the environmental dynamics model at time point t+1. Represents the i-th pair (s) t ,a t The mean of the predicted results follows a Gaussian distribution. Represents the i-th pair (s) t ,a t The variance of the predicted results follows a Gaussian distribution. N trained deep probabilistic neural networks are obtained, and these N networks are integrated into an environmental dynamics model. This environmental dynamics model is then sent to the first trajectory module and the second trajectory module.

[0032] Fourth step: The first trajectory module receives the environmental dynamics model from the first model module, uses the environmental dynamics model to estimate the uncertainty of the trajectories in the D4RL dataset, and constructs the O-MDP. The method is as follows:

[0033] 4.1 The first trajectory module receives the environmental dynamics model from the first model module.

[0034] 4.2 The first trajectory module uses an environmental dynamics model to estimate the uncertainty of scene state and action commands in T (T = 1000 × L) quadruplets of any class of dataset in the D4RL dataset. The method is as follows:

[0035] 4.2.1 Let the scene state and action command in the T quadruples of the D4RL dataset be represented by (s1, a1), ..., (s t ,a t ), …, (s T ,a T )express.

[0036] 4.2.2 Initialize t = 1;

[0037] 4.2.3 Calculation (s) t ,a t Uncertainty assessment of u(s) t a t ), Indicates to Calculate the L2 norm of the output result. This represents the maximum value among the L2 norms of the variances of the outputs of N deep probabilistic neural networks.

[0038] 4.2.4 The first trajectory module uses any deep probabilistic neural network pair (s) from the environmental dynamics model. t ,a t ) to make predictions, and obtain (s t ,a t The first reward function

[0039] 4.2.5 The first trajectory module calculates the t-th optimistic reward. Where λ oIt is the optimistic weight (λ) o λ is a positive integer. o The recommended value range is 0 to 5, which is used to construct O-MDP.

[0040] 4.2.6 The first trajectory module is based on optimistic reward. Construct an O-MDP, which is represented by a quintuple: Where R o Indicates optimistic reward The optimistic reward space is formed. It is an environmental dynamics model.

[0041] 4.2.7 Let t = t + 1. If t ≤ T, go to 4.2.3. If t > T, it means that the O-MDP construction was completed using T quadruplets (i.e., 1000 offline trajectories). Go to step 5.

[0042] Fifth, the second trajectory module receives the environmental dynamics model from the first model module, uses the environmental dynamics model to estimate the uncertainty of the trajectory in the D4RL dataset, and constructs the P-MDP.

[0043] 5.1 The second trajectory module receives the environmental dynamics model from the first model module.

[0044] 5.2 The second trajectory module uses an environmental dynamics model to estimate the uncertainty of scene states and action commands in T quadruplets of the same class of dataset selected in step 4.2 in the D4RL dataset. The method is as follows:

[0045] 5.2.1 Let the scene state and action command in the T quadruples of the D4RL dataset be represented by (s1, a1), ..., (s t ,a t ), …, (s T ,a T )express.

[0046] 5.2.2 Initialize t = 1;

[0047] 5.2.3 Calculate (s) using the same method as in 4.2.3. t ,a t Uncertainty assessment of u(s) t ,a t ).

[0048] 5.2.4 The second trajectory module uses any deep probabilistic neural network pair (s) from the environmental dynamics model. t ,a t ) to make predictions, and obtain (s t ,a t The second reward function

[0049] 5.2.5 The second trajectory module calculates the t-th pessimistic reward. Where λ p It is the pessimistic weight (λ) p λ is a positive integer. p The recommended value range is 0 to 5, which is used to construct P-MDP.

[0050] 5.2.6 The second trajectory module is based on pessimistic reward. Construct a P-MDP, which is represented by a quintuple: Where R p Indicates a pessimistic reward The resulting pessimistic reward space.

[0051] 5.2.7 Let t = t + 1. If t ≤ T, go to 5.2.3. If t > T, it means that the P-MDP construction of T quadruplets has been completed. Go to step 6.

[0052] Step 6: Train the first strategy module under O-MDP and the second strategy module under P-MDP, training the two strategy modules alternately to obtain the robot's search and rescue strategy. The method is as follows:

[0053] 6.1 Initialize the number of training epochs: epoch = 1; the maximum number of training epochs: I = 1000; initialize the training batch B, where B is a positive integer and the suggested value range for B is 32 to 64; initialize the weight parameters of the first policy network in the first policy module and the second policy network in the second policy module to random numbers between [0,1].

[0054] 6.2 The first policy module is trained as follows:

[0055] 6.2.1 Initialize m = 1;

[0056] 6.2.2 The first strategy module randomly selects the search and rescue scenario state s at the m-th time point from any four-tuple in any offline trajectory of the D4RL dataset, which is of the same category as in step 4.2. m ;

[0057] 6.2.3 The first strategy module is based on s m Generate action command a m ;

[0058] 6.2.4 The first strategy module is based on the environmental dynamics model. Calculate the optimistic reward at time point m. And predict and execute action instructions a mthe search and rescue scene state s at the m+1th time point m+1 ;

[0059] 6.2.5 The first strategy module sends the quadruple to the first trajectory module, which saves the quadruple as an optimistic trajectory;

[0060] 6.2.6 Let m = m + 1, if m ≤ M1, go to 6.2.2, if m > M1, it means that M1 optimistic trajectories are generated, go to 6.2.6;

[0061] 6.2.7 The first strategy module randomly samples B optimistic trajectories from the first trajectory module, trains the first strategy module using the SAC algorithm, and updates the network parameters in the first strategy module;

[0062] 6.3 Train the second strategy module by:

[0063] 6.3.1 Initialize the variable mm = 1;

[0064] 6.3.2 The second strategy module randomly selects the search and rescue scene state s at the mmth time point from any one of the quadruples in any one offline trajectory in the same category of data set in the D4RL data set as in step 4.2; mm ;

[0065] 6.3.3 The second strategy module generates an action instruction a mm based on s mm ;

[0066] 6.3.4 The second strategy module calculates the pessimistic reward at the mmth time point according to the environmental dynamics model and predicts the search and rescue scene state s mm at the mm+1th time point after executing the action instruction a mm+1 ;

[0067] 6.3.5 The second strategy module sends the quadruple to the second trajectory module, which saves the quadruple as an optimistic trajectory;

[0068] 6.3.6 Let mm = mm + 1, if mm ≤ M2, go to 6.3.2, if mm > M2, it means that M2 pessimistic trajectories are generated, go to 6.3.6;

[0069] 6.3.7 The second strategy module randomly samples B / 2 pessimistic trajectories from the second trajectory module, combines the B / 2 offline trajectories in the first storage module, and uses the TD3+BC algorithm (principle can refer to the paper "A minimalist approach to offline reinforcement learning", translated as "A minimalist approach to offline reinforcement learning", published in NIPS, published in 2021) to train the second strategy module and update the network parameters of the second strategy network.

[0070] 6.4 Let epoch = epoch + 1. If epoch is equal to the set maximum training round I, execute step 6.5; otherwise, go to step 6.2.

[0071] 6.5 The second strategy module generates a.data format file from the network parameters in the second strategy network through the TensorFlow deep learning framework. Since the.data format file contains the structure and weights of the deep neural network in the trained second strategy network, the structure and weights of the deep neural network in the second strategy network contain instructions for guiding the search and rescue actions of the robot node, so the.data format file is the search and rescue strategy.

[0072] Step 7, the second strategy module sends the.data parameter format file to the robot computing module, and the third strategy network in the computing module loads the.data parameter format file, that is, the computing module directly uses the trained network parameters of the second strategy module, without the need for training.

[0073] Step 8, deploy the robot search and rescue system constructed in step 1 in a real unknown open complex scene, and the robot node executes the search and rescue task according to the search and rescue strategy, i.e.data parameter format file. The method is:

[0074] 8.1 According to the requirements of the multi-target sampling task, set the number of target points that the robot system needs to sample, and the detection module of the robot node detects the coordinates of all search and rescue target points and stores them in the target point list. The computing module of the robot node selects the target point to go to in order from the target point list.

[0075] 8.2 Initialize variable k = 0;

[0076] 8.3 The detection module of the robot node detects the state s k of the kth task scene and sends it to the computing module of the robot node; k

[0077] 8.4 The computing module of the robot node receives s k ​and the target point position selected from the target point list, the state-target pair is (s k , target point position in the target point list);

[0078] 8.5 The third strategy network of the calculation module loads the.data parameter format file, and makes an action instruction decision according to the state-target pair, to obtain the kth action instruction a k k and sends ato the motion module;

[0079] 8.6 The motion module of the robot node executes a k and moves to the target point position in the target point list in the state-target pair;

[0080] 8.7 If the robot node executes a k and reaches the vicinity of the target point position in the target point list within d meters, d is a positive number, generally set to 1-3, target sampling is performed, and one target point within d meters is deleted from the target point list, and step 8.8 is executed; if the robot node executes a k and still does not reach the vicinity of the target point position within d meters, k=k+1, step 8.3 is turned to continue moving to the target point position;

[0081] 8.8 The robot node judges whether there is a target point in the target point list, if there is still a target point that has not been sampled and the robot is still moving, step 8.2 is turned; if there is no target point in the target point list that has not been sampled or the robot stops moving, the ninth step is executed;

[0082] The ninth step, if the target point list is empty, it is determined that the task is successful, and the process ends; if the target point list is not empty, it means that the robot cannot complete the search and rescue task in the complex terrain, and it is determined that the task fails, and the process ends.

[0083] The present application provides a robot search and rescue method based on offline reinforcement learning. Compared with the traditional robot search and rescue method, the method trains the strategy through offline data, reduces the dependence on real-time interaction, improves the efficiency and stability of robot search and rescue. In addition, the present application can also predict the return of different action instructions taken by the robot in a given state, and provide an optimistic trajectory and a pessimistic trajectory generation strategy based on model prediction, so as to realize the robot to obtain the optimal trajectory outside the offline data set. Therefore, the present application has important practical value and wide application prospect, for example, in the fields of industrial automation, intelligent logistics, military, etc.

[0084] The beneficial technical effects of the present application are:

[0085] (1) The fourth step of the present application is based on an offline data set, and optimistic trajectories and pessimistic trajectories are generated by an environmental dynamics model without interaction with the real environment, which can effectively solve the problem of low training efficiency and the need for a large number of samples in the training process of the reinforcement learning algorithm described in the background technology.

[0086] (2) The third step of the present application fully utilizes the uncertainty of the environmental dynamics model, and by constructing O-MDP and P-MDP, the robot obtains an efficient search and rescue strategy. The O-MDP ensures that the robot strategy fully utilizes the environmental dynamics model, and the P-MDP makes the safety of the search and rescue strategy obtained in the final training.

[0087] (3) Compared with the existing search and rescue strategy method mentioned in the background technology, the first strategy module and the second strategy module are trained using the D4RL offline data set in the present application, which can make the robot learn a more optimistic and bold strategy (such as jumping, running, etc.), to adapt to the needs of search and rescue tasks in complex terrain, and solve the problem of completing search and rescue tasks under the condition of limited number and low quality of offline data sets. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 is a logical structure diagram of the robot search and rescue system constructed in the first step of the present application.

[0089] Figure 2 is a general flowchart of the present application.

[0090] Figure 3 is an experimental schematic diagram of the HalfCheetah robot used in the simulation environment of the present application.

[0091] Figure 4 is an experimental schematic diagram of training on two representative data sets in D4RL. Each experiment is repeated three times, and the average value and variance are recorded. The most important part of the present application (denoted as ORPO) is to introduce O-MDP to train the strategy in the first strategy module. In order to prove the effectiveness of this structure, a comparative experiment is conducted, and the first trajectory module, the corresponding O-MDP and the first strategy module are removed, and this method is denoted as OROO. From the training curve, it can be seen that the task completion rate of the strategy generated by OROO is greatly reduced compared with ORPO, and the present application can control the robot to adapt to search and rescue tasks in complex scenarios.

[0092] Figure 5 is an example of the search and rescue effect of the present application. is the action that the robot can complete using the strategy generated by the present application in the eighth step, that is, the robot has learned to jump to adapt to search and rescue tasks in complex scenarios.

[0093] Table 1 is a quantitative representation of the test results on the public dataset. The dataset is divided into four categories: random, Medium, Medium-replay and Medium-expert. Compared with existing methods, the present application can further improve performance. Specifically, ORPO achieves higher normalized scores in D4RL and more stable performance (lower normalized score standard deviation) than the background art described MOPO and TD3+BC in the four categories of environment dataset configurations, which can reduce the search and rescue time and improve the search and rescue accuracy when applied to actual search and rescue scenarios in the eighth step. DETAILED DESCRIPTION

[0094] The specific embodiments of the present application will be described below with reference to the accompanying drawings, as shown in the drawings, the present application comprises the following steps: Figure 2

[0095] First, build a robot control system. As shown in the drawings, the system is composed of a server node and a robot node. Figure 1

[0096] The server node can be a PC or a server, which is installed with an Ubuntu (16.04 in this embodiment) operating system and a ROS and TensorFlow deep learning framework (version 1.10.0) matched with the operating system. It is also installed with a first storage module, a first model module, a first trajectory module, a first strategy module, a second trajectory module and a second strategy module.

[0097] The robot node refers to a robot that can perceive, move and has software running capability. The robot node used in this embodiment is a Half Cheetah robot. In addition to deploying the detection module and motion module provided by the robot, a computing module is also installed on each robot node. The detection module is a sensor that can perceive its own and environmental state and collect search and rescue scene environmental data, including odometer, speedometer, scanning radar, infrared camera, depth camera, etc. The detection module is connected to the computing module, and every α seconds, it periodically takes pictures or scans the search and rescue scene and the other M-1 robot nodes except itself within the visible range of the sensor, obtains the current search and rescue scene state, and sends the search and rescue scene state to the computing module. The value of α ranges from 0.3 to 1.

[0098] ​​The computation module is connected to the motion module, detection module, and second strategy module, and is responsible for generating action commands for the motion module to execute the search and rescue strategy. The computation module runs the same operating system as the server node, and also uses the same ROS and TensorFlow deep learning framework. The computation module contains a third strategy network created using the TensorFlow deep learning framework. This third strategy network consists of three fully connected neural network layers, each containing 256 neurons.

[0099] The first storage module is a memory with more than 1GB of available space, connected to the first model module and the second strategy module. It stores the offline dataset, which contains D offline trajectories (D is a positive integer, and in this embodiment, D is 1000). Each offline trajectory is a sequence of L consecutive quadruplets (L is a positive integer, and in this embodiment, L is 1000). The stored L quadruplets, i.e., one offline trajectory, are represented as follows: Where, r t This is a score evaluating the task completion rate based on the search and rescue scenario observed at time t. t+1 This represents the state of the search and rescue scenario observed at time point t+1. Logically, s t+1 This can be interpreted as being caused by s t After behavior a t The new search and rescue scenario state obtained afterward. t=1 represents the initial time, t=L represents the Lth time. This represents an offline trajectory consisting of L quadruples.

[0100] The first model module is connected to the first storage module, the first trajectory module, and the second trajectory module. The first model module is responsible for generating different environmental dynamics models based on different types of offline datasets. The first model module uses the TensorFlow deep learning framework to create N (N is a positive integer; in this embodiment, N is 7) deep probabilistic neural networks composed of fully connected layers. The output of each deep probabilistic neural network is the mean and variance of a Gaussian distribution. The N deep probabilistic neural networks in the first model module read the offline dataset from the first storage module and are trained using supervised learning. To further measure model uncertainty, the first model module typically trains N deep probabilistic neural networks simultaneously, and the N trained deep probabilistic neural networks are integrated into a single environmental dynamics model. The first model module then sends the environmental dynamics model to the first trajectory module and the second trajectory module.

[0101] Environmental dynamics models are models capable of predicting environmental dynamics and assessing uncertainties. They are modeled as Markov decision processes, represented by quintuples. where S represents the state space, A represents the action space, and γ represents the discount factor of future rewards, represents the environment dynamics model, which is used to simulate the real environment, which can be represented as SxA→S, i.e., an action in the action space A acts on a state in the state space S, resulting in a transition to a new state in the state space S; R represents the reward function, which is used to score the current action, and R can be represented as SxA→R, i.e., an action in the action space A acts on a state in the state space S, resulting in an immediate reward obtained by the reward function R. Given the state S and the action space A and the discount factor γ, training the environment dynamics model corresponds to predicting S and R, and the training process can be denoted as a function θ is the parameter of the environment dynamics model, s t represents the observed scene state at the t-th time point; a t represents the action taken by the robot between the t-th time point and the t+1-th time point; represents the scene state of the robot at time t+1 predicted by the environment dynamics model. is the reward obtained by the robot at time t predicted by the environment dynamics.

[0102] The first strategy module is connected to the first trajectory module and the second trajectory module. The first strategy module is responsible for generating action instructions for exploring the environment dynamics model and sending the action instructions to the first trajectory module and the second trajectory module. The trajectories generated by these action instructions contain more trajectories that do not appear in the offline data set; the first strategy module is composed of a first strategy network created using the TensorFlow deep learning framework, and the first strategy network is composed of three fully connected layers, each containing 256 neurons.

[0103] The first strategy module reads the optimistic trajectory from the first trajectory module and extracts s t from the optimistic trajectory. t The first strategy network multiplies the weight matrix between each layer of neurons and adds the bias vector between each layer of neurons, and passes layer by layer to obtain the action instruction a t Send a t to the first trajectory module and the second trajectory module.

[0104] The first trajectory module is connected to the first model module and the first strategy module, receives the environment dynamics model from the first model module, and receives the action instruction a t from the first strategy module, and generates the optimistic trajectory based on a tand the environment dynamics model to build an O-MDP (optimistic MDP), generate and store the optimistic trajectory, for the first strategy module to read. The first trajectory module has an experience data replay pool with a space of more than 200MB, which stores M1 optimistic trajectories generated by the first trajectory module (M1 is the size of the experience data replay pool, which is a positive integer, and the value in the embodiment is 10000). The mth (1≤m≤M1) (corresponding to the mth time point) optimistic trajectory is represented as s m s represents the environment state at the mth time point of the first trajectory module. m a represents the action instruction generated by the first strategy module between the mth and (m+1)th time points. is the feedback score of the O-MDP for the task completion at the mth time point. m+1 is the environment state predicted by the O-MDP at the (m+1)th time point.

[0105] The second trajectory module is connected with the first model module, the first strategy module, and the second strategy module. The second trajectory module receives the environment dynamics model from the first model module, receives the action instruction a t from the first strategy module, and builds a P-MDP (pessimistic MDP) based on a t and the environment dynamics model to generate and store the pessimistic trajectory, for the second strategy module to read. The second trajectory module has an experience data replay pool with a space of more than 200MB, which stores M2 pessimistic trajectories (M2 is the size of the experience data replay pool, which is a positive integer, and the value in the embodiment is 10000). The mmth (1≤mm≤M2) trajectory in the pessimistic trajectory can be represented as a four-tuple, and the four-tuple is s mm s represents the environment state at the mmth time point of the second trajectory module. mm a represents the action instruction generated by the first strategy module between the mmth and (mm+1)th time points. is the feedback score of the P-MDP for the task completion at the mmth time point. mm+1 is the environment state predicted by the P-MDP at the (mm+1)th time point.

[0106] The second strategy module is connected with the second trajectory module, the first storage module, and the calculation module, and is composed of a second strategy network created by using the TensorFlow deep learning framework; the second strategy network is the same as the first strategy network of the first strategy module, and is also composed of three fully connected layers. Each fully connected layer contains 256 neurons. The second strategy module reads the offline trajectory in the offline data set from the first storage module, reads the pessimistic trajectory from the second trajectory module, generates the final search and rescue strategy of the robot by using the offline strategy optimization method, and sends the search and rescue strategy to the calculation module of the robot node.

[0107] Second step, prepare the offline dataset for training the N deep probabilistic neural networks in the first model module, the method is:

[0108] 2.1 Select the D4RL dataset as the offline dataset for training the N deep probabilistic neural networks, the D4RL offline dataset contains various different task scenarios such as robot control, game play, etc., and provides high-quality trajectory data. The D4RL dataset can provide different task environments and evaluation indicators, including continuous control tasks and discrete control tasks.

[0109] The D4RL dataset includes 4 types of datasets: "Random", "Medium", "Medium-replay", and "Medium-expert". Each type of dataset contains 1000 trajectories. An offline trajectory is a sequence of L consecutive four-tuples (L is a positive integer, and the value of L in this embodiment is 1000), and the L four-tuples representing a trajectory are stored as The four-tuple is <s t ,a t ,r t ,s t+1 >, where s t represents the state of the robot node in the environment at the t-th time point. a t represents the action command generated by the robot node between the t-th and t+1-th time points. r t is the feedback score of the environment to the robot node at the t-th time point. s t+1 represents the search and rescue scene state observed by the robot node at the t+1-th time point. The strategy included in the D4RL dataset is how the robot dog "Halfcheetah" learns to walk in a simulated environment. Each type of data in the D4RL dataset contains 1000 trajectories, i.e. 1000*L four-tuples, and a total of 4*1000*L four-tuples.

[0110] 2.2 Store the D4RL dataset for offline training of the N deep probabilistic neural networks in the first storage module.

[0111] Third step, the first model module uses the D4RL dataset in the first storage module to train the N deep probabilistic neural networks, and integrates the N trained deep probabilistic neural networks into an environment dynamics model, and sends the environment dynamics model to the first trajectory module and the second trajectory module. The method is:

[0112] 3.1 The first model module reads the offline dataset (i.e. the D4RL dataset) from the first storage module.

[0113] 3.1.2 Initialize the weight parameters, initialize the weight parameters of N neural network models in the first model module to random numbers between [0, 1].

[0114] 3.1.3 Set the network training parameters, set the learning rate to 2.5 x 10 -4 , set the batch size to 32, and set the number of training iterations E to 100, so as to obtain the best environment dynamics model.

[0115] 3.1.4 The first model module uses the method of supervised learning to train N deep probabilistic neural networks at the same time, and the environment dynamics model after E iterations of training is expressed as This formula indicates that each deep probabilistic neural network is used to fit a Gaussian distribution. represents the function of predicting t , a t ) according to (s , a , where is the reward predicted by the environment dynamics model at the tth time point, is the state predicted by the environment dynamics model at the t+1th time point, represents the mean of the Gaussian distribution of the prediction result of the ith pair of two-tuple (s t , a t ), represents the variance of the Gaussian distribution of the prediction result of the ith pair of two-tuple (s t , a t ). N trained deep probabilistic neural networks are obtained, an environment dynamics model is integrated from the N trained deep probabilistic neural networks, and the environment dynamics model is sent to the first trajectory module and the second trajectory module.

[0116] Step 4, the first trajectory module receives the environment dynamics model from the first model module, estimates the uncertainty of the trajectory in the D4RL dataset using the environment dynamics model, and constructs an O-MDP, the method is:

[0117] 4.1 The first trajectory module receives the environment dynamics model from the first model module.

[0118] 4.2 The first trajectory module estimates the uncertainty of the scene state and action instruction in the T (T = 1000 x L) four-tuples in any one type of dataset in the D4RL dataset using the environment dynamics model, the method is:

[0119] 4.2.1 Let the scene state and action instruction in the T four-tuples in the D4RL dataset be (s1, a1), …, (st a t ),…,( T a T ) represent.

[0120] 4.2.2 Set t = 1;

[0121] 4.2.3 Calculate the uncertainty evaluation u(s t a t ) of (s t a t ), represent the two-norm of the output result of , represent the maximum value in the two-norm of the variance of N depth probabilistic neural network outputs.

[0122] 4.2.4 The first trajectory module uses any one of the depth probabilistic neural networks in the environment dynamics model to predict (s t a t ), to obtain the first reward function t a t ) of (s

[0123] 4.2.5 The first trajectory module calculates the t-th optimistic reward where λ o is an optimistic weight (λ o is a positive integer, and in the embodiment, λ o is 1), used to construct an O-MDP.

[0124] 4.2.6 The first trajectory module constructs an O-MDP based on the optimistic reward , and the O-MDP is represented by a five-tuple: where R o represents an optimistic reward space composed of the optimistic reward , is the environment dynamics model.

[0125] 4.2.7 Set t = t + 1, if t ≤ T, go to 4.2.3, if t > T, it is indicated that the O-MDP construction is completed by using T four-tuples (i.e. 1000 offline trajectories), and go to the fifth step.

[0126] In the fifth step, the second trajectory module receives the environment dynamics model from the first model module, estimates the uncertainty of the trajectories in the D4RL dataset using the environment dynamics model, and constructs a P-MDP;

[0127] 5.1 The second trajectory module receives the environmental dynamics model from the first model module.

[0128] 5.2 The second trajectory module uses an environmental dynamics model to estimate the uncertainty of scene states and action commands in T quadruplets of the same class of dataset selected in step 4.2 in the D4RL dataset. The method is as follows:

[0129] 5.2.1 Let the scene state and action command in the T quadruples of the D4RL dataset be represented by (s1, a1), ..., (s t ,a t ), …, (s T ,a T )express.

[0130] 5.2.2 Initialize t = 1;

[0131] 5.2.3 Calculate (s) using the same method as in 4.2.3. t ,a t Uncertainty assessment of u(s) t ,a t ).

[0132] 5.2.4 The second trajectory module uses any deep probabilistic neural network pair (s) from the environmental dynamics model. t ,a t ) to make predictions, and obtain (s t ,a t The second reward function

[0133] 5.2.5 The second trajectory module calculates the t-th pessimistic reward. Where λ p It is the pessimistic weight (λ) p λ is a positive integer, in this embodiment λ p The value of is 5), which is used to construct P-MDP.

[0134] 5.2.6 The second trajectory module is based on pessimistic reward. Construct a P-MDP, which is represented by a quintuple: Where R p Indicates a pessimistic reward The resulting pessimistic reward space.

[0135] 5.2.7 Let t = t + 1. If t ≤ T, go to 5.2.3. If t > T, it means that the P-MDP construction of T quadruplets has been completed. Go to step 6.

[0136] Step 6: Train the first strategy module under O-MDP and the second strategy module under P-MDP, training the two strategy modules alternately to obtain the robot's search and rescue strategy. The method is as follows:

[0137] 6.1 Initialize the number of training epochs: epoch = 1; the maximum number of training epochs: I = 1000; initialize the training batch B, where B is a positive integer and the suggested value range for B is 32 to 64; initialize the weight parameters of the first policy network in the first policy module and the second policy network in the second policy module to random numbers between [0,1].

[0138] 6.2 The first policy module is trained as follows:

[0139] 6.2.1 Initialize m = 1;

[0140] 6.2.2 The first strategy module randomly selects the search and rescue scenario state s at the m-th time point from any four-tuple in any offline trajectory of the D4RL dataset, which is of the same category as in step 4.2. m ;

[0141] 6.2.3 The first strategy module is based on s m Generate action command a m ;

[0142] 6.2.4 The first strategy module is based on the environmental dynamics model. Calculate the optimistic reward at time point m. And predict and execute action instructions a m Then, the search and rescue scenario state s at the (m+1)th time point. m+1 ;

[0143] 6.2.5 The first strategy module will use quadruples The data is sent to the first trajectory module, which then sends the quadruple. Save as an optimistic trajectory;

[0144] 6.2.6 Let m = m + 1. If m ≤ M1, go to 6.2.2. If m > M1, it means that M1 optimistic trajectories have been generated. Go to 6.2.6.

[0145] 6.2.7 The first strategy module randomly samples B optimistic trajectories from the first trajectory module, trains the first strategy module using the SAC algorithm, and updates the network parameters in the first strategy module;

[0146] 6.3 The second policy module is trained as follows:

[0147] 6.3.1 Initialize variable mm = 1;

[0148] 6.3.2 The second strategy module randomly selects the search and rescue scenario state s at the mm-th time point from any quadruple in any offline trajectory of the D4RL dataset of the same category as in step 4.2. mm ;

[0149] 6.3.3 The second strategy module is based on s mm Generate action command a mm ;

[0150] 6.3.4 The second strategy module is based on the environmental dynamics model. Calculate the pessimistic reward at the mm-th time point And predict and execute action instructions a mm The search and rescue scenario status s at the (mm+1)th time point mm+1 ;

[0151] 6.3.5 The second strategy module will use quadruples The data is sent to the second trajectory module, which then sends the quadruple. Saved as a pessimistic trajectory;

[0152] 6.3.6 Let mm = mm + 1. If mm ≤ M2, go to 6.3.2. If mm > M2, it means that M2 pessimistic trajectories have been generated. Go to 6.3.6.

[0153] 6.3.7 The second policy module randomly samples B / 2 pessimistic trajectories from the second trajectory module, combines them with B / 2 offline trajectories from the offline dataset in the first storage module, and uses the TD3+BC algorithm to train the second policy module and update the network parameters of the second policy network.

[0154] 6.4 Set epoch = epoch + 1. If epoch equals the set maximum number of training rounds I, then proceed to step 6.5; otherwise, go to step 6.2.

[0155] 6.5 The second strategy module generates a .data file from the network parameters of the second strategy network using the TensorFlow deep learning framework. Since the .data file contains the structure and weights of the deep neural network in the trained second strategy network, and the structure and weights of the deep neural network in the second strategy network contain instructions to guide the robot nodes in search and rescue actions, the .data file is the search and rescue strategy.

[0156] In the seventh step, the second strategy module sends the .data parameter format file to the robot computing module. The third strategy network in the computing module loads the .data parameter format file, meaning that the computing module directly uses the network parameters trained by the second strategy module without needing to train them.

[0157] Step 8, the robot search and rescue system constructed in the first step is deployed in a real unknown open complex scene, and the robot nodes perform search and rescue tasks according to the search and rescue strategy, i.e., the.data parameter format file. The method is as follows:

[0158] 8.1 According to the requirements of the multi-target sampling task, a plurality of target points to be sampled by the robot system are set, the detection module of the robot node detects the coordinates of all search and rescue target points, and stores the coordinates in a target point list. The calculation module of the robot node selects a target point to go to in sequence from the target point list.

[0159] 8.2 Initialize the variable k = 0;

[0160] 8.3 The detection module of the robot node detects the scene state s k of the kth task; k and sends s k to the calculation module of the robot node;

[0161] 8.4 The calculation module of the robot node forms a state-target pair with s k received from the detection module and the target point position selected from the target point list, and the state-target pair is (s k , the target point position in the target point list);

[0162] 8.5 The third strategy network of the calculation module loads the.data parameter format file, makes an action instruction decision according to the state-target pair, and obtains the kth action instruction a k , and sends a k to the motion module;

[0163] 8.6 The motion module of the robot node executes a k , and moves to the target point position in the target point list in the state-target pair;

[0164] 8.7 If the robot node executes a k and reaches within d meters of the target point position in the target point list, d is a positive number (set to 3 meters in this embodiment), performs target sampling, and deletes one target point within d meters from the target point list, and executes step 8.8; if the robot node executes a k and still does not reach within d meters of the target point position, let k = k + 1, and go to step 8.3 to continue moving to the target point position;

[0165] 8.8 The robot node judges whether there are still target points in the target point list, if there are still target points to be sampled and the robot is still moving, go to step 8.2; if there are no target points to be sampled in the target point list or the robot stops moving, execute step 9;

[0166] Figure 3is an experimental diagram of a HalfCheetah used in the simulation environment (Mujoco simulator, version number 2.1.0) used in the eighth step. The present application tests the effect in a simulation test search and rescue scene. The environment has a real physics engine, and there are objective physical factors such as friction and inertia. The task goal is that the robot system can successfully reach the target point to complete the search and rescue task through the detection module. These target points are set in front of the robot.

[0167] In the ninth step, if the target point list is empty, it is determined that the task is successful, and the process ends. If the target point list is not empty, it means that the robot cannot complete the search and rescue task in the complex terrain, and the task is determined to fail, and the process ends.

[0168] Figure 4 is a simulation test experimental result graph of the present application. The experimental environment is a simulation environment set in the Mujoco simulator (version number 2.1.0) as Figure 3 described. According to the steps described in Figure 2 , the following two indicators of search and rescue task completion rate were tested. Indicator one: forward movement reward. If the robot runs forward (right), it is given a positive reward. Indicator two: jumping reward. If the current center of gravity of the robot is higher than the initial center of gravity, it is given a positive reward. Figure 4 The abscissa is the training round number epoch of the strategy in the second strategy module during training, and the ordinate is the task completion degree score r t observed in the search and rescue scene at time t returned by the search and rescue scene. Figure 4 From , it can be seen that as the training round number increases, the task completion degree scores of different technologies will fluctuate, but the overall trend is increasing. It can be clearly seen that the data curve of the present application (represented by “x”) is much higher than the control group MOPO (represented by “o”). Higher scores mean that the robot can better complete the task of running forward and jumping to adapt to complex terrain, and thus can better complete the search and rescue task.

[0169] Figure 5 is an example of reaching the search and rescue effect of the present application. is the action that the robot can complete on the Mujoco simulator using the strategy generated by the present application in the eighth step, that is, the robot has learned to jump to adapt to the search and rescue task in a complex scene.

[0170] To illustrate the effect of the present application, the performance of the present application and the TD3+BC and MOPO offline reinforcement learning methods described in the background art on the D4RL dataset constructed in the second step is compared. In the numbers in Table 1, the front ± represents the average score, and the back ± represents the standard deviation. From Table 1, it can be seen that the present application can obtain higher scores, i.e. higher search and rescue success rates, and lower standard deviations, i.e. more stable search and rescue performance, compared with TD3+BC and MOPO.

[0171]

[0172] Table 1 Performance indicators of different algorithms on different datasets.

Claims

1. A robot search and rescue method based on offline reinforcement learning, characterized in that... Includes the following steps: The first step is to build a robot control system. This system consists of server nodes and robot nodes. The server nodes are PCs or servers with Ubuntu or Ubuntu Mate operating systems installed, along with ROS and TensorFlow deep learning frameworks that are compatible with the operating system. It also includes a first storage module, a first model module, a first trajectory module, a first strategy module, a second trajectory module, and a second strategy module. A robot node refers to a robot capable of sensing, moving, and running software. Each robot node, in addition to its built-in detection and motion modules, also has a computing module. The detection module is a sensor that can perceive its own and its environment's state and collect environmental data from the search and rescue scenario. The detection module is connected to the computing module, and every [time period missing]... The system periodically takes pictures or scans the search and rescue scene within the sensor's visible range and the other M-1 robot nodes besides itself every second to obtain the current search and rescue scene status, and then sends the search and rescue scene status to the computing module. The computing module is connected to the motion module, the detection module, and the second strategy module. It is responsible for generating action commands for the motion module to execute the search and rescue strategy. The computing module is equipped with the same operating system as the server node, as well as the same ROS and TensorFlow deep learning framework as the server node. The computing module contains a third strategy network created using the TensorFlow deep learning framework. The first storage module is a memory connected to the first model module and the second strategy module. It is used to store the offline dataset. The offline dataset contains D offline trajectories, where D is a positive integer. Each offline trajectory is a sequence of L consecutive quadruplets, where L is a positive integer. The stored L quadruplets constitute one offline trajectory. The first model module is connected to the first storage module, the first trajectory module, and the second trajectory module. The first model module is responsible for generating different environmental dynamics models based on different types of offline datasets. The first model module uses the TensorFlow deep learning framework to create N deep probabilistic neural networks (DNNs) composed of fully connected layers, where N is a positive integer. The output of each DNN is the mean and variance of a Gaussian distribution. The N DNNs in the first model module read offline datasets from the first storage module and are trained using supervised learning. To further measure model uncertainty, the first model module trains the N DNNs simultaneously, and the N trained DNNs are integrated into an environmental dynamics model. The first model module sends the environmental dynamics model to the first trajectory module and the second trajectory module. The environmental dynamics model is a model capable of predicting environmental dynamics and assessing uncertainty, represented as a quintuple. ,in The discount factor representing future rewards. This represents an environmental dynamics model used to simulate real-world environments. This can be represented as S×A→S, meaning an action instruction in the action instruction space A acts on a state in the state space S, causing a transition to a new state in the state space S; R represents the reward function, used to score the current action instruction, which can be represented as S×A→R, meaning an action instruction in the action instruction space A acts on a state in the state space S, resulting in an immediate reward obtained using the reward function R; given the state S, the action instruction space A, and the discount coefficient... In this case, the training environment dynamics model corresponds to the prediction of S and R, and the training process can be represented as a function. , These are parameters of the environmental dynamics model. This represents the scene state observed at time point t; This represents the action instructions that the robot takes between the t-th time point and the (t+1)-th time point; It is the reward that the robot receives at time t, as predicted by environmental dynamics; The first strategy module is connected to the first trajectory module and the second trajectory module. The first strategy module is responsible for generating action commands to explore the environmental dynamics model and sending these commands to the first and second trajectory modules. The trajectories generated by these action commands contain more trajectories not found in the offline dataset. The first strategy module consists of a first strategy network created using the TensorFlow deep learning framework. The "strategy" is specifically represented by the parameters of the first strategy network within the first strategy module: the first strategy module reads optimistic trajectories from the first trajectory module and extracts... The first strategy network will The weight matrices between neurons in each layer are multiplied, and the result is added to the bias vectors between neurons in each layer. This process is then passed layer by layer to obtain the action command. ,Will The first trajectory module is connected to the first model module and the first strategy module, receiving environmental dynamics models from the first model module and action commands from the first strategy module. based on An O-MDP (Optimistic MDP) is constructed using the environmental dynamics model to generate and store optimistic trajectories for the first strategy module to read. The first trajectory module contains an empirical data replay pool, storing M1 optimistic trajectories generated by the first trajectory module, where M1 refers to the size of the replay pool and is a positive integer. The m-th optimistic trajectory is represented by a quadruple […]. , , , ], 1≤m≤M1, This indicates the environmental state of the first trajectory module at time point m; This represents the action instructions generated by the first strategy module between the two time points m and m+1. It is the O-MDP feedback score for the task completion status at time point m; It is the environmental state at time point m+1 predicted by O-MDP; The second trajectory module is connected to the first model module, the first strategy module, and the second strategy module; the second trajectory module receives the environmental dynamics model from the first model module and action commands from the first strategy module. based on A pessimistic MDP (pessimistic trajectories) is constructed using the environmental dynamics model to generate and store pessimistic trajectories for the second strategy module to read. The second trajectory module contains an empirical data replay pool containing M2 pessimistic trajectories, where M2 is the size of the replay pool and is a positive integer. The mm-th pessimistic trajectory is represented by a quadruple < , , , >,1≤mm≤M2, This indicates the environmental state of the second trajectory module at time point mm; This indicates the action instructions generated by the first strategy module between time points mm and mm+1. It is the feedback score of P-MDP for the task completion status at time point mm; It is the environmental state predicted by P-MDP at time point mm+1. The second strategy module is connected to the second trajectory module, the first storage module, and the computing module. It consists of a second strategy network created using the TensorFlow deep learning framework. The second strategy module reads the offline trajectory from the offline dataset from the first storage module, reads the pessimistic trajectory from the second trajectory module, uses the offline strategy optimization method to generate the robot's final search and rescue strategy, and sends the search and rescue strategy to the computing module of the robot node. The second step is to prepare the offline dataset for training the N deep probabilistic neural networks in the first model module. The method is as follows: 2.1 The D4RL dataset was selected as the offline dataset for training N deep probabilistic neural networks. The D4RL offline dataset contains various task scenarios and provides trajectory data. The D4RL dataset includes four dataset classes: "Random", "Medium", "Medium-replay", and "Medium-expert". The "Random" dataset consists of trajectories collected from a randomly initialized policy network. "Medium" includes trajectories collected from early stopping policies trained using the SAC algorithm. "Medium-replay" includes all trajectories collected during the training of the SAC algorithm to the "Medium" performance level. "Medium-expert" is a 1:1 combination of "Medium" data and expert data collected using the SAC algorithm. Each dataset class contains 1000 offline trajectories. An offline trajectory is a sequence of L consecutive quadruples, and the stored L quadruples constitute one trajectory. The quadruple is < , , , >, This represents the environmental state of the robot node at time point t. This represents the action instructions generated by the robot node between time points t and t+1. It is the feedback score of the environment to the robot node at time point t; This represents the search and rescue scenario state observed by the robot node at time t+1; the policies contained in the D4RL dataset are the strategies by which the robot dog "Halfcheetah" learns to walk in the simulation environment; each class of data in the D4RL dataset contains 1000 trajectories, that is, 1000×L quadruples, and a total of 4×1000×L quadruples. 2.2 Store the D4RL dataset used for offline training of N deep probabilistic neural networks into the first storage module; Thirdly, the first model module trains N deep probabilistic neural networks using the D4RL dataset in the first storage module. These N trained deep probabilistic neural networks are then integrated into a trained environmental dynamics model, which is sent to the first trajectory module and the second trajectory module. The method is as follows: 3.1 The first model module reads the offline dataset, namely the D4RL dataset, from the first storage module; 3.1.2 Initialize weight parameters: Initialize the weight parameters of all N neural network models in the first model module to... Random numbers between; 3.1.3 Set network training parameters, and set the learning rate. Batch size Let the number of training iterations be... It is 100; 3.1.4 The first model module uses supervised learning to simultaneously train N deep probabilistic neural networks, and iterative training... The subsequent environmental dynamics model is expressed as follows: This formula indicates that each deep probabilistic neural network fits a Gaussian distribution; There are N Gaussian distributions here, among which Indicates a Gaussian distribution. ; Obtain N trained deep probabilistic neural networks, integrate the N trained deep probabilistic neural networks into an environmental dynamics model, and send the environmental dynamics model to the first trajectory module and the second trajectory module; Fourth step: The first trajectory module receives the environmental dynamics model from the first model module, uses the environmental dynamics model to estimate the uncertainty of the trajectories in the D4RL dataset, and constructs the O-MDP. The method is as follows: 4.1 The first trajectory module receives the environmental dynamics model from the first model module; 4.2 The first trajectory module uses an environmental dynamics model to estimate the uncertainty of scene state and action commands in T quadruplets of any class of dataset in the D4RL dataset, where T = 1000 × L. The method is as follows: 4.2.1 Let the scene state and action commands in the T quadruples of the D4RL dataset be represented by... , ..., express; 4.2.2 Initialize t=1; 4.2.3 Calculation Uncertainty assessment , Indicates to Calculate the L2 norm of the output result. This represents taking the maximum value among the L2 norms of the variances of the outputs of N deep probabilistic neural networks; 4.2.4 The first trajectory module uses any deep probabilistic neural network pair from the environmental dynamics model. Make a prediction and obtain First reward function ; 4.2.5 The first trajectory module calculates the t-th optimistic reward. ,in ; 4.2.6 The first trajectory module is based on optimistic reward. Construct an O-MDP, which is represented by a quintuple: ,in Optimistic Rewards Composition of optimistic reward space 4.2.7 Let t = t + 1. If t ≤ T, go to 4.2.

3. If t > T, it means that the O-MDP construction was completed using T quadruples. Go to step 5. Fifth, the second trajectory module receives the environmental dynamics model from the first model module, uses the environmental dynamics model to estimate the uncertainty of the trajectory in the D4RL dataset, and constructs the P-MDP. 5.1 The second trajectory module receives the environmental dynamics model from the first model module; 5.2 The second trajectory module uses an environmental dynamics model to estimate the uncertainty of scene states and action commands in T quadruplets of the same class of dataset selected in step 4.2 in the D4RL dataset. The method is as follows: 5.2.1 Let the scene state and action commands in the T quadruples of the D4RL dataset be represented by... , ..., express; 5.2.2 Initialize t=1; 5.2.3 Calculate using the same method as 4.2.

3. Uncertainty assessment ; 5.2.4 The second trajectory module uses any deep probabilistic neural network pair from the environmental dynamics model. Make a prediction and obtain reward function ; 5.2.5 The second trajectory module calculates the t-th pessimistic reward. ,in ; 5.2.6 The second trajectory module is based on pessimistic reward. Construct a P-MDP, which is represented by a quintuple: ,in Viewing Rewards Composition View the reward space; 5.2.7 Let t = t + 1. If t ≤ T, go to 5.2.

3. If t > T, it means that the P-MDP construction of T quadruplets has been completed. Go to step 6. The sixth step is to train the first strategy module under O-MDP and the second strategy module under P-MDP, and train the two strategy modules alternately to obtain the robot search and rescue strategy. The method is: 6.1 Initialize the training epoch number as 1; the maximum training epoch number as I = 1000; initialize the training batch B, where B is a positive integer; initialize the weight parameters of the first policy network in the first policy module and the second policy network in the second policy module to... Random numbers between; 6.2 The first policy module is trained as follows: 6.2.1 Initialize m=1; 6.2.2 The first strategy module randomly selects the search and rescue scenario state at the m-th time point from any four-tuple in any offline trajectory of the D4RL dataset, which belongs to the same category as in step 4.

2. 6.2.3 The first strategy module is based on Generate action instructions ; 6.2.4 The first strategy module is based on the environmental dynamics model. Calculate the optimistic reward at the m-th time point. And predict and execute action orders The search and rescue scenario state at the (m+1)th time point ; 6.2.5 The first strategy module will use the quadruple < , , >Sent to the first trajectory module, the first trajectory module will send the quadruple < , , Save as an optimistic trajectory; 6.2.6 Let m = m + 1. If m ≤ M1, go to 6.2.

2. If m > M1, it means that M1 optimistic trajectories have been generated. Go to 6.2.

7. 6.2.7 The first strategy module randomly samples B optimistic trajectories from the first trajectory module, trains the first strategy module using the SAC algorithm, and updates the network parameters in the first strategy module; 6.3 The second policy module is trained as follows: 6.3.1 Initialize variable mm = 1; 6.3.2 The second strategy module randomly selects the search and rescue scenario state at the mm-th time point from any quadruple in any offline trajectory of the D4RL dataset, which belongs to the same category as in step 4.

2. 6.3.3 The second strategy module is based on Generate action instructions ; 6.3.4 The second strategy module is based on the environmental dynamics model. Calculate the pessimistic reward at the mm-th time point. And predict and execute action orders ; 6.3.5 The second strategy module will use the quadruple < , , >Sent to the second trajectory module, the second trajectory module will send the quadruple < , , Save as an optimistic trajectory; 6.3.6 Let mm = mm + 1. If mm ≤ M2, go to 6.3.

2. If mm > M2, it means that M2 pessimistic trajectories have been generated. Go to 6.3.

7. 6.3.7 The second policy module randomly samples B / 2 pessimistic trajectories from the second trajectory module, combines them with B / 2 offline trajectories from the offline dataset in the first storage module, and uses the TD3+BC algorithm to train the second policy module and update the network parameters of the second policy network. 6.4 Set epoch = epoch + 1; if epoch equals the set maximum number of training rounds I, then proceed to step 6.5; otherwise, go to step 6.

2. 6.5 The second strategy module generates a .data file from the network parameters in the second strategy network using the TensorFlow deep learning framework. The .data file is the search and rescue strategy. Step 7: The second strategy module sends the .data parameter format file to the robot computing module, and the third strategy network in the computing module loads the .data parameter format file. Step 8: Deploy the robot search and rescue system built in Step 1 in a real, unknown, open, and complex environment. The robot nodes execute search and rescue tasks according to the search and rescue strategy, i.e., the .data parameter format file. The method is as follows: 8.1 According to the requirements of the multi-target sampling task, multiple target points that the robot system needs to sample are set. The detection module of the robot node detects the coordinates of all search and rescue target points and stores the coordinates in the target point list; the calculation module of the robot node selects the target point to go to in order from the target point list. 8.2 Initialize variable k = 0; 8.3 The robot node's detection module detects the state of the k-th task scenario. ,Will The computation module sent to the robot node; 8.4 The robot node's computation module will receive data from the detection module. The state-target tuple is composed of the target point locations selected from the target point list and the target point locations. The state-target tuple is ( (Target point locations in the target point list); The third strategy network of the 8.5 calculation module loads the .data parameter format file, makes action instruction decisions based on the state-target tuple, and obtains the k-th action instruction. ,Will Send to the motion module; 8.6 Motion Module Execution of Robot Nodes Move to the target point position in the target point list in the state target binary; 8.7 If the robot node executes Once the robot node reaches within d meters of a target point in the target point list (d is a positive number), target sampling is performed, and one target point within d meters is deleted from the target point list. Then, proceed to step 8.

8. If the target point is not within d meters, let k = k + 1, go to step 8.3, and continue moving towards the target point; 8.8 The robot node determines whether there are still target points in the target point list. If there are still target points that have not been sampled and the robot is still moving, proceed to step 8.2; if there are no target points that have not been sampled in the target point list or the robot stops moving, proceed to step 9. Step 9: If the target point list is empty, the task is considered successful and ends; if the target point list is not empty, it means that the robot cannot complete the search and rescue task in this complex terrain, the task is considered a failure and ends.

2. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The server node must be running Ubuntu version 16.04 or higher, Ubuntu Mate must be running Ubuntu version 16.04 or higher, and the TensorFlow deep learning framework must be running version 1.10.0 or higher.

3. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The robot nodes include the ground robot nodes Boston Dynamics' robot dog, Tencent Robotics X Lab's Jamoca robot dog, and the centaur robot HalfCheetah; the detection modules include odometry, speedometer, scanning radar, infrared camera, and depth camera; The value range is 0.3 to 1.

4. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The first storage module has an available space of more than 1GB, the value of D is in the range of 1000~1200, and the value of L is in the range of 800~1000.

5. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The number N of deep probabilistic neural networks created by the first model module ranges from 5 to 7.

6. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The first policy module consists of a first policy network created using the TensorFlow deep learning framework. The first policy network consists of three fully connected layers, each containing 256 neurons. The second policy module consists of a second policy network created using the TensorFlow deep learning framework. The second policy network is the same as the first policy network of the first policy module. The computation module includes a third policy network created using the TensorFlow deep learning framework. The third policy network is the same as the first policy network of the first policy module.

7. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The term "optimistic" means that for all uncertainties in the prediction process of the environmental dynamics model, the best-case scenario, i.e., the maximum reward, is taken; the term "pessimistic" means that for all uncertainties in the prediction, the worst-case scenario, i.e., the minimum reward, is taken.

8. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The available space of the first trajectory module is more than 200MB, and the value range of M1 is 5000 to 10000; the available space of the second trajectory module is more than 200MB, and the value range of the pessimistic trajectory M2 is 5000 to 10000.

9. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... Step 4.2.5 ; Step 5.2.5 .

10. The robot search and rescue method based on offline reinforcement learning as described in claim 1, characterized in that... The value range of B in step 6.1 is 32~64; the value range of d in step 8.7 is 1~3.

Images