A system and method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces

Abstract

The present invention discloses a learning and training method and system for escape strategies in dangerous scenarios on extraterrestrial surfaces. The method utilizes a high-fidelity digital simulation environment and a distributed architecture to efficiently train a rover based on a reinforcement learning algorithm, and verifies the effectiveness of the control strategy, so as to obtain a rich sample library of effective escape control strategies, thereby providing technical support and ground test means for ensuring the safety of rover detection missions.

Description

Technical Field

[0001] The present invention relates to a system and method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces, belonging to the technical field of deep space exploration. Background Art

[0002] In deep space exploration missions, patrol exploration after landing is an important way to carry out in-depth scientific exploration activities of extraterrestrial bodies. However, the terrain environment of extraterrestrial surfaces is complex and unknown, and the illumination difference is large. The detector can only obtain the geometric characteristics of the target by visual measurement, and the perception and recognition ability of dangerous terrain such as sand and floating pits is weak. Once the wheels sink, it will seriously affect the safety of the rover. For example, the Spirit Mars rover of the United States had to terminate its mission because its wheels sank into the fire soil and could not get out. Therefore, the ground should fully identify risks and study escape control strategies in dangerous scenarios.

[0003] Using physical experiments to carry out learning and training of escape control strategies is costly, risky, and inefficient. It is difficult to obtain force measurement information, and it cannot meet the needs of intelligent learning algorithms for large samples and repeated trial and error learning. Therefore, it is urgent to establish a digital simulation environment and a distributed learning and training platform to obtain an effective escape control strategy sample library to provide technical support and ground test means for successful escape in dangerous situations, but there are no relevant reports. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned defects and provide a method and system for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces. By utilizing a high-fidelity digital simulation environment and a distributed architecture, the rover is efficiently trained based on a reinforcement learning algorithm, and the effectiveness of the control strategy is verified, so as to obtain a rich sample library of effective escape control strategies, thereby providing technical support and ground test means for ensuring the safety of rover detection missions.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] A method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces, comprising the following steps:

[0007] (1) constructing a digital model of the extraterrestrial surface topography and environment and a digital model of the rover, wherein the digital model of the extraterrestrial surface topography and environment includes topography features and environmental parameters, and the digital model of the rover includes a rover dynamics model and state parameters;

[0008] (2) formulating training subjects, wherein the training subjects include the extraterrestrial surface topographic environment corresponding to the rover escape scenario, the rover dynamic state, the rover operation rules and the escape success or failure conditions; wherein the operation rules are some constraints for generating control instructions during the training of the escape control strategy neural network, and the success or failure conditions refer to the final evaluation criteria for the training results, i.e., whether the escape is successful;

[0009] (3) setting initial values ​​of environmental parameters in the digital model of the extraterrestrial surface topography environment and dynamic state parameters in the digital model of the rover according to the training subjects, and generating a simulation test environment corresponding to the training subjects;

[0010] (4) constructing a neural network model for escape control decision making, and using the data generated by the simulation test environment to train it in the training cloud platform to obtain a trained neural network model for escape control decision making;

[0011] (5) The trained escape control decision neural network model is transferred to an external simulation platform or actual application platform to perform real-time verification and secondary optimization of the escape control strategy.

[0012] Furthermore, in the step (3), the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the dynamic state parameters in the digital model of the rover are set to n groups of different values ​​according to the training subjects, so as to generate n groups of simulation test environments corresponding to the training subjects;

[0013] In the step (4), a neural network model for escape control decision making is constructed, and the model is trained in parallel in n groups of simulation test environments to obtain a trained neural network model for escape control decision making;

[0014] Said n>1.

[0015] Furthermore, in step (4), the method for training the model in a simulation test environment comprises the following steps:

[0016] (41) At the current time t, the escape control decision neural network model receives samples, makes decisions based on the samples and outputs control instructions to the patroller, and calculates the loss function value based on the decision result to optimize the model parameters; that is, in the training mode, the decision module (i.e., the escape control decision neural network model) in the intelligent body model (i.e., the control decision module on the patroller) will be distributed in multiple simulation environments for simulation control and decision-making. At the same time, the loss function module in the intelligent body model will count the loss values ​​(reward) generated in multiple simulation environments, and optimize the relevant parameters of the decision module (i.e., the escape control decision neural network model) on the training cloud platform through the distributed optimizer, and the optimized parameters will be updated to the decision modules of each simulation environment;

[0017] The samples include environmental parameters of the extraterrestrial surface topography environment at the current time t in the simulation test environment, dynamic state parameters of the rover at the current time t, control instructions output to the rover by the escape control decision neural network model at the last time t-1, and loss function values ​​in the escape control decision neural network model at the last time;

[0018] The decision result is determined according to the escape success or failure conditions in the training subject;

[0019] The constraint conditions for the escape control decision neural network model to make decisions based on samples include constraint conditions generated according to the patrol device operation rules;

[0020] (42) At the next time t+1, the patroller executes the action according to the control instruction;

[0021] Let t+1=t, and return to step (41) until the loss function value converges to a predetermined value.

[0022] Furthermore, in step (1), the landform features in the digital model of the extraterrestrial surface landform environment include one or more combinations of surface sand, bedrock or soft soil, and the environmental parameters include one or more combinations of gravity, ground hardness coefficient, and acceleration ratio.

[0023] Furthermore, in step (1), the patrol vehicle dynamic state parameters in the patrol vehicle digital model include one or more combinations of patrol vehicle position, speed, steering, wheel angular velocity, wheel sinking amount or suspension angle.

[0024] Furthermore, in step (4), the inference decision neural network model is a single scalable network structure or a combined network structure.

[0025] Furthermore, in step (4), the single scalable network structure includes a fully connected network, one or more combinations of LSTM, FCN or CNN, and the combined network structure includes one or more combinations of Transformer, PointerNet, ResNet, or MultilRNN.

[0026] Furthermore, in step (4), a distributed architecture is used to uniformly integrate and schedule CPU and GPU computing resources during the training of the model in a simulation test environment.

[0027] Furthermore, in the step (41), the method for optimizing the model parameters includes modifying the escape control decision neural network model components, core algorithms or evolutionary methods; in the step (41), the control instructions are used to control one or more combinations of the patroller's target speed, target position or motor drive torque.

[0028] Furthermore, in the step (5), the trained escape control decision neural network model is transferred to an external simulation platform or an actual application platform based on a unified simulation environment communication interface, ensuring the input and output variable types, formats, and communication protocols, that is, ensuring that the external simulation platform or the actual application platform is consistent with the framework of the training process and can be directly migrated.

[0029] A learning and training system for escaping strategies in dangerous scenarios on extraterrestrial surfaces, used to implement the above-mentioned learning and training method for escaping strategies in dangerous scenarios on extraterrestrial surfaces, comprising a digital simulation environment construction module, a training subject configuration module, an escaping control decision neural network model design module, a resource configuration and computing power scheduling module and a training cloud platform;

[0030] The digital simulation environment construction module is used to construct a digital model of the extraterrestrial surface topography environment and a digital model of the rover, set the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the dynamic state parameters in the digital model of the rover according to the training subjects, and generate a simulation test environment corresponding to the training subjects;

[0031] The training subject configuration module is used to formulate training subjects, wherein the training subjects include the extraterrestrial surface topographic environment corresponding to the rover escape scenario, the rover dynamic state, the rover operation rules and the escape success or failure conditions;

[0032] The escape control decision neural network model design module is used to establish an escape control decision neural network model;

[0033] The training cloud platform is used to train the escape control decision neural network model using a simulation test environment to obtain a trained model;

[0034] The resource allocation and computing power scheduling module is used to uniformly integrate and schedule CPU and GPU computing resources during the training process of the escape control decision neural network model.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The present invention innovatively proposes a learning and training system and method for escaping dangerous scenarios on extraterrestrial surfaces. Based on a high-fidelity digital simulation environment, it can accurately simulate the movement of the rover in a variety of complex environments, provide a large number of effective data samples for learning and training, reduce experimental costs and risks, and improve training efficiency.

[0037] (2) The existing technology uses a single machine iteration method for learning and training, which has a small sample size, low efficiency, and poor robustness of training results. The training platform based on the distributed architecture of the present invention can start multiple simulation environments at the same time, and reasonably allocate heterogeneous computing resources to carry out learning and training in parallel. It can adapt to both fine-grained resource scheduling requirements and large-scale resource management requirements, so that the training platform can truly realize the ability of elastic computing;

[0038] (3) The present invention can flexibly configure model components during the neural network training process, and adjust the core algorithm, evolutionary method, etc. according to the training effect to ensure that the output intelligence level is improved and a robust and stable control strategy is obtained;

[0039] (4) Based on a unified simulation environment communication interface, the decision model trained by this method can be quickly migrated to an external simulation platform or an actual application platform, improving the efficiency of virtual-real migration and facilitating rapid verification and secondary optimization of strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a method for learning and training an escape strategy for dangerous scenarios on extraterrestrial surfaces according to the present invention;

[0041] Figure 2 This is a flowchart of the simulation process of training the neural network model for escape control decision making of the present invention. DETAILED DESCRIPTION

[0042] The following detailed description of the present invention will make the features and advantages of the present invention more clear and explicit.

[0043] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0044] The present invention provides a learning and training method and system for escape strategies in dangerous scenarios on extraterrestrial planets, constructs large-scale learning samples to improve the efficiency of control strategy training, enhances the environmental adaptability and autonomous escape capability of patrol vehicles, and after completing online enhanced learning training, quickly migrates from a digital environment to a physical environment to perform secondary optimization and verification of the control strategy. Through learning and training in different dangerous scenarios on extraterrestrial planets, an effective escape control strategy sample library is obtained, providing technical support and ground test means for successful escape in dangerous situations.

[0045] like Figure 1 The present invention provides a method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces, comprising the following steps:

[0046] Step 1: Establish a digital simulation environment: construct a complex extraterrestrial surface environment and a digital model of the rover; the complex extraterrestrial surface environment includes an extraterrestrial surface environment with one or more characteristics such as surface sand, bedrock, soft soil, etc., and also includes one or more environmental parameters such as gravity, ground hardness coefficient, acceleration ratio, etc. The digital model of the rover includes one or more state parameters such as vehicle position, speed, steering, wheel angular velocity, wheel sinking, suspension angle, etc., and a rover dynamics model;

[0047] The complex terrain environment of the extraterrestrial surface and the digital model of the rover together constitute the dangerous scenes of the extraterrestrial surface and the working conditions of the rover walking and sinking. Based on the rigid body / flexible physics engine model, dynamics solution, collision detection and constraint solution are performed to simulate the movement state and wheel-soil interaction of the rover in the process of moving according to the control instructions in the Martian scene. It has high simulation accuracy while meeting the real-time requirements, and provides simulation models, verification environment and data foundation for intelligent control strategy learning and training;

[0048] Step 2: Configure training subjects: The system integrates a training subject library to formulate training subjects suitable for patrol vehicles to escape from difficulties. Training subjects are to design simulation scenarios based on different dangerous situations, clarify operation rules and winning and losing conditions, and finally abstract them into a deterministic closed problem.

[0049] According to the training subjects, the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the initial values ​​of the dynamic state parameters of the rover in the digital model of the rover are set to generate a simulation test environment corresponding to the training subjects;

[0050] Each training subject corresponds to one or more environmental parameters such as gravity, ground hardness coefficient or acceleration ratio, and one or more state parameters such as vehicle position, speed, steering, wheel angular velocity, wheel sinking amount, suspension angle, etc.

[0051] Step 3: Construct an escape control decision neural network model, and use the escape control decision neural network model design module to design an escape control decision neural network model. The escape control decision neural network model receives input data from the digital simulation environment, and after extracting, expressing, reasoning and calculating the environmental information, it outputs the control instructions of the patroller;

[0052] The escape control decision neural network model outputs control instructions to the simulation environment (i.e., the patrol vehicle), including one or more of the control instructions such as the patrol vehicle's target speed, target position, and motor drive torque;

[0053] The main components involved in the construction of the network model are a single scalable network structure or a combined network structure, which mainly extracts high-level information from the images and other state features received by the patroller, provides sufficient reasoning and decision-making capacity for the neural network structure, and stabilizes the learning effect; among them, the single scalable network structure includes: fully linked network, LSTM (long short-term memory network), FCN (fully convolutional network), CNN (convolutional network), etc.; the combined network structure includes: Transformer (feature conversion network), PointerNet (combat unit selection network), ResNet (image classification network) and MultilRNN (multi-layer recurrent neural network), etc.;

[0054] Step 4: Resource configuration and computing power scheduling: Use the resource configuration and computing power scheduling module to allocate computing power resources, namely CPU and GPU computing resources, for model training tasks. To improve training efficiency, a distributed architecture is used to uniformly integrate and flexibly schedule computing resources, supporting concurrent operation of multiple simulation environments;

[0055] Step 5: Escape simulation training: Figure 2 , create a simulation training task. In the training mode, multiple digital simulation environments are created through the resource configuration and computing power scheduling modules, and the escape control decision neural network model is distributed in each digital simulation environment for control and decision-making. At the same time, the loss function will count the loss values ​​generated in multiple digital simulation environments, and optimize the relevant parameters of the neural network model on the training cloud platform through the distributed optimizer. The optimized parameters are then updated to each neural network model. The distributed execution of multiple digital simulation environments not only speeds up the generation of training data and improves training efficiency, but also increases the diversity of a batch of training data, which helps to improve the stability of training. During training, the training strategy can be adjusted by modifying the network model components, core algorithms, evolutionary methods, etc. to ensure that the output intelligence level of the neural network continues to grow;

[0056] Step 6: Simulation sample generation: During the training process, the sample generation and data management module is used to manage the training sample data generated in step 5 and use it for subsequent training. The training sample data specifically includes four types of information: simulation environment state information (i.e., rover dynamic state parameters), environmental parameters of the extraterrestrial surface topography environment, rover control instructions, and loss function values;

[0057] Step 7: Intelligent escape reasoning: Under the condition of ensuring the consistency of environmental parameters, the escape control decision neural network model after training will be transferred and deployed from the training platform to an external simulation platform or actual application platform for real-time verification and secondary optimization of the escape control strategy.

[0058] The present invention provides an extraterrestrial dangerous scenario escape strategy learning and training system, which is used to implement the above-mentioned extraterrestrial dangerous scenario escape strategy learning and training method, including a digital simulation environment construction module, a training subject configuration module, an escape control decision neural network model design module, a resource configuration and computing power scheduling module, a simulation sample generation module and a training cloud platform;

[0059] The digital simulation environment construction module is used to construct a digital model of the extraterrestrial surface topography environment and a digital model of the rover, and to set the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the initial values ​​of the rover dynamic state parameters in the digital model of the rover according to the training subjects, so as to generate a simulation test environment corresponding to the training subjects; the simulation environment is provided with a simulation environment interface based on a remote call framework, which is used to set the numerical values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the initial values ​​of the rover dynamic state parameters in the digital model of the rover according to the training subjects, to perform simulation environment control and parameter control, to generate a simulation test environment corresponding to the training subjects, and is also used for data communication between the simulation environment and the escape control decision neural network model, so that the trained intelligent agent has control authority over the simulation environment;

[0060] The training subject configuration module is used to formulate training subjects, wherein the training subjects include the extraterrestrial surface topographic environment corresponding to the rover escape scenario, the rover dynamic state, the rover operation rules and the escape success or failure conditions;

[0061] The escape control decision neural network model design module is used to establish an escape control decision neural network model;

[0062] The training cloud platform is used to train the escape control decision neural network model using the simulation test environment to obtain a trained model;

[0063] The resource allocation and computing power scheduling module is used to uniformly integrate and schedule CPU and GPU computing resources during the training process of the escape control decision neural network model;

[0064] The simulation sample generation module is used to store samples of the input escape control decision neural network model, and the samples include simulation environment state information (i.e., rover dynamic state parameters), environmental parameters of the extraterrestrial surface topographic environment, rover control instructions and loss function values.

[0065] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

[0066] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces, characterized in that: The following steps are involved: (1) constructing a digital model of the extraterrestrial surface topography and environment and a digital model of the rover, wherein the digital model of the extraterrestrial surface topography and environment includes topography features and environmental parameters, and the digital model of the rover includes a rover dynamics model and state parameters; (2) Formulate training subjects, including the extraterrestrial surface topography environment, rover dynamic state, rover operation rules and escape success or failure conditions corresponding to the rover escape scenario; (3) setting initial values ​​of environmental parameters in the digital model of the extraterrestrial surface topography environment and dynamic state parameters in the digital model of the rover according to the training subjects, and generating a simulation test environment corresponding to the training subjects; (4) constructing a neural network model for escape control decision making, and training the model using a simulation test environment to obtain a trained neural network model for escape control decision making; (5) Transfer the trained escape control decision neural network model to an external simulation platform or actual application platform to conduct real-time verification and secondary optimization of the escape control strategy; In the step (4), the method for training the model in a simulation test environment comprises the following steps: (41) At the current time t, the escape control decision neural network model receives samples, makes decisions based on the samples and outputs control instructions to the patroller, and calculates the loss function value based on the decision result to optimize the model parameters; The samples include environmental parameters of the extraterrestrial surface topography environment at the current time t in the simulation test environment, dynamic state parameters of the rover at the current time t, control instructions output to the rover by the escape control decision neural network model at the last time t-1, and loss function values ​​in the escape control decision neural network model at the last time; The decision result is determined according to the escape success or failure conditions in the training subject; The constraints for the escape control decision neural network model to make decisions based on samples include patrol vehicle operation rules; (42) At the next time t+1, the patroller executes the action according to the control instruction; Set t+1=t, and return to step (41) until the loss function value converges to a predetermined value; In the step (4), a distributed architecture is used to uniformly integrate and schedule CPU and GPU computing resources during the training of the model in the simulation test environment; In the step (41), the method for optimizing the model parameters includes modifying the escape control decision neural network model component, core algorithm or evolution method; in the step (41), the control instruction is used to control one or more combinations of the target speed, target position or motor drive torque of the patroller.

2. The method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 1, characterized in that: In the step (3), the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the dynamic state parameters in the digital model of the rover are set to n groups of different values ​​according to the training subjects, so as to generate n groups of simulation test environments corresponding to the training subjects; In the step (4), a neural network model for escape control decision making is constructed, and the model is trained in parallel in n groups of simulation test environments to obtain a trained neural network model for escape control decision making; Said n>1.

3. A method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 1 or 2, characterized in that: In the step (1), the landform features in the digital model of the extraterrestrial surface landform environment include one or more combinations of surface sand, bedrock or soft soil, and the environmental parameters include one or more combinations of gravity, ground hardness coefficient, and acceleration ratio.

4. A method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 1 or 2, characterized in that: In the step (1), the patrol vehicle dynamic state parameters in the patrol vehicle digital model include one or more combinations of patrol vehicle position, speed, steering, wheel angular velocity, wheel sinking amount or suspension angle.

5. A method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 1 or 2, characterized in that: In the step (4), the escape control decision neural network model is a single scalable network structure or a combined network structure.

6. The method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 5, characterized in that: In the step (4), the single scalable network structure includes one or more combinations of a fully connected network, LSTM, FCN or CNN, and the combined network structure includes one or more combinations of Transformer, PointerNet, ResNet, or MultilRNN.

7. The method for learning and training escape strategies for dangerous scenarios on extraterrestrial surfaces according to claim 1, characterized in that: In the step (5), the trained escape control decision neural network model is transferred to an external simulation platform or an actual application platform based on a unified simulation environment communication interface.

8. A learning and training system for escaping strategies in dangerous extraterrestrial scenarios, characterized in that: A method for learning and training an escape strategy for dangerous scenarios on extraterrestrial surfaces, used to implement any one of claims 1-7, comprising a digital simulation environment construction module, a training subject configuration module, an escape control decision neural network model design module, a resource configuration and computing power scheduling module, and a training cloud platform; The digital simulation environment construction module is used to construct a digital model of the extraterrestrial surface topography environment and a digital model of the rover, set the initial values ​​of the environmental parameters in the digital model of the extraterrestrial surface topography environment and the dynamic state parameters in the digital model of the rover according to the training subjects, and generate a simulation test environment corresponding to the training subjects; The training subject configuration module is used to formulate training subjects, wherein the training subjects include the extraterrestrial surface topographic environment corresponding to the rover escape scenario, the rover dynamic state, the rover operation rules and the escape success or failure conditions; The escape control decision neural network model design module is used to establish an escape control decision neural network model; The training cloud platform is used to train the escape control decision neural network model using a simulation test environment to obtain a trained model; The resource allocation and computing power scheduling module is used to uniformly integrate and schedule CPU and GPU computing resources during the training process of the escape control decision neural network model; In the step (4), the method for training the model in a simulation test environment comprises the following steps: (41) At the current time t, the escape control decision neural network model receives samples, makes decisions based on the samples and outputs control instructions to the patroller, and calculates the loss function value based on the decision result to optimize the model parameters; The samples include environmental parameters of the extraterrestrial surface topography environment at the current time t in the simulation test environment, dynamic state parameters of the rover at the current time t, control instructions output to the rover by the escape control decision neural network model at the last time t-1, and loss function values ​​in the escape control decision neural network model at the last time; The decision result is determined according to the escape success or failure conditions in the training subject; The constraints for the escape control decision neural network model to make decisions based on samples include patrol vehicle operation rules; (42) At the next time t+1, the patroller executes the action according to the control instruction; Set t+1=t, and return to step (41) until the loss function value converges to a predetermined value; In the step (4), a distributed architecture is used to uniformly integrate and schedule CPU and GPU computing resources during the training of the model in the simulation test environment; In the step (41), the method for optimizing the model parameters includes modifying the escape control decision neural network model component, core algorithm or evolution method; in the step (41), the control instruction is used to control one or more combinations of the target speed, target position or motor drive torque of the patroller.

Images