A deep space cruise path planning method and system
By generating and uploading the policy network and value network to the deep space rover from the ground control center, and combining imitation learning and reinforcement learning, efficient path planning for deep space exploration missions was achieved, solving the problem of insufficient decision-making in high-latency and high-dynamic environments and improving the system's intelligence level.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING AEROSPACE CONTROL CENT
- Filing Date
- 2024-05-06
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies have failed to enable efficient implementation of deep space exploration missions under high latency and high dynamic conditions, and lack a full-process intelligent system architecture design, resulting in insufficient intelligent enhancement of spacecraft and ground systems.
By leveraging expert experience at the ground control center to generate initial parameters for the policy network and value network, these parameters are uploaded to the deep space rover for path planning. Furthermore, by combining imitation learning and reinforcement learning, the policy network and value network are updated in real time, enabling the spacecraft to make autonomous decisions and optimize its path.
By integrating the intelligence of spacecraft and ground systems, it overcomes the problem of state incoordination in high-latency and high-dynamic environments, and improves the decision-making ability and planning efficiency of deep space exploration missions.
Smart Images
Figure CN118502421B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of aerospace technology, and in particular to a method and system for deep space exploration path planning. Background Technology
[0002] The Moon and Mars are hotspots for human deep space exploration. The basic process of a rover scientific exploration mission is as follows: The rover perceives the surrounding terrain environment through its onboard sensors (such as a binocular navigation terrain camera); after the ground acquires camera images, it performs 3D reconstruction, determines the target point on the map based on the surrounding environment and the exploration target, plans the movement path, and generates control parameters; then the control parameters are uploaded to the rover, which moves to the target point according to the control parameters, perceives again, and repeats the process, gradually approaching the exploration point, and finally carrying out the exploration, transmitting scientific exploration data, and achieving the purpose of the exploration.
[0003] However, deep space exploration faces challenges such as vast distances and communication difficulties. Mars is at its closest point to Earth at 50 million kilometers, and at its farthest, it exceeds 400 million kilometers. Lunar rovers often require relay satellite support, limiting their tracking and control capabilities. These factors create significant difficulties for controlling deep space rovers, rendering traditional deep space exploration reliant on a large Earth-space loop. Artificial intelligence technology can improve the intelligence level of deep space exploration, enabling it to efficiently conduct scientific exploration missions even in environments with high latency, weak tracking and control, and high dynamics. Existing technologies include:
[0004] 1) In the invention patent with publication number "CN114019948A" and subject title "An Efficient Autonomous Operation Method for Rover Detection Missions," a parameterized instruction template is formed by creating the instruction sequence corresponding to typical rover detection missions; then, a task scheduling model corresponding to each detection mission is formed and stored in the rover's data management computer. This parameterized instruction template reduces the amount of data transmitted for instructions and improves the efficiency of ground instruction transmission.
[0005] 2) In the invention patent with patent publication number "CN114399225A" and subject title "A Deep Space Probe Mission Planning Method Based on Q-Learning", the state space itself and the state transition expressed by the structured expression method are input into the reinforcement learning training process in the form of flattened numerical, vector and matrix forms identified by the Q-Learning method. This completes the training of the planning by the Q-Learning method, which can improve the adaptability to sudden events and improve the planning and transfer efficiency of asteroid exploration missions.
[0006] 3) In the invention patent with patent publication number "CN114139776A" and subject name "A Deep Space Exploration Mission Planning Method and System Based on Cognitive Map", the meta-actions and states of the asteroid attachment mission of the deep space exploration spacecraft can be stored as knowledge. At the same time, the knowledge also has the advantage of rapid retrieval, which can lay the foundation for rapid iterative planning of subsequent steps.
[0007] 4) In the invention patent with patent publication number “CN114676471A” and subject title “A method and system for Mars rover mission planning”, the reinforcement learning method is used, which can simplify the modeling process of traditional rule-based mission planning methods and quickly search for solutions that meet the constraints.
[0008] However, current technologies focus only on optimizing a single aspect of deep space rover exploration systems, failing to design an intelligent architecture from the perspective of the entire "sensing-transmission-decision" process. Specifically:
[0009] 1) The invention patent with publication number "CN114019948A" and subject name "an efficient autonomous operation method for a rover detection mission" only focuses on reducing the amount of data transmitted and has a certain effect on the delay of decision-making in the process, but it lacks relevant mechanisms to ensure the quality of decision-making.
[0010] 2) The invention patent with publication number "CN114399225A" and subject title "A Deep Space Probe Mission Planning Method Based on Q-Learning" can improve the objectivity of decision-making, but the simulation environment on which the Q-learning algorithm is trained is difficult to model.
[0011] 3) The invention patent with publication number "CN114139776A" and subject name "A Deep Space Exploration Mission Planning Method and System Based on Cognitive Graph" plans deep space exploration missions based on cognitive graphs. However, the knowledge graph is static and cannot be updated with the status information collected by the rover. This can easily lead to a disconnect between the knowledge graph and the real Martian environment, resulting in a gradual increase in decision-making errors.
[0012] 4) The invention patent with publication number “CN114676471A” and subject title “A method and system for Mars rover mission planning” is based solely on a ground system. The mission planning process is mainly completed on the ground and cannot fully utilize the intelligent planning capabilities of the spacecraft.
[0013] Enhancing the capabilities and intelligence of deep space rover exploration systems is a systems engineering project, where any single component impacts the overall system performance. Currently, the intelligence of deep space rover exploration systems primarily focuses on the intelligence of individual components, either the spacecraft or the ground system. At the spacecraft level, the ground system mainly plays a monitoring and judgment role, but the information available to the spacecraft is less comprehensive than that from the ground, making it prone to getting trapped in local optima. Ground system intelligence typically involves the ground system using intelligent methods for decision-making and planning, aiming to improve the efficiency of the ground system alone, while the spacecraft merely receives control parameters and implements control, making it difficult to achieve intelligent operation across the entire space-ground loop to simultaneously improve efficiency.
[0014] In summary, although existing technologies have improved the planning and design of deep space exploration systems from different individual aspects, they have not carried out integrated system design, making it difficult to fundamentally ensure the efficient implementation of deep space surface exploration missions under high latency and high dynamic conditions. Summary of the Invention
[0015] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically by providing a deep space exploration path planning method and system, as detailed below:
[0016] 1) In a first aspect, the present invention provides a deep space exploration path planning method, the specific technical solution of which is as follows:
[0017] S1. Based on expert experience, the ground control center obtains the initial parameters of the strategy network and the initial parameters of the value network, and uploads them to the deep space rover.
[0018] S2. The deep space rover inputs the acquired perception data into the policy network with initial parameters of the policy network and the value network with initial parameters of the value network to perform path planning and obtain the planned path.
[0019] S3. The deep space rover moves along the planned path, acquires actual movement data during the movement, and updates the initial parameters of the policy network and the value network based on the actual movement data.
[0020] The beneficial effects of the deep space exploration path planning method provided by this invention are as follows:
[0021] It can leverage the capabilities of a large-scale space-ground system, integrating the intelligence of spacecraft agents, ground planning systems, and expert experience. Specifically, experts can deduce and reason based on spacecraft design and theoretical knowledge, completing tasks under conditions of small samples and no samples. Existing intelligent planning methods based on big data learning have made significant progress and are ready to be transferred to spacecraft. Spacecraft can collect sufficient data during hundreds or thousands of mission executions to train their decision-making capabilities. Furthermore, deep-space spacecraft can select more suitable methods in real time based on the environment, overcoming the time delay problem of ground planning. The ground control system can integrate macroscopic and microscopic information, combining expert intelligence and data collected by the spacecraft to achieve a combination of expert intelligence and data statistics. This overcomes the contradictions caused by delayed communication and inconsistent states and commands due to changing environments.
[0022] Based on the above scheme, the deep space exploration path planning method of the present invention can be further improved as follows.
[0023] Furthermore, it also includes: the deep space rover uses the updated policy network initial parameters as the policy network initial parameters and the updated value network initial parameters as the value network initial parameters, and then returns to execute S2.
[0024] Furthermore, it also includes: the deep space rover sending updated initial parameters of the policy network and updated initial parameters of the value network to the ground control center.
[0025] Furthermore, based on expert experience, the ground control center obtained the initial parameters of the policy network and the initial parameters of the value network, including:
[0026] The ground control center uses the behavior cloning method in imitation learning and, based on expert experience, obtains the initial parameters of the policy network and the initial parameters of the value network.
[0027] Furthermore, the process by which the deep space rover updates the initial parameters of the policy network and the value network based on actual movement data includes:
[0028] The path slope reward value, path roughness reward value, and path step reward value are determined based on the actual movement data.
[0029] The initial parameters of the policy network and the initial parameters of the value network are updated based on the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value, and the path step reward value.
[0030] 2) Secondly, the present invention also provides a deep space exploration path planning system, the specific technical solution of which is as follows:
[0031] Including ground control centers and deep space rover;
[0032] The ground control center is used to: obtain the initial parameters of the policy network and the initial parameters of the value network based on expert experience, and upload them to the deep space rover;
[0033] The deep space rover is used to: input the acquired perception data into the policy network with initial parameters of the policy network and the value network with initial parameters of the value network, perform path planning, and obtain the planned path;
[0034] The deep space rover is also used to: move along a planned path, acquire actual movement data during the movement, and update the initial parameters of the policy network and the value network based on the actual movement data.
[0035] Based on the above scheme, the deep space exploration path planning system of the present invention can be further improved as follows.
[0036] Furthermore, the deep space rover is also used to: repeatedly perform path planning by using the updated policy network initial parameters as policy network initial parameters and the updated value network initial parameters as value network initial parameters, and to update the policy network initial parameters and value network initial parameters.
[0037] Furthermore, the deep space rover is also used to: send the updated initial parameters of the policy network and the updated initial parameters of the value network to the ground control center.
[0038] Furthermore, the ground control center is specifically used for: utilizing the behavior cloning method in imitation learning and based on expert experience, to obtain the initial parameters of the policy network and the initial parameters of the value network.
[0039] Furthermore, deep space rovers are also specifically used for:
[0040] The path slope reward value, path roughness reward value, and path step reward value are determined based on the actual movement data.
[0041] The initial parameters of the policy network and the initial parameters of the value network are updated based on the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value, and the path step reward value.
[0042] It should be noted that the beneficial effects of the technical solution of the second aspect of the present invention and the corresponding possible implementation can be found in the above description of the technical effects of the first aspect and its corresponding possible implementation, and will not be repeated here. Attached Figure Description
[0043] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0044] Figure 1 This is a flowchart illustrating a deep space exploration path planning method according to an embodiment of the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0046] like Figure 1 As shown, an embodiment of the deep space exploration path planning method of the present invention includes the following steps:
[0047] S1. Based on expert experience, the ground control center obtains the initial parameters of the strategy network and the initial parameters of the value network, and uploads them to the deep space rover.
[0048] Among them, the ground control center uses the behavior cloning method in imitation learning and, based on expert experience, obtains the initial parameters of the policy network and the initial parameters of the value network.
[0049] Traditional expert experience is static and can only be matched and utilized by "lookup table". By imitating the behavior cloning method in learning, the policy network and value network of the patrol can be supervised and learned through a series of states and expert-suggested actions. This transforms expert experience into the initial parameters of the policy network and the initial parameters of the value network, thereby achieving the growth of task decision intelligence.
[0050] During the imitation learning phase, the deep space rover maintains the traditional sensing-movement mode of movement, and the ground relies on the sensing image data S from the deep space rover. t Experts plan the next movement path a t The imitation learning process is as follows: the policy network and the value network are trained based on the collected expert data. That is, a large amount of perceptual image data generated by experts and the matching planned movement path data are used, while some negative sample data generated by experts are added. The policy network and the corresponding value network are trained sequentially to achieve the imitation learning of the expert decision-making process.
[0051] S2. The deep space rover inputs the acquired sensing data into the policy network (configuration strategy network initial parameters) and the value network (configuration value network initial parameters) to perform path planning, obtaining a planned path. Specifically:
[0052] The deep space rover utilizes the policy network with initial parameters configured in the policy network and the value network with initial parameters configured in the value network to perform path planning, and processes the perception data S acquired by the deep space rover at the far end of the planet. t Input the policy network to obtain the planned path a t ;
[0053] S3. The deep space rover moves along a planned path, acquiring actual movement data during the process. Based on this data, it updates the initial parameters of the policy network and the value network. Specifically:
[0054] The deep space rover moves along a planned path, acquiring actual movement data during the process, including obstacle collision status, path slope, vibration (path roughness), etc. The reward value is calculated using the above data, and the initial parameters of the policy network and the value network are updated based on the actual movement data using the reward value.
[0055] Optionally, the above technical solution also includes:
[0056] S4. The deep space rover uses the updated policy network initial parameters as the policy network initial parameters and the updated value network initial parameters as the value network initial parameters, and then returns to execute S2.
[0057] Optionally, the above technical solution also includes:
[0058] The deep space rover sends the updated initial parameters of the policy network and the updated initial parameters of the value network to the ground control center.
[0059] Optionally, in S3, the process by which the deep space rover updates the initial parameters of the policy network and the initial parameters of the value network based on actual movement data includes:
[0060] S30. Determine the path slope reward value, path roughness reward value, and path step reward value based on the actual movement data.
[0061] S31. Update the initial parameters of the policy network and the initial parameters of the value network according to the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value and the path step reward value.
[0062] The present invention will now be explained and illustrated through another embodiment, including S10 to S12, specifically:
[0063] S10. Cloning of expert decision-making experience and behavior based on imitation learning:
[0064] Traditional expert experience is static and can only be used for matching through a "lookup table" approach. By employing the behavioral cloning method in imitation learning, the rover's policy and value networks can be supervised through a series of states and expert-suggested actions. This transforms expert experience into initial parameters for the policy and value networks, allowing the deep space rover to dynamically adjust and update these parameters in subsequent state feedback, thus enhancing mission decision-making intelligence. During the imitation learning phase, the rover maintains a traditional perception-movement mode, with ground control based on the rover's perceived image data S... t Experts plan the next movement path a t Imitation learning refers to training a policy network and a value network based on collected expert data. S10 includes S100 to S105, specifically:
[0065] S100, Expert Sample Data Collection:
[0066] Experts make plans based on historical information about the surrounding environment and its status, or on simulated information transmitted from deep space rovers. The dataset consists of data samples from each sensing-movement sequence: D = <(s1,a1),(s2,a2),(s3,a3),…,(s t ,a t )>. Among them, S t The state vector is extracted from the perceived image data, and the action a is... t The moving path curve is represented by a finite number of discrete curvature and arc length values.
[0067] S101. Obtain the initial policy network p using imitation learning with expert data. π (s|θ π ): Through the dataset D=<(s1,a1),(s2,a2),(s3,a3),…,(s t ,a t Training the policy network p π (s|θ π ).
[0068] In the actual environmental data or simulation data of deep space rovers, execute p π (s|θ π Obtain a new dataset
[0069] Experts in D π Mark the movement path curve 'a' of the deep space rover and aggregate them to obtain a new dataset D←D∪D π Return to continuously training the policy network p π (s|θ π ).
[0070] S102, Building a Value Network v (a,s|θ v ), using the initial policy network p π (s|θ π The policy sampling reward function is randomly generated in the environment. The reward function mainly includes:
[0071] 1) Plan the path slope and calculate the path slope bonus value r. r , represented as Where α represents the zenith angle of the path path inspector's base plate vector (perpendicular to the base plate and pointing upwards), and N represents the number of sampling points on the path;
[0072] 2) Plan the path roughness and calculate the path roughness reward value r. t , is represented as: Where z represents the sampled value of the celestial component of the position of the rover as it moves along the path. Let z represent the mean of all z, and N represent the number of sampling points on the path;
[0073] 3) Plan the path steps and calculate the path step reward value r. l If there are no stairs, the reward value r l If the value is 1, the reward decreases by 0.1 for each tier.
[0074] The final total reward value r total :r total =r r +r l +r t The total reward value, also known as the path safety reward value, is the reward amount r receives when the deep space rover encounters an obstacle. total =0.
[0075] S103, Utilizing the total reward value r from sampling total Training the value network, and then performing gradient descent on the value network:
[0076]
[0077] S104, Injecting strategy network p onto deep space rover π (s|θ π ) and value network p v (a,s|θ v ).
[0078] S11. On-orbit learning of decision parameters: Due to the limited detection and control arc of deep space rovers, the "sensing-movement cycle" has a large time span and a long feedback cycle. This invention replaces the traditional method of sending specific action commands by sending decision parameters to the deep space rover, placing the decision-making end on one side of the deep space rover. The ground plays a role in optimizing, monitoring, and uploading the planning parameters of the deep space rover. In this stage, the ground uploads the policy network and value network parameters learned through imitation learning to the remote deep space rover. The remote deep space rover updates the policy network and value network and uses them for path planning, moving according to the planned path. After moving, it collects the actual reward value on orbit and updates the policy network and value network. Specifically, this includes S110 to S112, specifically:
[0079] S110. The remote deep-space rover utilizes a tree search algorithm and a policy network p in the actual environment. π (s|θ π Generate a random policy using p v (a,s|θ v Evaluating the value of search strategies, specifically including S1100 to S1103, specifically:
[0080] S1100, in remote deep-space rover, utilizes a tree search algorithm and a policy network p in a real-world environment. π (s|θ π ) and value network p v (a,s|θ v Search strategy a), the search principles of strategy a are as follows:
[0081]
[0082] In the formula, Q(a) is the value of the action calculated by the tree search algorithm, N(a) is the number of times action a (policy a) has been visited, and π(a|s;θ) is the output value of the policy network.
[0083] S1101. After selecting an action 'a' each time, utilize the value network p. v (a,s|θ v The average of the actual benefits (weighted values in the environmental cost diagram) and the actual benefits received is taken, i.e.:
[0084] S1102. Take the average of all actions under action a and denote it as Q(a): Q(a) = mean(the recorded V(s)).
[0085] S1103. Repeat S1100 to S1102 until the termination condition is reached. Select action a according to a = argmaxN(a).
[0086] S111. Move to the target point according to the path sequence obtained by the search tree algorithm, and collect the reward r based on the movement result. The path and behavior reward function consists of four terms:
[0087] 1) Plan the path slope and calculate the path slope bonus value r. r , represented as Where α represents the zenith angle of the path path inspector's base plate vector (perpendicular to the base plate and pointing upwards), and N represents the number of sampling points on the path;
[0088] 2) Plan the path roughness and calculate the path roughness reward value r. t , is represented as: Where z represents the sampled value of the celestial component of the position of the rover as it moves along the path. Let z represent the mean of all z, and N represent the number of sampling points on the path;
[0089] 3) Plan the path steps and calculate the path step reward value r. l If there are no stairs, the reward value r l If the value is 1, the reward decreases by 0.1 for each tier.
[0090] The final total reward value r total :r total =r r +r l +r t The total reward value, also known as the path safety reward value, is the reward amount r receives when the deep space rover encounters an obstacle. total =0. r total Reverse propagation is used to update network parameters.
[0091] S112. Update the policy network p using gradient descent. π (s|θ π ) and value network p v (a,s|θ v Data collection (s) t ,a t ,s t+1 ,r).
[0092] Policy Networks:
[0093] Value Network:
[0094] S12. Iterative update of decision parameters based on feedback data:
[0095] The remote deep-space rover utilizes imitation learning to search for policies and value networks in a real-world environment, executes the searched policies, collects reward values, and uses these rewards to reinforce the policy and value networks. The rover then transmits telemetry parameters, reward values, and policy network Δθ data as needed during its movement. π and value network Δθ v The changes are transmitted to the ground. The ground updates the strategy network and value network for ground maintenance, specifically including S120 to S123:
[0096] S120, the remote deep-space rover transmits remote control parameters for the movement process, the reward value r for completing the movement process, and the policy network Δθ in the communication arc. π and value network Δθ v The change in quantity.
[0097] S121. Use the state sequence S in the newly transmitted decision data. t and the global status s on the ground t , forming S' t =[S t ,s t Using ground-based policy networks and value networks, search for policy sequence a' t Ground experts on strategy sequence a t and a' t Evaluate. If a t If better, then use the patrol policy network and value network, go to S120, if a' t If the result is better, then add the ground network parameter values to the subsequent arc segments as appropriate, and switch to S123.
[0098] S122, Update the ground policy network and value network; Decision data (s t ,a t ,s t+1 ,r) is added to the training dataset R.
[0099] S123, Injecting strategy network p to remote deep space rover π (s|θ π ) and the value network p_v(a,s|θ_v).
[0100] The key points of this invention are as follows:
[0101] 1) Two-way transmission of decision-making capabilities and status information:
[0102] Currently, deep-space rovers provide timely status information, but lack comprehensive global information and have weak decision-making capabilities. Ground-based tracking and control centers, on the other hand, provide more comprehensive status information and have stronger decision-making capabilities. Based on this, the following overall approach is proposed to address the problem: firstly, strengthen the transmission of decision-making capabilities from the ground control center to the deep-space rover; secondly, increase the efficiency of utilizing status feedback sent back by the deep-space rover. This two-pronged approach will bridge the capability imbalance between the rover and the ground control center, and improve the intelligence level of the rover's exploration mission planning.
[0103] 2) Decision parameters are updated iteratively across the entire system:
[0104] Due to the vast distances and high communication latency of deep-space rovers, real-time control is difficult. The logical starting point of this invention is to embed expert intelligence into a goal planning and recommendation network and transmit it to the deep-space rover, thus transferring expert intelligence to the rover. With the assistance of a policy network derived from imitation learning, the deep-space rover's decision-making capabilities are enhanced, enabling the remote release of expert intelligence. Simultaneously, through real-time feedback, the deep-space rover continuously accumulates high-value data. This newly generated data is then transmitted back to the ground for upgrading the expert policy network, achieving a continuous iterative increase in data value.
[0105] The task scenarios of this invention have two main characteristics:
[0106] 1) Ultra-long delay communication. Due to the long distance, there is an ultra-long delay in the uplink and downlink transmission of command and control signaling and extraterrestrial object status information.
[0107] 2) The environment is highly variable. Dynamic changes in planetary illumination and atmospheric activity cause deep space rovers to be in a state of high variability.
[0108] The two characteristics mentioned above can easily lead to a mismatch between delayed instructions and changing states. To address the specific characteristics of the mission scenario, this invention proposes a design approach for generating deep-space rover path strategies through iterative optimization using "expert experience cloning," "on-orbit strategy learning," and "state feedback iteration." The main steps of this invention are as follows:
[0109] 1) Imitation learning: Based on the initial deep space environment model, the policy network and value network are trained using the (state, action) provided by experts;
[0110] 2) The strategy network and value network are fed into the patrol's task decision-making unit;
[0111] 3) The deep space rover generates a movement path sequence based on its actual on-orbit status and the target point using its own policy network and value network. It moves to the target point according to the obtained movement path sequence and collects the current status, movement path sequence, and reward values from the actual environment. It then updates its policy network and value network based on the reward values from the environmental feedback.
[0112] 4) The deep space rover transmits its current status, movement path sequence, actual environmental feedback on gains, and the rover's strategy network and value network. Ground control center experts compare the merits of the space-ground model and update the deep space rover's decision network as needed.
[0113] The beneficial effects of this invention are as follows:
[0114] 1) This invention uses imitation learning to solidify expert intelligence into the target planning strategy network of a deep space rover and injects it into the deep space rover. While realizing the transfer of expert intelligence, it also shortens the distance between expert intelligence and the deep space rover, thereby effectively overcoming the problem caused by high-latency communication.
[0115] 2) This invention utilizes reinforcement learning to continuously transform the state feedback during the path planning process of a deep space rover into reward function values and continuously update the network parameters of the path planning, thereby achieving continuous upgrading of the path planning capability of the deep space rover and effectively overcoming the difficulties caused by high dynamic communication.
[0116] 3) The purpose of this invention is to leverage the capabilities of a large-scale space-ground system, integrating the intelligence of spacecraft agents, ground planning systems, and expert experience. Specifically, experts can deduce and reason based on spacecraft design and theoretical knowledge, enabling mission completion under conditions of small or no samples. Existing intelligent planning methods based on big data learning have made significant progress and are ready to be applied to spacecraft. Spacecraft can collect ample data during hundreds or thousands of mission executions, training their decision-making capabilities. Furthermore, deep-space spacecraft can select more suitable methods in real time based on the environment, overcoming the time delay problem of ground planning. The ground control system can integrate macroscopic and microscopic information, combining expert intelligence and data collected by the spacecraft to achieve a combination of expert intelligence and data statistics. This overcomes the contradictions caused by delayed communication and inconsistent states and commands due to changing environments.
[0117] In traditional reinforcement learning tasks, the optimal policy is typically learned by calculating cumulative rewards. This approach is simple, direct, and performs well when ample training data is available. However, in sequential decision-making, the learner cannot receive rewards frequently, and this reward-based learning method suffers from a very large search space. Imitation learning, after years of development, has become well-suited for solving sequential decision-making problems and has numerous applications in robotics, NLP, and other fields.
[0118] Reinforcement learning primarily consists of an agent, an environment, a state, an action, and a reward. After the agent performs an action, the environment transitions to a new state and provides a reward signal (positive or negative) for that new state. Subsequently, the agent, based on the new state and the reward from the environment, executes a new action according to a specific strategy. This process describes how the agent and environment interact through states, actions, and rewards.
[0119] Through reinforcement learning, an intelligent agent can understand its own state and the actions it should take to maximize its reward. Since the interaction between an intelligent agent and its environment is similar to that between humans and their environment, reinforcement learning can be considered a general learning framework that can be used to solve problems related to general artificial intelligence. Therefore, reinforcement learning is also known as a machine learning method for general artificial intelligence.
[0120] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.
[0121] An embodiment of the present invention provides a deep space exploration path planning system, comprising a ground control center and a deep space rover;
[0122] The ground control center is used to: obtain the initial parameters of the policy network and the initial parameters of the value network based on expert experience, and upload them to the deep space rover;
[0123] The deep space rover is used to: input the acquired perception data into the policy network with initial parameters of the policy network and the value network with initial parameters of the value network, perform path planning, and obtain the planned path;
[0124] The deep space rover is also used to: move along a planned path, acquire actual movement data during the movement, and update the initial parameters of the policy network and the value network based on the actual movement data.
[0125] Optionally, in the above technical solution, the deep space rover is also used to: use the updated policy network initial parameters as policy network initial parameters, use the updated value network initial parameters as value network initial parameters, repeatedly perform path planning, and update the policy network initial parameters and value network initial parameters.
[0126] Optionally, in the above technical solution, the deep space rover is also used to: send the updated initial parameters of the strategy network and the updated initial parameters of the value network to the ground control center.
[0127] Optionally, in the above technical solution, the ground control center is also specifically used to: use the behavior cloning method in imitation learning and based on expert experience to obtain the initial parameters of the policy network and the initial parameters of the value network of the policy network.
[0128] Optionally, in the above technical solution, the deep space rover is also specifically used for:
[0129] The path slope reward value, path roughness reward value, and path step reward value are determined based on the actual movement data.
[0130] The initial parameters of the policy network and the initial parameters of the value network are updated based on the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value, and the path step reward value.
[0131] It should be noted that the beneficial effects of the deep space exploration path planning system provided in the above embodiments are the same as those of the deep space exploration path planning method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.
[0132] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.
[0133] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.
[0134] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0135] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A deep space exploration path planning method, characterized in that, include: S1. Based on expert experience, the ground control center obtains the initial parameters of the strategy network and the initial parameters of the value network, and uploads them to the deep space rover. S2. The deep space rover inputs the acquired sensing data into the policy network with initial parameters of the policy network and the value network with initial parameters of the value network to perform path planning and obtain the planned path. S3. The deep space rover moves along the planned path, acquires actual movement data during the movement, and updates the initial parameters of the policy network and the initial parameters of the value network based on the actual movement data. Also includes: The deep space rover uses the updated policy network initial parameters as the policy network initial parameters and the updated value network initial parameters as the value network initial parameters, and then returns to execute S2. The process by which the deep space rover updates the initial parameters of the policy network and the initial parameters of the value network based on the actual movement data includes: The path slope reward value, path roughness reward value, and path step reward value are determined based on the actual movement data. The initial parameters of the policy network and the initial parameters of the value network are updated based on the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value, and the path step reward value; Based on expert experience, the ground control center obtained the initial parameters of the policy network and the initial parameters of the value network, including: The ground control center utilizes the behavior cloning method in imitation learning and, based on expert experience, obtains the initial parameters of the policy network and the initial parameters of the value network, including: S100, Expert Sample Data Collection: Experts make plans based on historical surrounding environment and status information or simulated surrounding environment and status information transmitted by deep space rovers. The dataset consists of data samples from each sensing-movement sequence: D=<( s 1, a 1),( s 2, a 2),( s 3, a 3),…,( s t , a t )>, where, S t The state vector and action are extracted from the perceived image data. a t The moving path curve is represented by a finite number of discrete curvature and arc length values; S101. Obtain the initial policy network using imitation learning with expert data. : Through dataset D=<( s 1, a 1),( s 2, a 2),( s 3, a 3),…,( s t , a t Training the policy network ; Execute in the actual environmental data or simulation data of the deep space rover Obtain a new dataset Experts Mark the movement path curve of the deep space rover a And perform aggregation to obtain a new dataset. Return to continuously training the policy network ; S102, Building a Value Network Using the initial policy network Randomly generate a policy sampling reward function in the environment. The reward function includes: 1) planning the path slope and calculating the path slope reward value. , represented as ,in, The zenith angle represents the vector on the base plate of the rovers at the path points. 1) Indicate the number of sampling points on the path; 2) Plan the path roughness and calculate the path roughness reward value. , represented as: ,in, This represents the sampled value of the celestial component of the position of the rover as it moves along the path. Indicates all z The mean, 3) Plan the path steps and calculate the path step reward value. If there are no tiers, then the reward value The initial reward is 1; if applicable, the reward decreases by 0.1 for each tier; the final total reward value is... r total : When the deep space rover's path encounters an obstacle, r total =0; S103, Utilizing the total reward value from sampling r total Training the value network, and then performing gradient descent on the value network: ; S104, Injecting Strategy Networks onto Deep Space Rover and value network .
2. The deep space exploration path planning method according to claim 1, characterized in that, Also includes: The deep space rover sends the updated initial parameters of the strategy network and the updated initial parameters of the value network to the ground control center.
3. A deep space exploration path planning system, characterized in that, Including ground control centers and deep space rover; The ground control center is used to: obtain the initial parameters of the policy network and the initial parameters of the value network based on expert experience, and upload them to the deep space rover. The deep space rover is used to: input the acquired perception data into the policy network with initial parameters of the policy network and the value network with initial parameters of the value network, perform path planning, and obtain a planned path; The deep space rover is also used to: move along the planned path, acquire actual movement data during the movement, and update the initial parameters of the policy network and the initial parameters of the value network based on the actual movement data; The deep space rover is also used to: use the updated policy network initial parameters as policy network initial parameters, use the updated value network initial parameters as value network initial parameters, repeatedly perform path planning, and update the policy network initial parameters and value network initial parameters. The deep space rover is also specifically used for: The path slope reward value, path roughness reward value, and path step reward value are determined based on the actual movement data. The initial parameters of the policy network and the initial parameters of the value network are updated based on the total reward value, wherein the total reward value is the sum of the path slope reward value, the path roughness reward value, and the path step reward value; The ground control center is further specifically used for: utilizing the behavior cloning method in imitation learning and based on expert experience, to obtain the initial parameters of the policy network and the initial parameters of the value network, including: S100, Expert Sample Data Collection: Experts make plans based on historical surrounding environment and status information or simulated surrounding environment and status information transmitted by deep space rovers. The dataset consists of data samples from each sensing-movement sequence: D=<( s 1, a 1),( s 2, a 2),( s 3, a 3),…,( s t , a t )>, where, S t The state vector and action are extracted from the perceived image data. a t The moving path curve is represented by a finite number of discrete curvature and arc length values; S101. Obtain the initial policy network using imitation learning with expert data. : Through dataset D=<( s 1, a 1),( s 2, a 2),( s 3, a 3),…,( s t , a t Training the policy network ; Execute in the actual environmental data or simulation data of the deep space rover Obtain a new dataset Experts Mark the movement path curve of the deep space rover a And perform aggregation to obtain a new dataset. Return to continuously training the policy network ; S102, Building a Value Network Using the initial policy network Randomly generate a policy sampling reward function in the environment. The reward function includes: 1) planning the path slope and calculating the path slope reward value. , represented as ,in, The zenith angle represents the vector on the base plate of the rovers at the path points. 1) Indicate the number of sampling points on the path; 2) Plan the path roughness and calculate the path roughness reward value. , represented as: ,in, This represents the sampled value of the celestial component of the position of the rover as it moves along the path. Indicates all z The mean, 3) Plan the path steps and calculate the path step reward value. If there are no tiers, then the reward value The initial reward is 1; if applicable, the reward decreases by 0.1 for each tier; the final total reward value is... r total : When the deep space rover's path encounters an obstacle, r total =0; S103, Utilizing the total reward value from sampling r total Training the value network, and then performing gradient descent on the value network: ; S104, Injecting Strategy Networks onto Deep Space Rover and value network .
4. A deep space exploration path planning system according to claim 3, characterized in that, The deep space rover is also used to: send the updated policy network initial parameters and the updated value network initial parameters to the ground control center.