Deep space network invulnerability optimization method and system based on CTDE meta-reinforcement learning

Abstract

The application discloses a deep space network invulnerability optimization method and system based on CTDE meta-reinforcement learning, and belongs to the technical field of deep space communication networks. The method comprises the following steps: constructing a deep space network model containing multiple types of heterogeneous nodes and dynamic links; establishing a three-layer invulnerability index system covering the structure layer, the function layer and the task layer; defining four types of threat models, namely random failure, directional attack, environmental interference and energy depletion; defining a continuous strategy parameter space for each node, which is composed of a link rerouting tendency, a sleep intensity and a power adjustment amplitude; iteratively optimizing the strategy parameters of each node by using a global reward signal through a meta-learning mechanism in a centralized training phase; and in a distributed execution phase, each node independently executes five types of adaptive actions, namely link enhancement, intelligent sleep, power control, energy management and load balancing, only relying on the three local parameters. The application realizes compression of the strategy search space from a combinatorial explosion level to a linear level.

Description

Technical Field

[0001] This invention belongs to the field of deep space communication network technology, specifically relating to a method and system for optimizing the resilience of deep space networks based on CTDE meta-reinforcement learning, and particularly to a meta-reinforcement learning (Meta-RL) algorithm based on a centralized training-decentralized execution (CTDE) architecture, used to improve the resilience and mission carrying capacity of networks composed of multiple types of communication nodes in deep space exploration missions. Background Technology

[0002] The deep space exploration network consists of various heterogeneous communication nodes, such as probes, relay satellites, ground base stations, and roving robots deployed in interplanetary space, undertaking key tasks such as scientific data transmission, remote control command distribution, and multi-node collaborative navigation. Compared with traditional ground communication networks, the deep space heterogeneous network has the following significant characteristics: (1) Extreme time-varying nature: nodes move continuously at high speeds, links are frequently connected and disconnected, and the network topology changes drastically on a minute-level time scale; (2) High heterogeneity: nodes have significant differences in communication range, energy reserves, computing power, and sensing capabilities; (3) Extremely limited resources: node energy cannot be replenished, communication bandwidth is limited, and computing resources are scarce; (4) Coexistence of multiple threats: the network faces multiple threats such as random failures caused by space radiation, targeted attacks by enemies, environmental interference such as solar storms, and the gradual depletion of node energy.

[0003] In existing technologies, the optimization of the resilience of deep space networks mainly faces the following technical challenges:

[0004] First, there is the scalability problem of centralized reinforcement learning methods. Centralized methods, represented by Deep Q-Network (DQN), require the central node to maintain a global state space and Q-value table. As the number of network nodes increases, the state-action space grows exponentially, leading to slow training convergence, excessive communication overhead, and single point of failure causing the entire optimization system to fail.

[0005] Second, there is the coordination difficulty of distributed multi-agent methods. Distributed methods, represented by Multi-Agent Reinforcement Learning (MARL), allow each node to make independent decisions, which avoids single points of failure. However, due to the lack of global information sharing and policy coordination, the local optimal behavior of each node can easily lead to global suboptimal behavior, resulting in insufficient performance when dealing with threats that require a coordinated response across the entire network, such as targeted attacks.

[0006] Third, there is the contradiction between policy search space and convergence efficiency. Traditional reinforcement learning methods typically use discrete action spaces. As the network size increases, the action combinations exhibit combinatorial explosion characteristics, resulting in extremely low policy search efficiency. On the other hand, high-dimensional policy networks that directly use continuous action spaces require a large number of training samples, which is difficult to apply in deep space environments where sample acquisition costs are extremely high.

[0007] Fourth, there is a lack of a unified multidimensional resilience assessment framework. Existing methods typically assess network resilience from only a single dimension (such as connectivity), which fails to comprehensively reflect the network's overall performance across multiple levels, including structural integrity, functional availability, and task carrying capacity, leading to a one-sided approach to optimization.

[0008] Therefore, there is an urgent need for a method to enhance the resilience of deep space heterogeneous time-varying networks that can operate efficiently under resource-constrained conditions, possess both global coordination capabilities and local autonomous decision-making advantages, and can comprehensively optimize multidimensional resilience. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention provides a method and system for deep space network resilience optimization based on CTDE meta-reinforcement learning. This method compresses the decision space of each node into only three continuous parameters (link rerouting tendency, sleep intensity, and power adjustment). Through a meta-reinforcement learning framework of "centralized training and distributed execution," these parameters are collaboratively optimized using global information during the training phase. During the execution phase, each node independently executes five types of adaptive actions relying only on its local three parameters. This simultaneously resolves the contradictions of poor scalability in centralized methods, insufficient coordination in distributed methods, and the explosion of the policy search space.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A deep space network resilience optimization method based on CTDE meta-reinforcement learning includes:

[0012] Step S1: Construct a deep-space heterogeneous time-varying network model;

[0013] Step S2: Establish a three-layer resilience indicator system, including structural layer indicators, functional layer indicators, and mission layer indicators;

[0014] Step S3: Define a set of threat models, including random failure threats, targeted attack threats, environmental interference threats, and energy depletion threats;

[0015] Step S4: Define a set of continuous policy parameters for each node in the deep space heterogeneous time-varying network model. The continuous policy parameters include link rerouting tendency parameters, sleep mode strength parameters, and power adjustment magnitude parameters.

[0016] Step S5: Using the global state information of the deep space heterogeneous time-varying network model, inject any threat from the threat model set with a preset probability. Calculate the global comprehensive reward signal based on the weighted synthesis of the three-layer resilience index system. Based on the average reward change between the current training round and the previous training round, update the continuous policy parameters of each node using the product of the preset meta-learning rate and Gaussian exploration noise as the update step size, and restrict the updated continuous policy parameters to a preset value range.

[0017] Step S6: Without the need for global information or inter-node communication, each node independently executes link quality enhancement actions, intelligent sleep actions, adaptive power control actions, energy management actions, and load balancing actions, relying only on the locally continuous policy parameters updated by gradients.

[0018] Step S7: Output the network configuration parameters obtained after each node independently executes the corresponding action, and the resilience assessment results under various threat scenarios based on the three-layer resilience index system.

[0019] Furthermore, in step S1, the network model includes multiple types of heterogeneous nodes, dynamic communication links, node energy consumption models, and node motion models.

[0020] Furthermore, in step S2, the structural layer indicators include connectivity, average degree, clustering coefficient, maximum connected component ratio, and degree variance; the functional layer indicators include average transmission latency, packet loss rate, network efficiency, and energy efficiency; and the task layer indicators include task completion rate, task latency, load balancing degree, and interruption probability.

[0021] Furthermore, in step S4, the link rerouting tendency parameter, the sleep mode strength parameter, and the power adjustment amplitude parameter all have the same value range; the continuous policy parameter compresses the node decision space from a high-dimensional discrete state-action space to a low-dimensional continuous parameter space. For a network containing N nodes, the total number of parameters to be optimized is three times N, and the policy search space is of linear order N.

[0022] Furthermore, in step S5, the global comprehensive reward signal is calculated by assigning preset weight coefficients to connectivity rate, task completion rate, maximum connectivity component ratio, normalized positive value of average transmission delay, load balancing degree, and proportion of inactive nodes, and then performing weighted summation.

[0023] Furthermore, in step S5, the gradient update based on the average reward change between the current training round and the previous training round specifically involves: obtaining the difference between the average reward of the current training round and the average reward of the previous training round; multiplying the preset meta-learning rate by the difference and then by Gaussian exploration noise following a preset distribution to obtain the parameter update amount; adding the continuous policy parameters of the current round to the parameter update amount to obtain the updated continuous policy parameters; and restricting the updated continuous policy parameters to a preset value range of the continuous policy parameters as the continuous policy parameters for the next training round.

[0024] Furthermore, in step S6:

[0025] The link quality enhancement action is driven by the link rerouting tendency parameter and is used to enhance the quality of the best neighbor link and weak links; the intelligent sleep action is driven by the sleep mode strength parameter and is used to reduce the load and shrink the communication range when the node energy is lower than the average level of the entire network; the adaptive power control action is driven by the power adjustment amplitude parameter and is used to asymmetrically expand or shrink the communication range according to the node energy level; the energy management action forcibly reduces the load and shrinks the communication range when the node's remaining energy is lower than the emergency threshold; the load balancing action transfers the load according to the neighbor energy order when the node load exceeds the threshold.

[0026] On the other hand, this invention provides a deep space network resilience optimization system based on CTDE meta-reinforcement learning, comprising:

[0027] The network modeling module is used to construct a deep space heterogeneous time-varying network model that includes multiple types of heterogeneous nodes, dynamic communication links, node energy consumption models, and node motion models.

[0028] The indicator system module is used to establish an indicator system for evaluating network resilience from three dimensions: structural layer, functional layer, and task layer.

[0029] The Threat Model module is used to define a set of threat models, including random failures, targeted attacks, environmental interference, and energy depletion.

[0030] The policy parameter module is used to define continuous policy parameters for each node, consisting of link rerouting tendency parameters, sleep mode strength parameters, and power adjustment magnitude parameters.

[0031] The centralized training module is used to utilize the global state information of the deep space heterogeneous time-varying network model to inject any threat from the threat model set with a preset probability, calculate the global comprehensive reward signal based on the weighted synthesis of the three-layer resilience index system, and perform gradient updates on the continuous policy parameters of each node based on the change of the global comprehensive reward signal using a meta-learning mechanism.

[0032] The distributed execution module enables each node to independently execute link quality enhancement actions, intelligent sleep actions, adaptive power control actions, energy management actions, and load balancing actions, relying only on the locally continuous policy parameters updated by gradients.

[0033] The results output module is used to output the optimized network configuration parameters and the resilience assessment results under various threat scenarios.

[0034] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned deep space network anti-damage optimization method based on CTDE meta-reinforcement learning.

[0035] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned deep space network anti-damage optimization method based on CTDE meta-reinforcement learning.

[0036] The beneficial effects of this invention are as follows:

[0037] The unique advantages of the CTDE architecture are: through the framework design of "centralized training and distributed execution", it uses global information to achieve collaborative optimization of policy parameters during the training phase, and each node can make high-quality decisions independently by relying only on three local continuous parameters during the execution phase. It takes into account both global optimality and distributed robustness, and fundamentally solves the technical contradiction between the poor scalability of centralized methods and the difficulty of coordination of distributed methods.

[0038] Extremely low-dimensional parameterized policy space: The decision space of each node is compressed from the thousands of dimensions of the state-action space in traditional methods to only three continuous parameters. The policy search space is reduced from combinatorial explosion level to linear level, which greatly reduces the number of samples required for training and computational complexity, enabling the algorithm to converge efficiently even under the condition that the cost of obtaining samples in the deep space environment is extremely high.

[0039] Meta-learning-driven fast adaptation: Employing a meta-learning parameter update mechanism based on global reward changes, policy parameters can adaptively adjust according to changes in the overall network performance. Due to its extremely small parameter search space and simple, efficient update rules, the algorithm's convergence speed significantly outperforms existing deep Q-networks and multi-agent reinforcement learning methods.

[0040] Synergistic Effect of Five Types of Adaptive Actions: Through the coordinated implementation of five types of actions—link quality enhancement, intelligent sleep mode, adaptive power control, energy management, and load balancing—multi-layered resilience optimization from the link layer to the network layer is achieved. Simulation results show that connectivity is improved by approximately 450% compared to the baseline strategy, and task completion rate is improved by approximately 35%.

[0041] The comprehensiveness of the three-layer multidimensional resilience assessment: The network performance is comprehensively evaluated from three dimensions—structural layer, functional layer, and mission layer—with more than ten indicators. This avoids the one-sidedness of traditional single-indicator assessments and provides a complete evaluation benchmark for the comprehensive optimization of deep space networks.

[0042] Unified protection against multiple threats: The algorithm modeled multiple threats faced by deep space networks, including random failures, targeted attacks, environmental interference, and energy depletion. The algorithm maintained optimal performance in all threat scenarios, verifying the generality and robustness of the method. Attached Figure Description

[0043] Figure 1 This is a flowchart of the deep space network anti-damage optimization method based on CTDE meta-reinforcement learning of the present invention;

[0044] Figure 2 This is a schematic diagram of the architecture of the CTDE-based Meta-RL algorithm, including the logical relationship between the centralized training phase and the distributed execution phase;

[0045] Figure 3 This is a hierarchical structure diagram of the three-tiered resilience index system;

[0046] Figure 4 The time-series comparison curves of the three algorithms on the three-layer resilience indicators (structural layer, functional layer, and task layer) are shown.

[0047] Figure 5 A bar chart comparing the three algorithms in three key metrics: connectivity, task completion rate, and network efficiency under four threat scenarios;

[0048] Figure 6 This is a comprehensive comparison curve of the three algorithms across multiple performance metrics. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] like Figure 1 As shown, this invention provides a deep space network resilience optimization method based on CTDE meta-reinforcement learning, comprising the following steps:

[0051] Step S1: Construct a deep space heterogeneous time-varying network model.

[0052] A deep-space heterogeneous time-varying network model is constructed, comprising multiple types of heterogeneous nodes, dynamic communication links, energy consumption models, and node motion models. The heterogeneous nodes include, but are not limited to, four types: detector nodes, satellite nodes, base station nodes, and robot nodes. Each type has different communication range, energy coefficient, computing power, and sensing capability parameters. The link quality q_ij of the dynamic communication link is determined by the distance between nodes and the remaining energy of each node.

[0053] q_ij = (E_i + E_j) / (2·E_max) · exp(-d_ij / min(R_i, R_j)),

[0054] Where E_i and E_j are the remaining energy of the i-th node and the j-th node, E_max is the maximum energy value, d_ij is the distance between the i-th node and the j-th node, and R_i and R_j are the communication range of the i-th node and the j-th node.

[0055] The node motion model adopts a random walk model in a two-dimensional plane.

[0056] Step S2: Establish a three-tiered survivability index system.

[0057] Establish an indicator system to comprehensively evaluate network resilience from three dimensions: structural layer, functional layer, and task layer. Figure 3 As shown. The structural layer metrics include connectivity, average degree, clustering coefficient, maximum connected component ratio, and degree variance, reflecting the integrity and robustness of the network topology. The functional layer metrics include average transmission delay, packet loss rate, network efficiency, and energy efficiency, reflecting the availability and efficiency of network communication functions. The task layer metrics include task completion rate, task delay, load balancing, and outage probability, reflecting the network's overall ability to carry probing tasks. The network efficiency η in the functional layer is calculated by a weighted average of link quality factor, active node factor, coverage integrity factor, and connectivity factor.

[0058] η = κ1·Q_link + κ2·η_active + κ3·η_cover + κ4·C,

[0059] Where Q_link is the average link quality of the entire network, η_active is the proportion of active nodes, η_cover is the coverage integrity, C is the connectivity rate, and κ1 to κ4 are the weight coefficients of the corresponding items.

[0060] Step S3: Define the set of threat models.

[0061] Define a set of threat models covering typical failure modes of deep space networks, including at least the following four threat types: random failure threat—randomly selecting at least one node in the network to fail, simulating space radiation or random hardware failure; targeted attack threat—selecting the hub node with the highest node degree in the network to fail, simulating a precision strike against a core node; environmental interference threat—randomly selecting a node with a predetermined probability and reducing the link quality of all its links by 30% to 50%, simulating solar storms or electromagnetic interference; and energy depletion threat—randomly selecting a node from nodes with remaining energy below a preset threshold to fail, simulating gradual failure caused by energy depletion.

[0062] Step S4: Define a continuous parameterized policy space for each node.

[0063] For each node i in the network, a set of low-dimensional continuous policy parameters θ_i = (α_i, β_i, γ_i) is defined, where: α_i represents the link rerouting tendency parameter, controlling the strength of the node's behavior in optimizing link quality; β_i represents the sleep mode strength parameter, controlling the strength of the node entering a low-power state when energy is insufficient; and γ_i represents the power adjustment magnitude parameter, controlling the magnitude of the node's adaptive adjustment of the communication range according to the energy state. The introduction of these policy parameters compresses the decision space of each node from the high-dimensional discrete state-action space of traditional reinforcement learning to an extremely low-dimensional continuous space with only three continuous parameters, significantly reducing the complexity of policy search. The values ​​of α_i, β_i, and γ_i all range from [θ_min, θ_max], where θ_min = 0.15 and θ_max = 0.99. The initial values ​​of the policy parameters are randomly initialized within the range of [0.7, 0.98]. Only 3N consecutive parameters need to be optimized for N nodes, and the policy search space is O(N), which is much smaller than the traditional discrete action space of O(|A|^N), where A represents the number of actions that a single node can choose.

[0064] Step S5: Perform the CTDE-based Meta-RL intensive training phase.

[0065] During the intensive training phase, the global state information of the deep space heterogeneous time-varying network model described in the simulation environment is used to iteratively optimize the policy parameters of each node through a meta-learning mechanism, such as... Figure 2 As shown: After randomly injecting threats (at least one of the threat types selected in step S3) with a preset probability (e.g., 40% to 50%), perform multi-round simulations. Calculate the global comprehensive reward signal based on the weighted synthesis of the resilience index described in step S2. Based on the change in the global comprehensive reward signal, perform meta-learning gradient updates on the policy parameters of each node defined in step S4; specifically including:

[0066] S5.1 Initialization - Randomly initialize the policy parameters (α_i, β_i, γ_i) of all nodes within a predetermined range, and set the meta-learning rate β, the number of training rounds E, and the number of simulation time steps per round T;

[0067] S5.2 Multi-round simulation - In each training round, K simulation rounds are executed. Each round includes a threat randomly injected with a preset probability. Within T simulation time steps, each node performs distributed adaptive actions based on the current policy parameters. Each step updates the node position, link status and network topology, and calculates the global comprehensive reward signal.

[0068] S5.3 Global Comprehensive Reward Calculation – The global comprehensive reward signal r is weighted and synthesized based on the survivability index:

[0069] r = ω1·C + ω2·R_task + ω3·R_lcc + ω4·(1 - D / 10) + ω5·LB +ω6·(1 - P_active),

[0070] Where C is the connectivity rate, R_task is the task completion rate, R_lcc is the maximum connected component ratio, D is the average transmission delay, LB is the load balancing degree, P_active is the proportion of active nodes, and ω1 to ω6 are the weight coefficients of the corresponding items.

[0071] S5.4 Meta-learning parameter update—Based on the average reward change ΔR between the current round and the previous round, the policy parameters of all nodes are updated using gradients:

[0072] θ_i^(epoch+1) = clip(θ_i^(epoch) + β·ΔR·N(μ, σ 2 ), θ_min, θ_max),

[0073] Where, N(μ, σ) 2 The mean is μ and the variance is σ. 2 Gaussian exploration noise, clip operation is used to restrict parameters to a legal range, epoch represents the training epoch, ΔR=R_epoch - R_{epoch-1}, R_epoch is the average reward of the current epoch, and R_{epoch-1} is the average reward of the previous epoch.

[0074] It should be noted that, in the context of this invention, "meta-learning mechanism" or "meta-learning update" specifically refers to the rule described above that updates parameter gradients based on the difference between the average reward of the current round and the previous round, combined with Gaussian exploration noise. This update rule differs from traditional meta-learning methods, such as MAML (Model-Agnostic Meta-Learning), which require calculating quadratic gradients. Its core lies in using the change in the reward signal to directly drive the rapid adjustment of policy parameters, thereby adapting to the dynamic changes in the deep space network environment.

[0075] Step S6: Execute the distributed adaptive action phase.

[0076] During the distributed execution phase, each node i relies only on its own three local policy parameters (α_i, β_i, γ_i) and independently executes at least one of the following five types of adaptive actions without requiring global information or explicit communication between nodes:

[0077] S6.1 Link Quality Enhancement Action – Driven by α_i, when α_i exceeds the preset threshold, the enhancement operation for the best neighbor link is q' = min(1.0, q·(1 + k1·α_i)), and the enhancement operation for the weak link is q' = min(1.0, q·(k2 + k3·α_i)), where k1, k2, and k3 are preset enhancement coefficients, q is the current link quality, and q' is the enhanced link quality;

[0078] S6.2 Intelligent Sleep Action – Driven by β_i, when β_i exceeds a preset threshold and the remaining energy of the node is lower than a preset proportion of the average energy of the entire network, the communication load is reduced and the communication range is shrunk. Specifically, the following operations are performed: the communication load is reduced to L' = max(L_min, L·(p1 + p2·β_i)); the communication range is shrunk to R' = max(R_min, R·(p3+ p4·β_i)); where p1 to p4 are preset sleep coefficients, L is the current load, R is the current communication range, and L_min and R_min are the preset lower limits of the load and communication range, respectively.

[0079] S6.3 Adaptive Power Control Action – Driven by γ_i, when γ_i exceeds a preset threshold, asymmetric power adjustment is performed based on the energy state: when the remaining energy of a node is higher than a first preset proportion of the average energy of the entire network, the communication range is expanded by R1' = min(R_max, R·(1 + m1·γ_i)); when the remaining energy of a node is lower than a second preset proportion of the average energy of the entire network, the communication range is contracted by R2' = max(R_min, R·m2); where m1 and m2 are scaling factors, and R_max and R_min are the preset upper and lower limits of the communication range, respectively, with the first preset proportion being greater than the second preset proportion.

[0080] S6.4 Global Coordination Energy Management Action—When the remaining energy E_i of a node is lower than the preset emergency threshold E_emergency, energy management is forcibly executed: the load is reduced to L'_i = max(L_min, L_i · λ_L), and the communication range is shrunk to R'_i = max(R_min, R_i · λ_R), where λ_L and λ_R are the load shrinkage factor and the range shrinkage factor, respectively, and L_min and R_min are the preset lower limits of the load and the communication range, respectively, to sacrifice local functions to ensure the sustainability of the entire network's energy. L_i is the communication load of the i-th node before the energy management action is executed, and L'_i is the communication load value of the i-th node after the energy management action is executed.

[0081] S6.5 Intelligent Load Balancing Action: When a node's load exceeds a preset threshold, the node's own preset proportion of load is preferentially transferred to neighboring nodes that meet the energy requirements and whose load is below the preset receiving threshold, according to the order of remaining energy of neighboring nodes from high to low, thereby achieving collaborative load sharing.

[0082] Step S7: Output the optimized network configuration and performance evaluation. After training, output the optimization strategy parameters for each node, the evaluation results of the optimized network resilience index, and the comparative experimental results under various threat scenarios.

[0083] Example 1: Modeling of heterogeneous time-varying networks in deep space.

[0084] The deep-space heterogeneous time-varying network model constructed in this embodiment includes 20 nodes, deployed within a 500km × 500km two-dimensional planar simulation region. The nodes are divided into four categories, and their parameter settings are shown in Table 1 below:

[0085] Table 1

[0086]

[0087] The initial positions of each node are uniformly and randomly distributed within the simulation area. The initial energy is randomly selected between 70 and 100, and the initial load is randomly selected between 0.25 and 0.75. The node movement speed is randomly set between 0.5 km / step and 1.0 km / step, the movement direction is random, and the boundary adopts an elastic rebound mechanism.

[0088] The link is established when the distance d_ij between the i-th node and the j-th node is less than the minimum communication range min(R_i, R_j). The link quality q_ij is calculated by the following formula:

[0089] q_ij = (E_i + E_j) / 200 · exp(-d_ij / min(R_i, R_j)),

[0090] Where E_i and E_j are the remaining energy of the i-th and j-th nodes, respectively. The link is valid if q_ij > 0.1. At each time step, the node energy consumption is ΔE_i = 0.02 + 0.01·L_i, where L_i is the current communication load of the i-th node. The energy E_i ranges from [0, 100], and the node fails when E_i drops to 0.

[0091] Example 2: Training process of Meta-RL algorithm based on CTDE.

[0092] In this embodiment, the training parameters of the CTDE-based Meta-RL algorithm are set as follows: number of training rounds E = 30 rounds, number of rounds per round K = 50 rounds, number of simulation steps per round T = 100 steps, meta-learning rate η = 0.08, and Gaussian exploration noise distribution N(μ=0.5, σ 2 =0.5 2 The policy parameters are in the range of α_i, β_i, γ_i ∈ [0.15, 0.99], and the initial values ​​of the policy parameters are randomly initialized in the range of [0.7, 0.98].

[0093] In each training round, a threat is randomly injected with a probability of 40%–50% (randomly selected from four types: random failure, targeted attack, environmental interference, and energy depletion), and the injection occurs around step 50 of the simulation. The weighting coefficients of the comprehensive reward function are set as follows: ω1 = 0.30, ω2 = 0.25, ω3 = 0.20, ω4 = 0.10, ω5 = 0.10, ω6 = 0.05.

[0094] Example 3: Detailed execution of adaptive actions driven by strategy parameters.

[0095] The specific implementation parameters for the five types of adaptive actions in the embodiment are described in detail below:

[0096] Action 1 – Link Quality Enhancement Action: Triggered when α_i > 0.15. Execute q' = min(1.0, q×(1 + 0.65×α_i)) for the best neighbor; execute q' = min(1.0, q×(1.65 + 0.4×α_i)) for all weak links with link quality q_ij < 0.65.

[0097] Action 2 – Intelligent Sleep Action: Triggered when β_i > 0.15 and E_i < 0.92·Ē (Ē is the average energy of the entire network). The load is reduced to L'_i = max(0.03, L_i×(0.22 + 0.22×β_i)), and the communication range is shrunk to R'_i = max(55, R_i×(0.60 + 0.22×β_i)). If E_i < 22, the node enters a forced failure state.

[0098] Action 3 – Adaptive Power Control Action: Triggered when γ_i > 0.15. If E_i > 0.72·Ē, the communication range is expanded to R'_i = min(230, R_i×(1 + 0.55×γ_i)); if E_i < 0.42·Ē, the communication range is contracted to R'_i = max(55, R_i×0.72).

[0099] Action 4 – Global Coordination Energy Management Action: Forced to trigger when E_i < 28. The load is reduced to 0.3·L_i, and the communication range is reduced to 0.72·R_i to conserve residual energy for maintaining basic network connectivity.

[0100] Action 5 – Intelligent Load Balancing Action: Triggered when L_i > 0.52. Neighboring nodes are sorted from highest to lowest remaining energy. For each neighbor j, if E_j > 28 and L_j < 0.42, then the current node i unloads ΔL = 0.35·L_i of load, and neighbor node j receives a load increment of ΔL_j = 0.10·L_j.

[0101] The above actions are executed independently by each node at each time step based on three local policy parameters, without the need for communication and coordination between nodes. However, at the global level, the division of labor and cooperation of actions are achieved through parameter co-optimization during the CTDE training phase.

[0102] Example 4: Comparison and verification with existing methods.

[0103] To verify the effectiveness of this invention, it was compared with the following existing methods under the same simulation environment:

[0104] (1) Baseline method: Unoptimized raw network, operating only according to the basic communication protocol.

[0105] (2) DQN method: A centralized deep Q network is used for optimization. The state space is a 100-dimensional global network state, and the action space is 8 discrete actions. The Q network adopts a double hidden layer (64-32 neurons) structure.

[0106] (3) MARL method: Distributed multi-agent Q learning is adopted. Each node independently maintains 20 Q tables. The state is 5-dimensional local observation discretized into 100 state indices. The action is 4 discrete actions.

[0107] The total number of training steps for the three RL methods was uniformly set to 150,000 steps to ensure a fair comparison.

[0108] like Figure 4As shown, the time-series curves of nine resilience indicators for the three methods and the baseline method over 100 time steps are presented. Regarding structural layer indicators, connectivity C: Meta-RL maintained a range of 0.40–0.45 throughout, DQN was approximately 0.20–0.28 with significant fluctuations, MARL was approximately 0.12–0.20, and the baseline was below 0.10. Mean degree k̄: Meta-RL was approximately 7.5–8.5, DQN was approximately 4.0–5.5, MARL was approximately 2.0–3.5, and the baseline was approximately 1.0–2.0. Clustering coefficient CC: Meta-RL was approximately 0.35–0.40, DQN was approximately 0.22–0.28, MARL was approximately 0.20–0.24, and the baseline was the lowest. Regarding functional layer metrics, transmission latency (D): Meta-RL approximately 4.40~4.65 (lowest), DQN approximately 4.65~4.85, MARL approximately 4.80~5.05, and baseline approximately 5.00~5.25 (highest). Packet loss rate (PL): Meta-RL approximately 0.98~1.02, DQN approximately 1.02~1.08, MARL approximately 1.05~1.12, and baseline approximately 1.10~1.22. Network efficiency (η): Meta-RL approximately 0.45~0.50, DQN approximately 0.35~0.40, MARL approximately 0.30~0.35, and baseline approximately 0.25~0.30. Regarding task-level metrics, task completion rate R_task: Meta-RL approximately 0.47~0.51, DQN approximately 0.41~0.45, MARL approximately 0.38~0.42, and the baseline approximately 0.35~0.38. Energy efficiency ηE: Meta-RL approximately 0.35~0.40, DQN approximately 0.27~0.33, MARL approximately 0.25~0.30, and the baseline approximately 0.20~0.25. Load balancing LB: MARL approximately 0.50~0.60 (due to independent decisions by each node leading to naturally dispersed load), Meta-RL approximately 0.35~0.45 (more optimized load distribution under global coordination), DQN approximately 0.15~0.25, and the baseline close to 0. Among all nine metrics, Meta-RL achieves the best results in eight (load balancing is slightly lower than MARL), and all metrics are superior to DQN and the baseline method.

[0109] like Figure 5As shown, the comparison of the four methods in three key metrics—connectivity, task completion rate, and network efficiency—under four threat scenarios: random failure, targeted attack, environmental interference, and energy depletion. Under the random failure threat, Meta-RL achieves the best connectivity (approximately 0.44), task completion rate (approximately 0.48), and network efficiency (approximately 0.49); DQN is approximately 0.26, 0.42, and 0.40 respectively; MARL is approximately 0.16, 0.39, and 0.34 respectively; and the baseline is approximately 0.07, 0.36, and 0.27 respectively. Under the targeted attack threat, Meta-RL achieves the highest connectivity (approximately 0.38, lower than other methods but still the highest), DQN (approximately 0.24), MARL is approximately 0.13, and the baseline is approximately 0.05, indicating that targeted attacks have the greatest impact on all methods, but Meta-RL still maintains a significant advantage. Under environmental interference threats, Meta-RL effectively compensated for link quality degradation through link optimization and power control, achieving a connectivity rate of approximately 0.42 and a task completion rate of approximately 0.48. Under energy depletion threats, Meta-RL demonstrated its most significant advantages through the synergistic effect of intelligent hibernation and global energy management, achieving a connectivity rate of approximately 0.43 and a network efficiency of approximately 0.48. Across all four threat scenarios and the three key performance indicators, the performance ranking was Meta-RL > DQN > MARL > benchmark, validating the versatility and robustness of the proposed method.

[0110] like Figure 6 As shown, Figure 6 This paper compares the performance curves of three methods across six dimensions: connectivity, robustness, transmission efficiency, energy efficiency, load balancing, and task completion rate. In connectivity, Meta-RL's curve is significantly higher than DQN and MARL, maintaining a stable upward trend over time. In robustness, Meta-RL exhibits the best anti-interference capability, with the smallest curve fluctuation. In transmission efficiency, Meta-RL significantly outperforms the two compared methods in the later stages of training. In energy efficiency, Meta-RL's advantage gradually widens in the later stages, thanks to the synergy of intelligent sleep and energy management actions. In load balancing, MARL performs best due to the naturally distributed load caused by independent decisions of each node, followed by Meta-RL, which achieves a more optimized load distribution through global coordination; DQN performs the worst. In task completion rate, Meta-RL maintains the highest level throughout. Overall, Meta-RL achieves the best performance in five of the six dimensions, only slightly lagging behind MARL in load balancing.

[0111] On the other hand, this invention provides a deep space network resilience optimization system based on CTDE meta-reinforcement learning, whose modules can implement the steps of the aforementioned method, specifically including:

[0112] The network modeling module is used to construct a deep-space heterogeneous time-varying network model, including multiple types of heterogeneous nodes, dynamic communication links, node energy consumption models, and node motion models. The indicator system module is used to establish an indicator system for evaluating network resilience from three dimensions: structural layer, functional layer, and task layer. The threat model module is used to define a set of threat models, including random failures, targeted attacks, environmental interference, and energy depletion. The policy parameter module is used to define continuous policy parameters for each node, consisting of link rerouting tendency parameters, sleep mode strength parameters, and power adjustment amplitude parameters. The centralized training module is used to utilize the global state of the deep-space heterogeneous time-varying network model. Information is injected into any threat in the threat model set with a preset probability. A global comprehensive reward signal is calculated based on the weighted synthesis of the three-layer resilience index system. Based on the change in the global comprehensive reward signal, a meta-learning mechanism is used to perform gradient updates on the continuous policy parameters of each node. A distributed execution module is used to enable each node to independently execute link quality enhancement actions, intelligent sleep actions, adaptive power control actions, energy management actions, and load balancing actions, relying only on the locally updated continuous policy parameters. A result output module is used to output the optimized network configuration parameters and resilience assessment results under various threat scenarios.

[0113] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned deep space network anti-damage optimization method based on CTDE meta-reinforcement learning.

[0114] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned deep space network anti-damage optimization method based on CTDE meta-reinforcement learning.

[0115] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing the resilience of deep space networks based on CTDE meta-reinforcement learning, characterized in that, include: Step S1: Construct a deep-space heterogeneous time-varying network model; Step S2: Establish a three-layer resilience indicator system, including structural layer indicators, functional layer indicators, and task layer indicators; wherein, the structural layer indicators include connectivity rate, average degree, clustering coefficient, maximum connected component ratio, and degree variance; the functional layer indicators include average transmission latency, packet loss rate, network efficiency, and energy efficiency; and the task layer indicators include task completion rate, task latency, load balancing degree, and outage probability. Step S3: Define a set of threat models, including random failure threats, targeted attack threats, environmental interference threats, and energy depletion threats; Step S4: Define a set of continuous policy parameters for each node in the deep space heterogeneous time-varying network model. The continuous policy parameters include link rerouting tendency parameters, sleep mode strength parameters, and power adjustment magnitude parameters. The link rerouting tendency parameters are used to control the intensity of the node's behavior in optimizing link quality. The sleep mode strength parameters are used to control the intensity of the node entering a low-power state when energy is insufficient. The power adjustment magnitude parameters are used to control the magnitude of the node's adaptive adjustment of the communication range according to the energy state. Step S5: Using the global state information of the deep space heterogeneous time-varying network model, inject any threat from the threat model set with a preset probability. Calculate the global comprehensive reward signal based on the weighted synthesis of the three-layer resilience index system. Based on the average reward change between the current training round and the previous training round, update the continuous policy parameters of each node using the product of the preset meta-learning rate and Gaussian exploration noise as the update step size, and restrict the updated continuous policy parameters to a preset value range. Step S6: Without the need for global information or inter-node communication, each node independently executes link quality enhancement actions, intelligent sleep actions, adaptive power control actions, energy management actions, and load balancing actions, relying only on the locally continuous policy parameters updated by gradients. Step S7: Output the network configuration parameters obtained after each node independently executes the corresponding action, and the resilience assessment results under various threat scenarios based on the three-layer resilience index system.

2. The deep space network resilience optimization method based on CTDE meta-reinforcement learning according to claim 1, characterized in that, In step S1, the network model includes multiple types of heterogeneous nodes, dynamic communication links, node energy consumption model, and node motion model.

3. The deep space network resilience optimization method based on CTDE meta-reinforcement learning according to claim 1, characterized in that, In step S4, the link rerouting tendency parameter, the sleep mode strength parameter, and the power adjustment amplitude parameter all have the same value range; the continuous policy parameter compresses the node decision space from a high-dimensional discrete state-action space to a low-dimensional continuous parameter space. For a network with N nodes, the total number of parameters to be optimized is three times N, and the policy search space is of linear order N.

4. The deep space network resilience optimization method based on CTDE meta-reinforcement learning according to claim 1, characterized in that, In step S5, the global comprehensive reward signal is calculated by assigning preset weight coefficients to connectivity rate, task completion rate, maximum connectivity component ratio, normalized positive value of average transmission delay, load balance degree and proportion of inactive nodes, and then performing weighted summation.

5. The deep space network resilience optimization method based on CTDE meta-reinforcement learning according to claim 1, characterized in that, In step S5, the gradient update based on the average reward change between the current training round and the previous training round specifically involves: obtaining the difference between the average reward of the current training round and the average reward of the previous training round; multiplying the preset meta-learning rate by the difference and then by Gaussian exploration noise following a preset distribution to obtain the parameter update amount; adding the continuous policy parameters of the current round to the parameter update amount to obtain the updated continuous policy parameters; and restricting the updated continuous policy parameters to a preset value range of the continuous policy parameters as the continuous policy parameters for the next training round.

6. The deep space network resilience optimization method based on CTDE meta-reinforcement learning according to claim 1, characterized in that, In step S6: The link quality enhancement action is driven by the link rerouting tendency parameter and is used to enhance the quality of the best neighbor link and weak links; the intelligent sleep action is driven by the sleep mode strength parameter and is used to reduce the load and shrink the communication range when the node energy is lower than the average level of the entire network; the adaptive power control action is driven by the power adjustment amplitude parameter and is used to asymmetrically expand or shrink the communication range according to the node energy level; the energy management action forcibly reduces the load and shrinks the communication range when the node's remaining energy is lower than the emergency threshold; the load balancing action transfers the load according to the neighbor energy order when the node load exceeds the threshold.

7. A deep space network resilience optimization system based on CTDE meta-reinforcement learning, characterized in that, include: The network modeling module is used to construct a deep space heterogeneous time-varying network model that includes multiple types of heterogeneous nodes, dynamic communication links, node energy consumption models, and node motion models. The indicator system module is used to establish a three-layer resilience indicator system to evaluate network resilience from three dimensions: structural layer, functional layer, and task layer. The structural layer indicators include connectivity, average degree, clustering coefficient, maximum connected component ratio, and degree variance. The functional layer indicators include average transmission latency, packet loss rate, network efficiency, and energy efficiency. The task layer indicators include task completion rate, task latency, load balancing, and outage probability. The Threat Model module is used to define a set of threat models, including random failures, targeted attacks, environmental interference, and energy depletion. The policy parameter module is used to define continuous policy parameters for each node, consisting of a link rerouting tendency parameter, a sleep mode strength parameter, and a power adjustment magnitude parameter. The link rerouting tendency parameter controls the intensity of the node's link quality optimization behavior; the sleep mode strength parameter controls the intensity at which the node enters a low-power state when energy is insufficient; and the power adjustment magnitude parameter controls the magnitude at which the node adaptively adjusts its communication range based on energy status. The centralized training module is used to utilize the global state information of the deep space heterogeneous time-varying network model to inject any threat from the threat model set with a preset probability, calculate the global comprehensive reward signal based on the weighted synthesis of the three-layer resilience index system, and perform gradient updates on the continuous policy parameters of each node based on the change of the global comprehensive reward signal using a meta-learning mechanism. The distributed execution module enables each node to independently execute link quality enhancement actions, intelligent sleep actions, adaptive power control actions, energy management actions, and load balancing actions, relying only on the locally continuous policy parameters updated by gradients. The results output module is used to output the optimized network configuration parameters and the resilience assessment results under various threat scenarios.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the deep space network anti-damage optimization method based on CTDE meta-reinforcement learning as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement the deep space network anti-destruction optimization method based on CTDE meta-reinforcement learning as described in any one of claims 1-6.

Images