Markov game-based satellite cluster observation resource allocation method and system

Abstract

The invention relates to a Markov game-based satellite cluster observation resource allocation method and system. Through a two-stage decomposition strategy, a multi-target balance problem of a task integrity rate, a resource utilization rate and a cost-efficiency ratio is effectively solved. In the first stage, a genetic algorithm (GA) is adopted to complete satellite-task-time window three-dimensional matching, and optimal distribution of limited visible windows is achieved. And in the second stage, a distributed decision-making mechanism is implemented based on an improved Improved-MADDPG framework, and after an intelligent agent adopts a random game strategy to execute exploration in a distributed manner, optimal dynamic configuration of observation resources is achieved through global information sharing and local strategy iteration, and key calculation indexes such as an average reward value and the like are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of satellite resource allocation, and relates to a satellite cluster observation resource allocation method and system based on Markov game. Background Art

[0002] In recent years, the innovative application of heuristic algorithms and the integration of multiple methods have become a research hotspot for satellite mission planning. In the field of satellite cluster collaborative scheduling, researchers have proposed a two-stage optimization strategy based on an improved genetic algorithm. This strategy introduces population perturbations and an elite elimination mechanism during the encoding phase to enhance global search capabilities. Subsequently, a dynamic resource conflict resolution model is established to address satellite resource competition under multiple constraints. This approach reduces computational complexity while achieving the coordinated optimization of task completion rate and resource utilization. With the further development of metaheuristic algorithms, researchers have innovatively constructed a metaheuristic-exact algorithm integrated model (EHE-DCF) within a divide-and-conquer framework. This two-stage optimization strategy addresses the task allocation challenge of large-scale satellite clusters, effectively balancing solution efficiency and solution optimality. In the area of dynamic autonomous decision-making, researchers have proposed a rolling scheduling framework with a hybrid trigger mechanism based on heuristic algorithms to meet the real-time response requirements of agile satellites. By establishing a dynamic task priority assessment model and a resource utilization prediction mechanism, this framework significantly improves the response speed of emergency tasks while ensuring the efficiency of routine tasks. To address the technical difficulties of dynamic observation of heterogeneous satellites, a dynamic collaborative scheduling model based on discrete particle swarm optimization (DPSO) has been proposed. By constructing a multi-dimensional fitness function, it provides a new paradigm for solving the problems of dynamic satellite-ground matching and optimal resource allocation. However, heuristic algorithms also have significant limitations. For example, they may lack adaptability in dynamic heterogeneous environments, are prone to falling into local optimality, and have difficulty effectively handling high-dimensional multi-objective collaborative optimization problems.

[0003] To address these challenges, hybrid intelligent optimization algorithms are emerging as a breakthrough. This paradigm, combining the advantages of data-driven and knowledge-guided approaches, integrates multiple algorithms to complement each other in areas such as model interpretability, dynamic multi-objective trade-offs, and heterogeneous resource coordination. For example, the proposed Data-Driven Improved Genetic Algorithm (DDIGA) combines artificial neural networks with genetic algorithms. Using artificial neural networks to pre-train the initial population of the genetic algorithm, it also incorporates frequent patterns and a competitive adaptive local optimization strategy, achieving promising results in dynamic and complex scenarios. Others have proposed a hybrid framework that combines deep reinforcement learning with heuristic rules. Deep reinforcement learning (DRL) is used to model the global dynamic decision-making for multi-antenna task allocation, while a heuristic algorithm is used to rapidly generate local optimal solutions for single-antenna task scheduling. Cross-layer iterative feedback between the two forms a closed-loop decision optimization, enabling efficient resolution of timing conflicts. This demonstrates the effectiveness of hybrid intelligent algorithm fusion in observation coordination and scheduling. However, these traditional techniques still struggle to achieve optimal dynamic allocation of autonomous observation resources for heterogeneous satellite clusters. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional technologies, the present invention proposes a satellite cluster observation resource allocation method based on Markov game and a satellite cluster observation resource allocation system based on Markov game, which can achieve the optimal dynamic configuration of autonomous observation resources of heterogeneous satellite clusters.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a method for allocating satellite cluster observation resources based on Markov game is provided, comprising the steps of: Obtain resource parameters of a heterogeneous satellite cluster; resource parameters include the number of satellites, tasks to be performed, satellite type, and visibility time window; A mixed integer programming model based on resource scheduling requirements of autonomous observation of heterogeneous satellite clusters is called according to resource parameters; The mixed integer programming model is optimized for the first phase of tasks, satellites, and visible time windows using a genetic algorithm to generate an initial task allocation plan that matches tasks with satellites. Sort the initial task allocation plan according to the order in which the tasks arrive, and generate a complete initial task execution plan for each satellite; The Improved-MADDPG deep reinforcement learning framework based on genetic algorithm is used to guide target optimization with the complete initial mission execution plan as prior knowledge to obtain the optimal task allocation plan for heterogeneous satellite clusters. In the Improved-MADDPG deep reinforcement learning framework, satellites are regarded as intelligent agents and the task execution process is modeled as a Markov decision process. The intelligent agents use a random game strategy to perform distributed exploration and then converge global information through centralized decision-making. The optimal task allocation plan is the observation resource allocation plan that achieves the optimal mission yield, completion rate and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

[0006] On the other hand, a satellite cluster observation resource allocation system based on Markov game is also provided, including: The resource acquisition module is used to obtain the resource parameters of the heterogeneous satellite cluster; the resource parameters include the number of satellites, the tasks to be performed, the satellite type and the visible time window; The model calling module is used to call the mixed integer programming model built based on the resource scheduling requirements of the autonomous observation of heterogeneous satellite clusters according to resource parameters; The first allocation module is used to jointly optimize the tasks, satellites and visible time windows in the first stage of the mixed integer programming model through a genetic algorithm to generate an initial task allocation plan that matches tasks with satellites; The program sorting module is used to sort the initial task allocation plan according to the order of task arrival and generate the complete task execution initial plan for each satellite; The second allocation module is used to utilize the Improved-MADDPG deep reinforcement learning framework based on genetic algorithms, using the complete initial mission execution plan as prior knowledge to guide target optimization and obtain the optimal task allocation plan for the heterogeneous satellite cluster. Specifically, within the Improved-MADDPG deep reinforcement learning framework, satellites are regarded as intelligent agents and the task execution process is modeled as a Markov decision process. The intelligent agents use random game strategies to perform distributed exploration and then converge global information through centralized decision-making. The optimal task allocation plan is the observation resource allocation plan that achieves the optimal task yield, completion rate and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

[0007] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned Markov game-based satellite cluster observation resource allocation method and system effectively balances mission completion rate, resource utilization, and cost-effectiveness through a two-stage decomposition strategy. In the first stage, a genetic algorithm (GA) is used to achieve three-dimensional matching of satellite, mission, and visible time window, achieving optimal allocation within the limited visible window. In the second stage, a distributed decision-making mechanism is implemented based on the improved Improved-MADDPG framework. Intelligent agents use a randomized game strategy to perform distributed exploration. After global information sharing and local policy iteration, they achieve optimal dynamic allocation of observation resources, significantly improving key computational metrics such as reward value and reward acquisition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 1 is a flow chart of a method for allocating satellite cluster observation resources based on Markov game in one embodiment; Figure 2 This is an overlapping diagram of time windows for multi-satellite Earth observation in one embodiment; Figure 3 Schematic diagram of a hybrid optimization algorithm framework based on a genetic algorithm and an improved multi-agent deep deterministic policy gradient (GA-Improved-MADDPG) in one embodiment; Figure 4 A schematic diagram of an improved architecture of the MADDPG algorithm in one embodiment; Figure 5 A schematic diagram of the basic framework of a multi-head attention mechanism in one embodiment; Figure 6 A schematic diagram of an independently designed research and experimental verification scenario in an embodiment; Figure 7 A schematic diagram of rewards under the same task scale in one embodiment; Figure 8 The profit acquisition rate cloud-rain graph of the Improved-MADDPG algorithm and its variants in one embodiment is shown below. Figure 8 (a) is the scale of 50 tasks, Figure 8 (b) The scale is 100 tasks; Figure 9 A comparison chart of the running times of different algorithms at various task scales in one embodiment; Figure 10Schematic diagram of the module framework of a satellite cluster observation resource allocation system based on Markov game in one embodiment. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the description of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0011] It should be noted that the reference to "embodiment" in this document means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the specification of the present invention and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0012] The following describes the implementation of the present invention in detail with reference to the accompanying drawings in the embodiments of the present invention.

[0013] In recent years, with the rapid development of "Internet + Satellite" technology, low-Earth orbit satellite constellation projects, represented by Starlink, have sparked a new revolution in the global space industry. These projects have not only achieved breakthroughs in global communications coverage through large-scale satellite deployments, but have also driven the rapid iteration of space technology with significant commercial value. However, as user needs become increasingly diverse and complex, core contradictions facing satellite systems have become increasingly prominent. This core contradiction manifests itself in the structural imbalance between the large-scale deployment of satellite constellations and the dynamic adaptability of observation resources. Existing research suffers from three cognitive flaws: First, inappropriate dimension selection, excessive focus on macro-variances such as satellite type and orbital parameters, while ignoring micro-parameter characteristics such as payload performance baselines, AI model iterations, and onboard algorithm baselines; second, ambiguous modeling methods, simplifying heterogeneous satellites into homogenized entities, resulting in significant errors in identifying capability gradients; third, rigid scheduling mechanisms, relying on manual rules and static scheduling of individual satellites, are subject to drawbacks such as frequent manual intervention, high mission latency, and insufficient resource utilization. These systemic flaws at these three levels collectively lead to a vicious cycle of "heterogeneous homogeneous processing, extensive resource scheduling, and exponentially declining efficiency," severely hindering the constellation system's evolutionary goal of achieving autonomous collaborative reconfiguration. Therefore, improving the autonomous planning capabilities of heterogeneous satellite clusters to resolve this supply-demand contradiction is urgent.

[0014] The Markov game-based satellite cluster observation resource allocation problem involves dynamically changing space mission scenarios. A satellite cluster system, comprised of satellites of various types and configurations, uses the autonomous decision-making capabilities of onboard intelligent agents to achieve the optimal dynamic match between observation resources and user mission requirements, while satisfying constraints. Satellite clusters offer distinct advantages over the fragmented scheduling of individual satellites. Satellite clusters possess three key characteristics: individuality, socialization, and evolutionary capabilities. Individuality emphasizes the differences in satellite types and configuration capabilities. Even for the same mission type, satellites exhibit heterogeneity, and homogenous treatment often leads to resource waste. Socialization emphasizes collaboration between satellites, achieving efficient coordination through distributed execution and centralized decision-making. Evolutionary capabilities indicate that the number of satellites will continue to increase in the future, requiring the system to be adaptable and scalable. The Markov game-based satellite cluster observation resource allocation problem consists of two main phases: time window allocation and mission observation. The visibility time window (VTW) refers to the time period during which a satellite is visible to a ground station during its orbit around Earth. During this timeframe, ground stations must be able to observe satellites and receive data or transmit commands, which is a key constraint in satellite observation. In the Markov game-based satellite cluster observation resource allocation phase, the key lies in leveraging the heterogeneity of satellites, rationally dividing labor and cooperating, and allocating limited observation resources appropriately without violating constraints to maximize mission benefits. This requires not only that the system recognize and exploit the heterogeneity between satellites, but also that, guided by collaborative game principles, satellites autonomously execute decisions and commands through a combination of distributed execution and centralized decision-making to successfully complete the observation mission.

[0015] Satellite observation resource scheduling is a typical NP-hard combinatorial optimization problem. Its complexity stems from the coupling of multiple dynamic and static constraints: it is necessary to coordinate basic constraints such as mission time window conflicts and observation station capacity limitations, and to cope with dynamic challenges such as inter-satellite communication delays, payload compatibility, and insufficient autonomous decision-making capabilities derived from satellite heterogeneity and clustering. It is also necessary to reconcile the inherent contradiction between the efficiency of a single satellite and the global benefits of the cluster. This multi-level constraint system makes it difficult to directly migrate the traditional single-satellite scheduling model to cluster observation scenarios. To address this complex problem, researchers have gradually overcome technical bottlenecks through innovative mathematical programming algorithms. In the field of collaborative scheduling of heterogeneous satellites, a hybrid algorithm based on branch-and-bound and heuristic pruning was proposed, constructing a multi-agent collaborative model to achieve efficient mission planning for heterogeneous satellites. This collaborative mechanism, combining distributed decision-making with global optimization, significantly improved overall system performance. To address the surge in communication and decision-making costs caused by the expansion of satellite clusters, network flow theory was innovatively introduced into the field of autonomous decision-making, proposing a minimum-cost flow dynamic optimization method based on a core network. This method achieves low-latency autonomous scheduling through real-time task supply and demand matching and a local update mechanism. Furthermore, to address the prominent contradiction between individual and group interests, a two-stage dynamic allocation framework was designed. Based on the minimum conflict set strategy, a multi-objective optimization model was established to achieve a dynamic balance between minimizing scheduling perturbations and maximizing observation benefits during the task insertion phase. Although current research trends indicate that the deep integration of precise algorithms and heuristic strategies, as well as the parallel innovation of multi-agent collaboration and network flow architectures, are providing systematic solutions to the "dynamic multi-constrained optimization" dilemma of satellite cluster observation, significant shortcomings remain. For example, due to the inherently high computational complexity of the problem, the computational time of traditional mathematical programming algorithms (such as branch-and-bound and integer programming) increases exponentially, making it difficult to meet the real-time requirements of dynamic scenarios. In heterogeneous satellite clusters, the introduction of collaborative constraints further exacerbates the problem of model dimensionality explosion. Secondly, faced with the conflict between model accuracy and the cost of simplification, mathematical programming methods often approximate some complex constraints, making overly idealized model assumptions and easily causing deviations between the model and the actual system. These shortcomings essentially reflect the inherent bottlenecks of mathematical programming methods in handling "dynamic, distributed, and multi-objective" coupled optimization problems, and urgently need to be integrated with new paradigms such as reinforcement learning and metaheuristics to break through theoretical boundaries.

[0016] Artificial intelligence-based optimization algorithms offer a new path to overcoming these bottlenecks. In the area of autonomous decision-making, a distributed scheduling method based on multi-agent deep reinforcement learning (MADRL) has been proposed. This method, using a multi-agent deep deterministic policy gradient (MADDPG) network, enables satellites to share decision-making policies rather than specific states or decision data, significantly reducing communication overhead and improving response speed. Compared to traditional contract network protocols (CNPs), this method reduces communication load while optimizing satellite resource utilization, offering enhanced real-time and robustness. Building on this, a framework for autonomous decision-making based on deep neural networks (DNNs) has been proposed. Its core idea is to replace manual rule design with data-driven approaches, leveraging the autonomous learning capabilities of neural networks to directly link task requirements with resource allocation strategies, achieving end-to-end autonomous optimization. This algorithm meets the real-time dynamic scheduling requirements of large-scale satellite constellations while reducing reliance on manual rules and demonstrating strong adaptability. It provides an efficient and reliable technical path for autonomous collaborative decision-making in complex scenarios. To address the challenges of integrating heterogeneous observation systems and optimizing resources, the problem has been modeled as a Markov decision process (MDP). The DQN algorithm, which combines deep neural networks with Q-learning, significantly optimizes resource utilization efficiency and alleviates resource conflicts in the coordinated scheduling of state-owned and commercial observation resources. Currently, with the exponential growth of low-Earth orbit constellations, a proposed deep reinforcement learning distributed scheduling algorithm has demonstrated significant advantages. This algorithm constructs a competitive-cooperative game model and, by dynamically sensing satellite operational status and resource competition, generates an observation link allocation scheme that satisfies spatiotemporal constraints. This improves response efficiency while adaptively adjusting resource allocation weights, providing a new technical paradigm for observation management in large-scale constellation clusters. While traditional supervised learning algorithms have demonstrated application value in specific areas, their adaptability to the dynamic operational environment of satellites remains significantly limited. They suffer from three major drawbacks: First, model training relies on massive amounts of historical data. The unpredictability of real-time on-orbit status can lead to a distributional shift between training and observed data, which in turn risks reduced model decision reliability. Second, the algorithm's high computational overhead conflicts with limited onboard computing power, which can lead to mission delays. Third, the black-box nature of deep networks impairs decision interpretability, posing a safety hazard for space missions requiring precise fault tracing.

[0017] In one embodiment, Figure 1 As shown, a method for allocating satellite cluster observation resources based on Markov game is provided, which may include the following steps S10 to S18: S10, obtaining resource parameters of the heterogeneous satellite cluster; the resource parameters include the number of satellites, tasks to be executed, satellite types, and visible time windows; S12, calling a mixed integer programming model based on the resource scheduling requirements of the autonomous observation of heterogeneous satellite clusters according to the resource parameters; S14, performing a first-stage joint optimization of tasks, satellites, and visible time windows on the mixed integer programming model using a genetic algorithm to generate an initial task allocation plan that matches tasks with satellites; S16, sorting the initial task allocation plan according to the order of task arrival, and generating a complete task execution initial plan for each satellite; S18 uses the Improved-MADDPG deep reinforcement learning framework improved by genetic algorithm, and uses the complete initial mission execution plan as prior knowledge to guide multi-objective Pareto optimization to obtain the optimal task allocation plan for heterogeneous satellite clusters. Among them, within the Improved-MADDPG deep reinforcement learning framework, satellites are regarded as intelligent agents and the task execution process is modeled as a Markov decision process. The intelligent agents use random game strategies to perform distributed exploration and then converge global information through centralized decision-making. The optimal task allocation plan is the observation resource allocation plan that achieves the optimal task yield, completion rate and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

[0018] It can be understood that the Heterogeneous Satellite Cluster Measurement & Control Resource Allocation Problem (HSCMC-RAP) based on Markov game in this specification is described as a cluster system composed of satellite nodes with homogeneous platforms but heterogeneous capabilities. In dynamic mission scenarios, for the application requirements of single-point observation targets (which can be completed with a single observation), it is necessary to rely on the autonomous collaborative decision-making capabilities of satellites to complete the dynamic adaptation and optimal configuration of observation resources under the conditions of meeting multi-dimensional constraints, so as to achieve efficient response to user tasks. The system description is as follows: Figure 2 As shown. Therefore, the core challenge of this problem is that when the mission objectives are exposed within the visible time windows of multiple satellites, a single optimization objective based solely on mission benefits may lead to improper time window selection, thereby causing waste of resources. Secondly, existing research has not paid enough attention to satellite clusters. In real observation scenarios, satellite systems present a complex relationship of competition and cooperation. Single satellites tend to maximize individual benefits by seizing high-value tasks, but this local optimal strategy may lead to an imbalance in task allocation and damage the overall benefits of the system. Especially when there are overlapping areas in the observation capabilities of multiple satellites, how to balance individual rationality and collective rationality becomes a key challenge to improve the observation efficiency of the system. Although such problems are common in practical applications, existing literature has not yet established an effective modeling framework or proposed corresponding algorithms to solve this target balance optimization problem.

[0019] The problem is modeled as a two-stage dynamic resource allocation problem. Observation resource allocation is decomposed into two phases: time window allocation and mission observation. These phases address the dynamic matching of visibility time windows (VTWs) and the optimal allocation of observation resources, respectively. A hierarchical optimization strategy is employed to enhance satellite decision-making capabilities. Furthermore, the capability differences between heterogeneous satellites are carefully considered in the modeling of heterogeneous satellite clusters. Quantitative analysis of satellite capability gradients effectively avoids the decision-making blind spots associated with categorizing heterogeneous scenarios as homogenous.

[0020] A hybrid algorithm combining a genetic algorithm with an improved Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm was proposed. This algorithm employs a two-stage optimization mechanism: In the first stage, the global search capability of the genetic algorithm is used to pre-optimize the initial policy space, generating a high-quality initial policy distribution for mission-satellite matching. In the second stage, multi-agent collaborative optimization is performed based on the improved MADDPG algorithm, optimizing objectives such as mission yield, completion rate, and energy consumption while satisfying observation resource constraints. This collaborative optimization mechanism effectively overcomes the inefficient exploration of large-scale state spaces by traditional reinforcement learning, achieving the complementary advantages of data-driven and prior knowledge-guided approaches.

[0021] Specifically, a systematic improvement scheme was proposed to address the limitations of the MADDPG algorithm, such as slow convergence and insufficient training stability. At the training optimization level, an efficient and stable training framework was constructed by integrating the Fixup parameter initialization method with the Adax adaptive optimizer. The Fixup initialization strategy abandons the traditional batch normalization operation and adopts a layered weight scaling mechanism to alleviate the model's gradient vanishing or convergence problems. The Adax optimizer dynamically adjusts the parameter update step size to effectively improve computational efficiency while maintaining training accuracy. Combining the advantages of attention mechanisms and residual learning, an Attention-Enhanced Residual Block (AERB) was introduced. This AERB implements dynamic feature filtering through a multi-head self-attention mechanism, enabling the network to focus on key information domains of the input data. Incorporating cross-layer skip connections, a deep feature propagation path was constructed, improving the model's representational capabilities while effectively alleviating the gradient vanishing problem in deep networks through a gradient direct connection mechanism. The MADDPG algorithm is designed to handle continuous state and action spaces, whereas the observation scheduling problem based on a heterogeneous satellite cluster is a complex integer programming problem with discrete decision and solution spaces. To meet practical application needs and address the problem of continuous space discretization, this implementation, based on the preprocessing principles of probability extreme value theory and utilizing the Gumbel-Softmax parameterization technique, superimposes Gumbel noise on the discrete action logits output by the policy network and performs temperature annealing. This achieves a continuous relaxation representation of discrete decisions, effectively addressing the action space mismatch problem of the basic MADDPG algorithm in discrete combinatorial optimization problems. Furthermore, a moving average algorithm is employed to smooth data fluctuations and improve the stability of convergence results.

[0022] At the system architecture level, a hybrid "distributed execution-centralized learning" architecture is proposed. The satellite implements autonomous decision-making (distributed execution) through an onboard agent, and then optimizes global returns through a centralized policy network (critic network). Furthermore, the problem is modeled as a sequential decision-making problem involving a Markov process, incorporating stochastic game theory. During the exploration phase, perturbations are performed by adding perturbation factors with annealing properties to simulate stochastic game behavior, thereby enhancing the diversity of policy search and effectively overcoming the drawback of the traditional MADDPG algorithm, which is prone to falling into local optimal solutions during policy search.

[0023] This Markov game-based satellite cluster observation resource allocation method effectively balances mission completion rate, resource utilization, and cost-effectiveness through a two-stage decomposition strategy. In the first stage, a genetic algorithm (GA) is used to construct a fitness function centered on cost-effectiveness, achieving a three-dimensional matching of satellite, mission, and time window, and achieving optimal allocation within the limited visibility window. In the second stage, a distributed decision-making mechanism is implemented based on the improved MADDPG framework. The intelligent agent uses a randomized game strategy to perform distributed exploration, then achieves optimal dynamic allocation of observation resources through global information sharing and local policy iteration. Compared to the baseline MADDPG algorithm, the improved algorithm demonstrates significant advantages in key metrics such as reward value and reward acquisition rate.

[0024] First, in terms of mathematical models, in the process of satellite cluster observation resource allocation based on Markov game, its working environment changes constantly over time, and the modeling process is extremely complex. In order to transform this practical problem into a mathematical model for easy analysis, the following assumptions are made: (1) The orbital dynamics environment exhibits piecewise stationary characteristics, and its dynamic changes can be modeled by discrete time series; (2) The state of the intersatellite link obeys a Markov random process, and its state transition probability matrix can be obtained a priori; (3) Each satellite node has the ability to perceive its own state and can obtain the working status of its own payload and the remaining energy information in real time; (4) The task is a single-point task, that is, each task must maintain the continuity of execution and cannot be reassigned, and the execution process is not allowed to be interrupted; (5) Satellite node resources have limited and non-renewable characteristics, and the storage capacity and energy reserves are dynamically consumed as the task is executed and cannot be replenished midway; (6) The failure rate between satellites and observation equipment is ignored; (7) Satellite capabilities are heterogeneous, which is manifested in the differences in the task execution cost matrix and is also subject to the constraints of single-task processing (each satellite can only process a single task per unit time period); (8) The satellite task switching process has a resource reconfiguration phase with a fixed time cost, and its duration is independent of the task type.

[0025] This hypothesis system characterizes the link dynamics through Markov processes and combines discrete-time modeling to handle the time-varying nature of the environment. While ensuring the solvability of the model, it effectively retains the main characteristics of the satellite resource allocation problem. Among them, assumptions (3)-(5) constitute the perception basis for autonomous decision-making, and assumptions (6)-(8) define the rigid constraints of the system, together constructing a theoretical analysis framework with engineering guidance value.

[0026] Based on the above assumptions, the mathematical model is given below. First, the necessary parameters are explained: Satellite, with S express, , where M represents the total number of satellites performing the mission; ,in, si It consists of a sextet; Indicates the satellite identifier, Represents a collection of heterogeneous capability satellites, whose satellite type belongs to one of the three heterogeneous capability satellites A, B and C. ABC belong to satellites of the same type but with different capabilities. The satellite capabilities are assumed to decrease in the order of C, B and A. For example, when performing the same mission The memory consumed will be few; Represents a set of time windows; N Represents the total number of tasks that need to be executed in a cycle, Indicates the Satellites for Task No. k time windows; Represents the set of all time windows of a task; Indicates whether the satellite is available in the current period, that is, whether the satellite is performing other tasks in the current period. ; Indicates the memory of the satellite; Indicates the remaining satellite observation time.

[0027] Task, use T express, ,in N Represents the total number of tasks that need to be executed in a cycle. ,in It consists of a sextet; An identifier representing the task; Indicates the location where the task occurs, consisting of the latitude and longitude where the task occurs; Represents the time attributes of a task, which consists of the start, end, and duration of the task; Represents the benefits of the task; Indicates the satellite storage consumed by the mission, indicating the amount of power consumed by A, B, and C type satellites when observing respectively; Indicates which satellite is performing the mission.

[0028] Therefore, to meet the needs of autonomous observation resource scheduling for heterogeneous satellite clusters, this specification constructs a mixed integer programming model that integrates an autonomous collaborative optimization mechanism. This model achieves mathematical abstraction of the spatiotemporal configuration of satellite observation resources through the design of multi-dimensional decision variables and constraint systems. Its core architecture can be described as follows: The decision variables are , Indicates the Satellites perform j The effective time of a task, that is, the effective time of task execution, The time when the task starts to execute effectively, is the end time of effective execution of the task. The complete optimization mathematical model (i.e., mixed integer programming model) is: (1) (2) (3) (4) (5) (6) (7) Among them, formula (1) is the target optimization function, which is a hybrid optimization model that pursues the dual requirements of maximizing benefits and minimizing costs, while taking into account the task completion rate. In order to facilitate the solution, the weight factor is introduced to perform a linear weighted combination of the two. Represents the cost-effectiveness of the task; thus taking into account multiple constraints; Formula (2) It is a specific calculation method of the weight factor. By using the adaptive adjustment mechanism based on the Sigmoid function to smooth the weight function, it can effectively avoid the influence of parameter mutation on system stability. Furthermore, by introducing the threshold parameter c By standardizing the Sigmoid function, we can precisely adjust the sensitivity of the weight to the target state change, thereby establishing a nonlinear mapping relationship between task completion rate and cost consumption, and achieving dynamic balance control in the collaborative optimization process. is a decision variable, representing Whether the execution is successful Formula (4) means that each task can only be executed once at most. Formula (5) means that each satellite can only execute one task at most at the same time. Formula (6) means that for the same satellite, the switching time between two adjacent tasks must be greater than or equal to , is a constant. Formula (7) shows that the time it takes for a satellite to perform a certain task is greater than the effective time of the task and the conversion time of the equipment. express.

[0029] Genetic algorithms, a classic metaheuristic optimization method, demonstrate significant advantages in solving NP-hard combinatorial optimization problems, such as satellite observation resource allocation, thanks to their swarm intelligence search mechanism and global optimization capabilities. This paper innovatively combines genetic algorithms with the Improved-MADDPG deep reinforcement learning framework to construct an efficient hybrid optimization paradigm. This approach uses genetic algorithms to pre-explore the problem space and generate high-quality initial solution distributions, providing prior knowledge guidance for subsequent reinforcement learning agents. This effectively reduces the dimensionality of the policy search and accelerates algorithm convergence.

[0030] The modeling process is as follows: This manual proposes the GA-IMADDP algorithm to address the problem of satellite cluster observation resource allocation based on Markov game. It divides the task allocation process into two parts. The first part is the allocation of tasks to satellites, which is solved by the GA algorithm; the other part is whether the task is observed, that is, how to use limited observation resources to maximize the mission benefits while minimizing costs and balancing the mission completion rate.

[0031] The first step is to build a GA algorithm. The GA algorithm is used to implement task allocation preprocessing, which is mainly divided into several parts: population initialization, encoding method, fitness function, selection operation, crossover operation and mutation operation.

[0032] In one embodiment, population initialization is a basic step of the genetic algorithm, and its quality directly affects the convergence efficiency of the algorithm. This embodiment uses the global random generation method to construct the initial population to ensure the diversity and coverage of the population. Specifically, for each task , first from its available satellite set Uniformly randomly select satellite numbers from , and then gather in the available time window of the selected satellite The time window index is randomly selected in [ ]. This process is repeated until an initial population of the desired number of individuals is generated. This dual randomization mechanism ensures spatial coverage of the initial solution while implicitly satisfying the hard constraints on satellite selection and time window.

[0033] In this embodiment, the encoding method uses a two-dimensional encoding strategy based on integers to construct a solution space representation. N The problem of allocating observation tasks is that the chromosome structure of each individual is N For the first A task, whose genetic encoding form is defined as an ordered pair , among which , ; , ; , ;i Indicates the selected satellite number. Indicates the Satellites for Task No. k time windows. Therefore, the complete chromosome is represented as , its essence is to realize the joint encoding of task-resource-time window through two-dimensional discrete space mapping.

[0034] In this embodiment, the fitness function of the genetic algorithm takes minimizing memory consumption as the core goal and considers the problem of satellites with heterogeneous capabilities. For the same set of observation tasks, the cost paid by the satellites is , the three satellite capabilities belong to heterogeneous capabilities. Therefore, the specific form of the fitness function is: (8) in, Indicates the memory consumption of the mission on the satellite, is the time window conflict penalty. Expressed as a penalty coefficient, it controls the time window conflict penalty term, affecting its contribution to the objective function. Two core constraints must be met when determining feasibility: the mission observation duration must be within the visible time window of the satellite and the ground station. Second, the mission time windows of the same satellite must not overlap. The infinite penalty term effectively eliminates illegal solutions and guides the search towards optimization within the feasible region.

[0035] In this embodiment, the selection operation uses a tournament selection mechanism to balance selection pressure and population diversity. The specific process is as follows: appropriate individuals are randomly selected from the population to form a tournament group, which is then sorted in ascending order of fitness (prioritizing individuals with lower memory usage). The top two best individuals are selected as parents for genetic operations. This process is repeated until the required mating pool size is met. This strategy retains high-quality individuals while maintaining population diversity through random sampling, avoiding premature convergence.

[0036] In this embodiment, the crossover operation promotes the recombination of gene segments by designing a single-point crossover operator. Specifically, a crossover point is randomly generated. , the parent chromosome and exchange d The gene segment after the dot generates two offspring individuals: (9) (10) The crossover operation explores new combination patterns of task assignment sequences by exchanging gene segments.

[0037] In this embodiment, the mutation operation enhances the local search capability by introducing a directed mutation strategy. , randomly select a new satellite and reselect the satisfied window index from its corresponding time window set. The mutation process adopts a constraint-aware mechanism to ensure that the newly generated solution is always within the feasible region, thereby improving the search efficiency and preventing the algorithm from converging to a local optimal solution too early during the search process.

[0038] Therefore, the specific operation process of the GA algorithm is as follows Figure 3 As shown in the figure, the genetic algorithm proposed in the first phase constructs a feasible initial task allocation scheme through the joint optimization of mission, satellite and visible time window. This scheme can provide prior knowledge guidance for the reinforcement learning algorithm based on improved multi-agent deep deterministic policy gradient (Improved-MADDPG), effectively alleviate the dimensionality curse problem faced by reinforcement learning in the exploration of large action spaces, improve the real-time allocation and autonomous decision-making capabilities of satellites, and provide a foundation for reinforcement learning modeling.

[0039] Then comes the modeling of the Improved-MADDPG algorithm. The initial satellite-task assignment scheme obtained from the GA algorithm is sorted in the order of task arrival, and all task plans that need to be executed on each satellite can be obtained. Then the Improved-MADDPG algorithm is used to maximize the mission benefits under limited observation resources, while taking into account the mission completion rate and cost consumption. The MADDPG algorithm is a typical multi-agent reinforcement learning. With its centralized decision-making and distributed execution advantages, it is widely used in satellite observation resource allocation problems. The satellite cluster observation resource allocation problem based on Markov game is a typical combinatorial optimization problem. Its process can be modeled as a Markov decision process. The observation resources that the satellite can choose are only related to the current state and have nothing to do with its past schedulable state. This process can be established as a four-tuple , representing state, action, state transition probability and reward function respectively.

[0040] Based on the initial satellite-task assignment scheme obtained using a genetic algorithm (GA), the matching results are sorted according to the task arrival sequence using a time-series scheduling strategy, thereby generating a complete initial mission execution plan for each satellite. Based on this, this example proposes the Improved-MADDPG algorithm, which aims to achieve target optimization under conditions of limited observation resources: maximizing mission benefits while simultaneously improving mission completion rates and controlling observation costs. As a typical paradigm for multi-agent reinforcement learning (MARL), MADDPG has demonstrated significant application value in the field of satellite observation resource allocation due to its "centralized training-distributed execution" architecture. The Markov game-based satellite cluster observation resource allocation problem studied in this paper has the following characteristics: First, it is an NP-hard combinatorial optimization problem; second, its dynamic decision-making process satisfies the Markov property, meaning that the satellite's choice of observation resources depends only on its current state and is independent of its historical state. To this end, this paper considers satellites as agents and models the task execution process as a Markov decision process (MDP), formally defined as a four-tuple < S , A ,P,R>, where: S Represents the state space, characterizing the system environment and satellite status; A is the action space, describing whether to observe; P is the state transition probability function, describing the law of environmental evolution; R is the reward function, used to quantitatively evaluate the comprehensive benefits of resource allocation schemes.

[0041] The state space is defined as the agent's abstract representation of the system environment and satellite operating status, which contains multi-dimensional normalized feature vectors, specifically divided into satellite state and mission state. The state of each satellite agent is composed of a three-dimensional observation vector. ,in Represents the ratio of the remaining time of each satellite to the maximum observation time, that is, the normalized remaining time; is the normalization of the remaining storage, Indicates whether the satellite is performing other tasks in the current period. The state of each task is composed of a four-dimensional observation vector. , which represent which satellite executes each mission, mission duration, consumed storage and reward of the mission respectively.

[0042] The action space represents the agent's resource allocation decision for the task, which is designed as a binary discrete space , corresponding to the "reject" and "accept" tasks respectively. , the satellite will consume time and storage to perform the task and update its time window status; if the action , then the resource state remains unchanged and the action selection must satisfy the Markov property.

[0043] State transition function. State transition function Describes the process of environmental evolution, whose determinism is dominated by the task execution logic and resource consumption rules. Specifically, if you accept the task , the satellite will and Deductions and , and set the task time interval Add to the satellite time window queue, and then perform subsequent task allocation conflict detection. The resource status and time window remain unchanged, and the task feature vector is reset to the next task to be assigned. If the task is rejected , the resource status and time window remain unchanged, and the task feature vector is reset to the attributes of the next task to be assigned.

[0044] The reward function is used to quantify the overall benefits of the resource allocation plan and is a specific representation of the objective function. The reward function, also known as the single-step reward or immediate reward, is an important basis for guiding the learning direction of the algorithm. The reward function is as follows: (11) The reward function is composed of the current state The action taken by the agent Feedback with the environment, is the weight factor, represents the number of tasks accepted by the satellite, and the hyperparameter c controls the intensity of the collaborative penalty. This function limits the scale of collaboration to a reasonable range through nonlinear mapping. When , the weight approaches 1, indicating that continuing to increase the number of tasks will have little effect on the weight, and the reward function tends to be stable, and vice versa; It represents the cost-effectiveness of the mission. Introducing this ratio into the denominator can suppress the reward value of tasks with high storage consumption and low benefits, and guide satellites to give priority to tasks with higher cost-effectiveness.

[0045] In this way, the above-mentioned fusion algorithm based on genetic algorithm and improved multi-agent deep deterministic policy gradient (GA-Improved-MADDPG) is proposed to solve the problem of dynamic allocation of autonomous observation resources of heterogeneous satellite clusters.

[0046] Figure 3The figure shows the hybrid optimization framework of GA-Improved-MADDPG. The system workflow is divided into two phases. In the first phase, a set of user mission requirements within a unit period is collected. Combined with schedulable observation resources, a set of time window resources subject to multidimensional constraints is generated through spatiotemporal constraint analysis. A genetic algorithm, acting as a preprocessing module, constructs a fitness function aimed at minimizing the overall cost. It then performs a global search and iterates the elite retention strategy in the solution space. The output is a matching relationship between mission, satellite with heterogeneous capabilities, and visible time window, which serves as the initialization scheme for the Improved-MADDPG algorithm.

[0047] During the reinforcement learning optimization phase (Phase II), the classic MADDPG architecture was improved, adopting a centralized training and distributed execution (CTDE) architecture. This architecture incorporates Markov game theory for strategy exploration, and each satellite is configured with an independent actor network to enable autonomous decision-making. First, to adapt to the discrete decision-making nature of observation resource allocation, a Gumbel-Softmax-based reparameterization method was utilized, which probabilistically samples and discretizes the continuous action space, effectively addressing the compatibility issue between the policy network output and the discrete action space. Second, global state information was shared through a critic network, and an exponential moving average function was introduced to smooth the immediate reward signal, significantly improving the algorithm's convergence stability. At the network level, a residual neural network structure based on an attention enhancement mechanism is introduced, combined with a fix-up initialization method and the Adax optimizer to improve the network's convergence speed. Considering the characteristics of multi-agent games, each satellite corresponds to an independent individual with an observation mission to perform. Given the limited total amount of observation resources, the resource strategy selected by each satellite affects the available resources of other satellites, leading to potential competition between satellites. However, because the common goal of satellite observation allocation is the same, the satellites are essentially in a competitive and cooperative relationship. This behavior is simulated by designing a stochastic game-driven multi-agent interaction mechanism to facilitate the iterative solution of the Bellman optimal equation and ultimately form a global state value evaluation function.

[0048] Among them, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a traditional reinforcement learning algorithm for multi-agent systems. This algorithm is based on the Actor-Critic (AC) framework, and its core design concept lies in the synergy of centralized training and decentralized execution. Specifically, each agent independently maintains an actor network with autonomous decision-making capabilities. A centralized critic network is introduced during the training phase, which integrates global state information and the joint action space to evaluate the state-action value function. This architectural design effectively addresses the environmental non-stationarity problem inherent in multi-agent systems. The global value function guides the policy optimization direction of each agent, maintaining decision-making autonomy while ensuring system-wide synergy. The specific implementation mechanism can be summarized as follows: Each agent, guided by the global Q-value provided by the critic network, concurrently updates the actor network parameters using a policy gradient method. Simultaneously, the critic network optimizes its parameters using a temporal difference method. Ultimately, through alternating iterations, policy optimization is achieved progressively, as follows: Policy function. It is defined as the mapping from state to behavior, using The essence is that in a given state s Behavior set in the case The probability distribution function of is expressed as follows: (12) in, is the probability distribution function. This algorithm determines the action strategy. For deterministic strategies, this function degenerates into a mapping: (13) The strategy is uniquely determined by the actions.

[0049] As a core element of reinforcement learning theory, the expected cumulative return function is designed to overcome the limitations of the immediate reward function. The immediate reward function only represents the agent’s state. s Execute an action a Transfer to state s ', this method has inherent similarities with the local optimal selection mechanism of the Greedy Algorithm, which can easily lead to the strategy optimization falling into a local convergence state. In view of this, by defining the discounted cumulative return: (14) in, is the current agent, , Represents the expectation of the joint distribution of state and action; is a discount factor that weighs the importance of current rewards against future rewards. The closer it is to 0, the more emphasis is placed on recent rewards, and vice versa.

[0050] Value function. The cumulative expected return function can be mathematically formalized into the state value function and the state-action value function as follows: (15) (16) Where (15) is the state value function, (16) is the state-action value function. (Bellman) formula for temporal recursive modeling: (17) (18) in Indicates that from Departure arrive The state transition probability.

[0051] Strategy update. This mechanism is the core module of the algorithm, and its implementation process follows the distributed strategy optimization paradigm. The algorithm builds a strategy actor network for each satellite agent, and the design parameters are , , M is the number of satellites. The strategy is updated using the stochastic gradient descent method: (19) is the standard expected discounted return, is the logarithmic gradient of the strategy, reflecting the sensitivity of action selection to parameters. D Experience Replay Buffer is an experience replay buffer that stores trajectory data when it reaches the preset capacity threshold. , from which importance sampling is performed to obtain batch data: . is the state-action value function, which is implemented by minimizing the temporal difference error: (20) in, is the target network parameter, It is an experience sampling trajectory, and a soft update mechanism is used to maintain training stability. It effectively solves the coupling oscillation problem between policy evaluation and policy improvement. This method updates the parameters of the target action network and the target policy network as follows: (twenty one) (twenty two) Sampling soft update method, is a hyperparameter, usually taken as ( , such as 0.001). This mechanism ensures a smooth transition of target value estimation by weighted averaging of online network and target network parameters. It is generally a constant. By using the soft update method, the target network parameters can be changed less, thus making the algorithm converge stably. Experimental studies have shown that when When the value approaches 0, the target network is approximately frozen, which can be regarded as a special form of delayed update; When it approaches 1, it degenerates into real-time synchronous update of the online network.

[0052] To address the limitations of the basic MADDPG algorithm, this study proposes an improved multi-agent reinforcement learning framework, Improved-MADDPG. Its core improvement strategies include the following aspects: Considering the particularity of multi-satellite observation scenarios and the constraints of limited observation resources, each satellite, as an independent decision-maker, will have its resource selection strategies form a mutually influential game relationship. To this end, a heterogeneous agent collaborative decision-making model based on stochastic game theory was constructed. By simulating the perturbations of other agents in selecting observation resources, a value assessment system that meets the Bellman optimality condition was established, thereby exploring more practical observation strategies.

[0053] At the network level, a residual neural network structure enhanced by the attention mechanism is proposed, and the Fix-up parameter initialization method and Adax adaptive optimizer are used to accelerate the model convergence process.

[0054] In view of the discrete nature of the action space in the observed resource allocation task, a Gumbel-Softmax reparameterization technique is used to construct a discrete-continuous hybrid action space mapping mechanism. Probabilistic sampling is used to discretize the policy network output, effectively resolving the compatibility issues of traditional continuous action strategies in discrete decision-making scenarios.

[0055] The following are some of the innovations mentioned above: (1) The Markov stochastic game (Stochastic Game), an important model in dynamic game theory, was proposed in 1953. It mathematically describes the interaction process of multiple agents by constructing multiple state transition functions. This model essentially extends the Markov decision process (MDP) to multi-agent scenarios and effectively characterizes the strategic interaction relationship between agents by integrating matrix game theory. Compared with the single-agent assumption of traditional MDP, stochastic game is particularly suitable for dynamic multi-agent collaboration scenarios. Its core advantage lies in its ability to accurately represent the joint impact mechanism of agent strategy selection on state transition probability and immediate benefits.

[0056] Based on this theoretical framework, this study establishes a partially observable stochastic game model for the dynamic allocation of autonomous observation resources for heterogeneous satellite clusters. The complex characteristics of this problem are mainly reflected in: (1) the dynamic and unpredictable space environment makes the system state only partially observable; (2) the resource competition and collaboration between heterogeneous satellites form a nonlinear coupling relationship; (3) the real-time interaction of multi-agent strategies triggers the need for dynamic evolution. To this end, the stochastic game quintuple constructed in this paper is: (twenty three) in, N is the set of intelligent agents, is the joint state space, is the action space of a single individual, is the state transition function, is the joint action of all agents, from Departure for joint action arrive The probability distribution of For all agents in Take joint action The joint reward obtained. It includes the agent set, joint state space, individual action space, state transition function and joint reward function. The specific design steps are as follows: First, explore the perturbation, which is uniformly distributed Applying action offset term constitutes, which explores It shows exponential decay with the increase of training rounds, thus effectively balancing the initial exploration and the later convergence process.

[0057] Secondly, perform game perturbations and construct the game operator: (twenty four) in Represents the players (agents) in the game, For intelligent agents j right i The influence weight of is dynamically adjusted by the intensity of resource competition. Representing an agent j The original action value, Represents Gaussian random noise with a mean of 0 and a standard deviation of 0.1, simulating the uncertainty of decision-making under incomplete information. Finally, the actions are normalized to ensure the feasibility of the actions, thereby enhancing the stability of the algorithm.

[0058] (2) Optimizing the network architecture and training mechanism of the MADDPG algorithm, the algorithm systematically improves the MADDPG algorithm from three dimensions: neural network architecture design, parameter initialization method, and optimizer selection. Figure 4 As shown in the figure, the details are as follows: The MADDPG algorithm is prone to falling into the disadvantage of local optimal solution during the strategy search process. By adaptively adjusting the perturbation intensity through the annealing mechanism, the premature convergence of the algorithm is significantly delayed, thereby increasing the probability of finding the global optimal strategy. to Indicates the corresponding value.

[0059] In one embodiment, a residual neural network structure enhanced by an attention mechanism is adopted in the Improved-MADDPG deep reinforcement learning framework, and the model convergence is accelerated through the Fix-up parameter initialization method and the Adax adaptive optimizer.

[0060] Specifically, the residual network design based on attention enhancement. The traditional MADDPG algorithm uses a two-layer fully connected neural network as the basic architecture of the policy network. This design has two significant defects: first, the parameter space of the fully connected layer grows squared with the input dimension, and the computational efficiency is significantly reduced when processing high-dimensional state space, which is not suitable for the spatial state of the heterogeneous satellite observation cluster; second, the fixed-structure fully connected layer is difficult to effectively capture the dynamic interaction relationship between multiple agents, especially in partially observable environments, which is prone to policy degradation problems. To this end, this paper proposes the introduction of an attention-enhanced residual network (ARN), whose basic framework is as follows: Figure 5 As shown in Figure 2, its advantages are reflected in the following two aspects. First, a dynamic feature extraction module is constructed through a multi-head attention mechanism, enabling each agent to adaptively adjust its attention weight to the observation information of other agents based on the current environment state. In partially observable collaborative task scenarios, this helps improve the efficiency of capturing key state features. Second, a deep residual structure is adopted to replace the traditional fully connected layer, and skip connections are used to effectively alleviate the vanishing gradient problem in deep network training.

[0061] Application of the Fix-up Initialization Method. Neural network parameter initialization has a significant impact on the model's convergence speed and generalization performance. To address the problem that traditional initialization methods (such as random initialization and Xavier initialization) are prone to causing gradient anomalies in deep networks, a research team has proposed a new solution called Fix-up initialization. This technology, through a unique parameter scaling mechanism, achieves two major breakthroughs without relying on batch normalization: first, it ensures that the output variance of each residual module remains stable during the initialization phase, effectively avoiding gradient explosion and vanishing gradient phenomena; second, by dynamically adjusting the weight scale of the residual branch, it significantly improves the convergence efficiency of deep networks, making it very suitable for residual network training.

[0062] Adaptability improvement of Adax optimizer. In response to the non-stationary optimization challenges commonly found in deep reinforcement learning, Adax provides a systematic solution to the limitations of the mainstream Adam optimizer currently used by most neural networks by innovatively invoking the second-order moment estimation reconstruction mechanism. The core breakthrough of this existing algorithm is to introduce an inverse time-series weighting strategy with an exponential decay factor and establish a second-order momentum estimation model with a long-range memory effect. This effectively overcomes the momentum estimation bias of Adam when the gradient exhibits exponential decay characteristics, and reduces the parameter adjustment complexity of multi-target collaborative scenarios from [1]. Dimensionality reduction to , significantly improving the deployment efficiency of large-scale distributed systems.

[0063] In one embodiment, the Gumbel-Softmax function is used in Improved-MADDPG deep reinforcement learning to construct a discrete-continuous hybrid action space mapping mechanism.

[0064] To address the inherent limitation of the original MADDPG algorithm, which is non-differentiable in discrete action spaces, the improved algorithm utilizes the Gumbel-Softmax function to effectively address the adaptability challenges of continuous policy gradient algorithms in discrete action spaces. To address the convergence difficulties caused by high variance in reward signals in multi-agent allocation, the algorithm dynamically smooths immediate rewards by constructing an exponential moving average function, giving greater weight to recent rewards, suppressing long-term fluctuations while maintaining sensitivity to recent state changes.

[0065] In one embodiment, the immediate reward is dynamically smoothed by constructing an exponential moving average function in the Improved-MADDPG deep reinforcement learning framework.

[0066] As can be understood, the improved algorithm addresses the convergence oscillation problem caused by high variance in reward signals in multi-agent allocation by proposing a reward correction mechanism based on time-series decay. By constructing an exponential moving average (EMA) function to dynamically smooth immediate rewards, it gives higher weight to recent rewards. This suppresses long-term fluctuations while maintaining sensitivity to recent state changes. This makes it suitable for real-time streaming reward processing, eliminating the need to wait for a fixed window period to complete and adapting to dynamic changes in the environment.

[0067] In some embodiments, a series of simulation experiments are conducted to validate the effectiveness of the Improved-MADDPG algorithm. First, multiple experimental scenarios are designed and corresponding parameter values are set in the Improved-MADDPG algorithm. Then, the Improved-MADDPG algorithm is compared with the basic MADDPG algorithm in terms of profit margin and runtime.

[0068] Due to the lack of standard test cases for the satellite cluster observation resource allocation problem based on Markov game, this study independently designed experimental verification scenarios, such as Figure 6 As shown. The experimental design uses real satellite orbital parameters to build a simulation environment. The satellite data source is the publicly released Starlink satellite TLE data. The satellite dynamic characteristics are obtained by analyzing its orbital parameters (TLE). In order to ensure the orbital heterogeneity of the satellite cluster, a sampling method is used to select 10 satellites operating in different orbital planes from the Starlink constellation, approximately 15 to 16 orbits, to establish an experimental satellite group numbered 1-10. The time is set from 3 Dec 2024 04:00:00.000 to 4 Dec 2024 04:00:00.000. This method based on the discretization of orbital parameters can effectively avoid the problem of convergence of orbital characteristics of adjacent satellites, thereby ensuring the differences in key parameters such as orbital altitude, inclination, and ascending node right ascension of the experimental samples, providing a statistically significant test environment for verifying the effectiveness of the resource allocation algorithm. The specific selected satellite numbers and TLE parameters are shown in Table 1: Table 1

[0069] Secondly, the basic attribute parameters of all experimental tasks strictly follow the following standardized settings: (1) Geographical location. It is determined by both longitude and latitude coordinates, where latitude is randomly generated uniformly in the range of 3° to 53° north latitude, and longitude is uniformly distributed in the range of 74° to 133° east longitude.

[0070] (2) Arrival time. The arrival time of the task adopts the 24-hour time system and is discretely modeled by converting it to second-level accuracy. The specific value is obtained by uniform random sampling in the interval [0, 86400] seconds.

[0071] (3) Duration. The observation requirement is set as a single observation mode of a single point target, and the observation duration follows a discrete uniform distribution of 3 to 6 seconds.

[0072] (4) Observed returns. Observed returns are generated using the Monte Carlo method and are uniformly randomly assigned in the integer range of 1-10 to quantify the differences in returns between different tasks.

[0073] (5) Consumption and storage. In view of the heterogeneous characteristics of satellites, memory consumption parameters are modeled differently according to the processing capabilities of the satellite platforms: Type A satellites with weak processing capabilities have the largest memory consumption (5-10 units), followed by Type B satellites with medium processing capabilities (3-7 units), while Type C satellites with high performance achieve the lowest memory usage (2-5 units) through optimization algorithms. This parameter setting effectively reflects the heterogeneous capability characteristic of "the stronger the processing capability of the satellite, the less resource consumption during mission execution". The memory consumption ranges of the three types of satellites are randomly generated through discrete uniform distribution, and strictly meet the inclusion relationship that the consumption range of Type A satellites completely covers Type B, and Type B covers Type C.

[0074] Finally, the time window is generated and set. The generation of the time window is based on the collaborative calculation of the orbital dynamics model and the visibility analysis algorithm. The specific process is as follows: (1) The high-precision orbit predictor is called through the Connect module of the STK software, and the spatial geometric relationship between the satellite and the ground station is calculated using the J4 perturbation model. (2) The actual visible range of the ground station is corrected using terrain elevation data, and the ground station altitude threshold is set to 50 meters to meet actual deployment conditions.

[0075] (3) Traverse the spatiotemporal relationship between the satellite constellation and the mission geographic coordinates, and use the numerical integration method to solve the set of visible time intervals that meet the elevation angle constraint conditions.

[0076] (4) The calculation results are discretized and stored in the form of UTC time periods. Each access time window is represented as a continuous sequence of time-state tuples. The final output format is a two-dimensional array structure of [start time, end time].

[0077] This generation mechanism fully considers the multi-physical field coupling effects such as the Earth's rotation, orbital attenuation and terrain shielding, providing precise time-constrained input parameters for subsequent scheduling optimization.

[0078] Based on the above basic settings, we then set the hyperparameters needed in the Improved-MADDPG algorithm. The specific configuration strategy is shown in Table 2.

[0079] Table 2

[0080] This manual uses a controlled variable experimental design to systematically evaluate the performance of the Improved-MADDPG algorithm and its variants. The specific difference design is shown in the variant difference details in Table 3 below: Table 3

[0081] To systematically evaluate the effectiveness of the improved Improved_MADDPG algorithm, this study designed three control variants for its core components: ImprovedV1, ImprovedV2, and ImprovedV3 (see Table 3 for detailed module configurations). Among them, ImprovedV1 removed the random game module, ImprovedV2 removed the optimized components of the network structure, and ImprovedV3 completely restored the basic MADDPG framework. In the comparative experiments under the same task scale (see Figure 7 ), the three variants exhibited significant performance differences: ImprovedV3 exhibited the lowest overall effectiveness, ImprovedV2 outperformed ImprovedV1 in both convergence speed and stability, and ImprovedV achieved the best performance. Notably, ImprovedV1 showed the most significant performance improvement compared to the base version, fully demonstrating the key role of randomized game mechanisms in multi-agent collaborative environments. Further analysis showed that improvements to the optimizer and network structure resulted in a 75.7% performance improvement for ImprovedV2 compared to ImprovedV3, while the synergistic effect of the complete set of improved components ultimately enabled Improved_MADDPG to achieve the best experimental results.

[0082] In order to verify the generalization ability of the algorithm, the specific details can be as follows Figure 8 As shown in the figure, the profit acquisition rate of the four algorithms at scales of 50 and 100 is shown. The profit acquisition rate formula is as follows:

[0083] in Indicates the profit obtained from completing the task. is the total revenue of all tasks waiting to be observed. The cloud-rain graph allows you to see the specific details more intuitively. This visualization method combines multiple layers of data expression: the cloud-like shape in the upper part shows the distribution of the kernel density estimation curve, which intuitively presents the probability density distribution of the data; the thin line in the middle represents the box plot, which shows the median and interquartile range of the data; the raindrop graph in the lower part accurately displays each observation value in the form of discrete points, making the local density distribution clear at a glance. Comparative analysis shows that ImprovedV, the Improved_MADDPG algorithm proposed in this article, shows the highest profit acquisition rate in all scale scenarios, concentrated in the range of 0.5-0.8, while ImprovedV3 has the worst effect, especially in the 100-scale scenario shown in the right figure, where almost all of its data points are close to 0, indicating that the performance of the algorithm has seriously degraded in large-scale applications. Running time is also an important indicator for evaluating algorithm performance. Figure 9 As shown in Figure 2, ImprovedV sacrifices slightly more running time, but performs well in key indicators such as profit acquisition rate and rewards.

[0084] Secondly, the runtime was analyzed. Experimental results demonstrate that a Markov game-based satellite cluster observation resource allocation method based on the improved MADDPG algorithm effectively addresses the multi-objective balance problem of mission completion rate, resource utilization, and cost-effectiveness through a two-stage decomposition strategy. In the first stage, a genetic algorithm (GA) was used to construct a fitness function centered on cost-effectiveness, achieving a three-dimensional matching of satellites, missions, and time windows, and achieving optimal allocation within the limited visibility window. In the second stage, a distributed decision-making mechanism was implemented based on the improved MADDPG framework. Through global information sharing and local policy iteration, the optimal dynamic configuration of observation resources was achieved. Comparative experiments show that while the improved algorithm has a slightly longer runtime than its variant, it exhibits significant advantages in key metrics such as rewards and reward acquisition rate.

[0085] It should be understood that although the above process Figure 1 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0086] In one embodiment, Figure 10As shown, a satellite cluster observation resource allocation system 100 based on Markov game theory is provided, comprising a resource acquisition module 11, a model invocation module 13, a first allocation module 15, a solution sorting module 17, and a second allocation module 19. The resource acquisition module 11 is used to obtain resource parameters for a heterogeneous satellite cluster; these resource parameters include the number of satellites, tasks to be executed, satellite type, and visibility time window. The model invocation module 13 is used to invoke a mixed integer programming model based on the autonomous observation resource scheduling requirements of the heterogeneous satellite cluster, based on the resource parameters. The first allocation module 15 is used to perform a first-stage joint optimization of tasks, satellites, and visibility time windows on the mixed integer programming model using a genetic algorithm to generate an initial task allocation solution that matches tasks to satellites. The solution sorting module 17 is used to sort the initial task allocation solutions according to the order in which the tasks arrive, generating a complete initial task execution plan for each satellite. The second allocation module 19 is used to utilize the Improved-MADDPG deep reinforcement learning framework improved based on the MADDPG algorithm, and use the complete task execution initial plan as prior knowledge to guide target optimization to obtain the optimal task allocation plan for the heterogeneous satellite cluster; wherein, within the Improved-MADDPG deep reinforcement learning framework, the satellite is regarded as an intelligent agent and the task execution process is modeled as a Markov decision process. The intelligent agent adopts a random game strategy to perform distributed exploration and then converges global information through centralized decision-making. The optimal task allocation plan is an observation resource allocation plan that achieves the optimal task yield, completion rate and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

[0087] The aforementioned Markov game-based satellite cluster observation resource allocation system 100 effectively balances mission completion rate, resource utilization, and cost-effectiveness through a two-stage decomposition strategy. In the first stage, a genetic algorithm (GA) is used to achieve three-dimensional matching of satellite, mission, and visibility time window, achieving optimal allocation within the limited visibility window. In the second stage, a distributed decision-making mechanism is implemented based on the improved MADDPG framework. Intelligent agents use a randomized game strategy to perform distributed exploration. After global information sharing and local policy iteration, the optimal dynamic allocation of observation resources is achieved, significantly improving key computational metrics such as reward value and reward acquisition rate.

[0088] In one embodiment, the first-stage joint optimization of tasks, satellites, and visible time windows in a mixed integer programming model using a genetic algorithm includes: constructing an initial population using a global random generation method; constructing a solution space representation using an integer-based two-dimensional encoding strategy; performing calculations using a fitness function with the core goal of minimizing the memory consumption of the task on the satellite; performing selection operations using a tournament selection mechanism; performing crossover operations using a designed single-point crossover operator; and performing mutation operations by introducing a directed mutation strategy and adopting a constraint-aware mechanism.

[0089] In one embodiment, a residual neural network structure enhanced by an attention mechanism is adopted in the Improved-MADDPG deep reinforcement learning framework, and the model convergence is accelerated through the Fix-up parameter initialization method and the Adax adaptive optimizer.

[0090] Regarding the specific limitations of a satellite cluster observation resource allocation system 100 based on Markov game, please refer to the corresponding limitations of a satellite cluster observation resource allocation method based on Markov game above, which will not be repeated here.

[0091] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus DRAM (RDRAM), and DDR DRAM.

[0092] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. A satellite cluster observation resource allocation method based on Markov game, characterized in that: Including steps: Obtain resource parameters of a heterogeneous satellite cluster; the resource parameters include the number of satellites, tasks to be performed, satellite types, and visible time windows; Invoking a mixed integer programming model based on the resource scheduling requirements of the autonomous observation of a heterogeneous satellite cluster according to the resource parameters; The mixed integer programming model is optimized for tasks, satellites, and visible time windows in the first phase by using a genetic algorithm to generate an initial task allocation plan that matches tasks with satellites; Sorting the initial task allocation plan according to the order in which the tasks arrive, and generating a complete initial task execution plan for each satellite; The improved-MADDPG deep reinforcement learning framework based on the genetic algorithm is used to guide target optimization using the complete mission execution initial plan as prior knowledge to obtain the optimal task allocation scheme for the heterogeneous satellite cluster. Specifically, within the improved-MADDPG deep reinforcement learning framework, the satellites are regarded as intelligent agents and the task execution process is modeled as a Markov decision process. The intelligent agents use a random game strategy to perform distributed exploration and then converge global information through centralized decision-making. The optimal task allocation scheme is an observation resource allocation scheme that achieves the optimal task yield, completion rate, and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

2. The satellite cluster observation resource allocation method based on Markov game according to claim 1, characterized in that: The process of performing the first phase of joint optimization of the mission, satellites and time windows of the mixed integer programming model using a genetic algorithm includes: The initial population is constructed using the global random generation method; The solution space representation is constructed using an integer-based two-dimensional encoding strategy; The calculation is performed using a fitness function with the core goal of minimizing the memory consumption of the mission on the satellite; Use tournament selection mechanism for selection operation; Perform crossover operation through the designed single-point crossover operator; The mutation operation is performed by introducing a directed mutation strategy and adopting a constraint-aware mechanism.

3. The satellite cluster observation resource allocation method based on Markov game according to claim 1 or 2, characterized in that: In the Improved-MADDPG deep reinforcement learning framework, a residual neural network structure enhanced by the attention mechanism is adopted, and the model convergence is accelerated through the Fix-up parameter initialization method and the Adax adaptive optimizer.

4. The method for allocating satellite cluster observation resources based on Markov game according to claim 3, characterized in that: In the Improved-MADDPG deep reinforcement learning framework, the Gumbel-Softmax function is used to construct a discrete-continuous hybrid action space mapping mechanism.

5. The satellite cluster observation resource allocation method based on Markov game according to claim 3, characterized in that: In the Improved-MADDPG deep reinforcement learning framework, the immediate reward is dynamically smoothed by constructing an exponential moving average function.

6. A satellite cluster observation resource allocation system based on Markov game, characterized in that: include: Resource acquisition module, used to obtain resource parameters of heterogeneous satellite clusters; The resource parameters include the number of satellites, the mission to be performed, the type of satellite and the visible time window; A model calling module is used to call a mixed integer programming model built based on the resource scheduling requirements of the autonomous observation of a heterogeneous satellite cluster according to the resource parameters; A first allocation module is configured to perform a first-stage joint optimization of tasks, satellites, and visible time windows on the mixed integer programming model using a genetic algorithm to generate an initial task allocation plan that matches tasks with satellites; A scheme sorting module is used to sort the initial task allocation schemes according to the order in which the tasks arrive, and generate a complete initial task execution plan for each satellite; The second allocation module is used to utilize the Improved-MADDPG deep reinforcement learning framework improved based on the genetic algorithm, and use the complete mission execution initial plan as prior knowledge to guide target optimization to obtain the optimal task allocation plan for the heterogeneous satellite cluster; wherein, within the Improved-MADDPG deep reinforcement learning framework, the satellites are regarded as intelligent agents and the task execution process is modeled as a Markov decision process. The intelligent agents adopt a random game strategy to perform distributed exploration and then converge global information through centralized decision-making. The optimal task allocation plan is an observation resource allocation plan that achieves the optimal task yield, completion rate and energy consumption under the observation resource constraints of the heterogeneous satellite cluster.

7. The satellite cluster observation resource allocation system based on Markov game according to claim 6, characterized in that: The process of performing the first phase of joint optimization of the mission, satellites and time windows of the mixed integer programming model using a genetic algorithm includes: The initial population is constructed using the global random generation method; The solution space representation is constructed using an integer-based two-dimensional encoding strategy; The calculation is performed using a fitness function with the core goal of minimizing the memory consumption of the mission on the satellite; Use tournament selection mechanism for selection operation; Perform crossover operation through the designed single-point crossover operator; The mutation operation is performed by introducing a directed mutation strategy and adopting a constraint-aware mechanism.

8. The satellite cluster observation resource allocation system based on Markov game according to claim 7 or 8, characterized in that: In the Improved-MADDPG deep reinforcement learning framework, a residual neural network structure enhanced by the attention mechanism is adopted, and the model convergence is accelerated through the Fix-up parameter initialization method and the Adax adaptive optimizer.

Images