Satellite cooperative tracking and pointing moving target method and system based on multi-agent reinforcement learning

Abstract

The application provides a satellite cooperative tracking and pointing moving target method and system based on multi-agent reinforcement learning, the method comprising: loading a trained strategy network on each satellite and initializing an interaction scene; constructing a local observation of the satellite, inputting the strategy network, and outputting a satellite tracking and pointing action at each time; the strategy network comprising a parallel convolution channel and a fully connected channel; after the convolution channel and the fully connected channel are spliced in a feature dimension, an action probability distribution is output through a fusion layer, and a satellite tracking and pointing action is obtained according to probability sampling. The application has the advantages that: the response time delay is significantly shortened; the tracking continuity and accuracy are improved; the energy consumption and maneuvering safety are controlled; and good scalability and fault tolerance are achieved.

Description

Technical Field

[0001] This application belongs to the field of aerospace remote sensing and satellite formation cooperative observation technology, specifically involving a satellite cooperative tracking and aiming method and system based on multi-agent reinforcement learning. Background Technology

[0002] In recent years, agile satellites, relying on high-torque reaction flywheels and attitude-orbit coupled optimal control algorithms, can achieve rapid revisits and high-precision imaging across regions and multiple target types, and are widely used in mission scenarios such as disaster emergency response, maritime surveillance, and battlefield intelligence. However, when facing the observation needs of multiple targets, high dynamics, and high timeliness on a global scale, a single satellite is limited by its orbit and field of view, making it difficult to balance high revisit frequency and continuous tracking duration. To compensate for this deficiency, the industry typically adopts multi-satellite formation collaborative observation, using a two-stage paradigm of "centralized task allocation—independent execution by each satellite": the ground or parent satellite solves an NP-hard scheduling problem after gathering global information, and then issues commands to each member satellite.

[0003] This centralized approach reveals several bottlenecks in dynamic target scenarios: the computational latency caused by centralized solutions weakens the response speed to sudden events; frequent global state feedback and command issuance occupy limited link resources, resulting in high communication load; the plan is highly dependent on prior information, and once the target trajectory changes abruptly or the on-board execution is disturbed, the entire calculation needs to be recalculated, lacking flexibility and robustness; at the same time, as the formation size increases, the computational complexity of centralized optimization increases exponentially, limiting scalability.

[0004] Deep reinforcement learning has demonstrated advantages in end-to-end optimization and online adaptation in attitude control and resource scheduling. However, existing research still largely follows a divide-and-conquer approach of "allocation first, execution later," separating target selection from attitude tracking and failing to fully explore the potential of decentralized decision-making and multi-agent collaborative learning. Furthermore, it lacks systematic evaluation and validation in complex scenarios involving multiple coupled factors such as energy constraints, orbital differences, and communication topologies. Therefore, achieving efficient, reliable, and scalable decentralized collaborative decision-making for large-scale agile satellite formations while ensuring attitude accuracy and energy consumption constraints remains a key technical challenge that urgently needs to be overcome. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of the traditional centralized "task allocation-attitude control" mode, such as large computational latency, high communication overhead, and lack of online adjustment capability in dynamic multi-target scenarios.

[0006] To achieve the above objectives, this application proposes a satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning, including:

[0007] Load the trained policy network onto each satellite and initialize the interaction scenario;

[0008] Construct local observations of the satellite, input them into the strategy network, and output the satellite tracking actions at each moment;

[0009] The policy network includes parallel convolutional channels and fully connected channels; after the convolutional channels and fully connected channels are concatenated along the feature dimension, the action probability distribution is output through the fusion layer, and the satellite tracking action is obtained by sampling based on the probability.

[0010] As an improvement to the above method, the training process of the policy network includes:

[0011] Step 1: Generate satellite and target trajectories and expand the area of ​​interest, then filter available satellites and targets;

[0012] Step 2: Set up local observation and reward functions, aggregate satellite capabilities using generated scene data and train the policy network, and obtain the optimal global parameters through iterative updates.

[0013] As an improvement to the above method, step 1 includes:

[0014] Step 1-1: Randomly generate weight values ​​for all targets and normalize them;

[0015] Step 1-2: Designate targets that fall within the region of interest as targets to be observed;

[0016] Steps 1-3: For each satellite, considering the maneuverability of the agile satellite, first expand the area of ​​interest by a set angle, then determine whether the satellite is located in the expanded area, and use the satellite located in the expanded area as the observation satellite.

[0017] As an improvement to the above method, step 2 includes:

[0018] Step 2-1: Randomly initialize the global policy network parameters;

[0019] Step 2-2: Randomly set the initial attitude quaternion and angular velocity for each satellite, load the global policy network parameters, and in subsequent scene training, all satellites in the scene share a single policy network parameter, called the aggregated policy network;

[0020] Steps 2-3: Calculate local satellite observations;

[0021] Steps 2-4: Input local observations into the strategy network to obtain satellite tracking actions, and determine control torque based on the actions; calculate the satellite's angular acceleration, update the angular velocity, and then update the satellite attitude quaternion based on the angular velocity;

[0022] Step 2-5: For each satellite, repeat steps 2-3 and 2-4 to obtain its respective local observations;

[0023] Steps 2-6: Calculate the attitude accuracy reward and tracking performance reward, as well as the energy consumption penalty and angular velocity penalty for each satellite. Weight the calculation results to obtain the overall reward. Use a centralized evaluation network to evaluate the satellite's actions based on the overall reward.

[0024] Steps 2-7: Update the positions of the satellite and the target at the next moment;

[0025] Steps 2-8: Repeat steps 2-3 to 2-7 within the same scene until all tracking tasks are completed;

[0026] Steps 2-9: The multi-agent proximal policy optimization algorithm is used to update the parameters of the aggregation policy network and the centralized evaluation network. The global parameters are updated by comprehensively considering the overall reward and the evaluation calculation gradient output by the centralized evaluation network.

[0027] Step 2-10: Repeat steps 2-2 to 2-9 until the policy network supports collaborative tracking tasks in multiple heterogeneous scenarios.

[0028] As an improvement to the above method, step 2-3, calculating the local observations of the satellite, includes:

[0029] Generate multiple desired attitude quaternions based on the vectors pointing to each target and other satellite nadir points;

[0030] Multiplying the conjugate of these desired attitude quaternions by the current attitude quaternion yields multiple attitude error quaternions;

[0031] For each satellite, extract the N sets of attitude error quaternions with the largest real part and the M sets of attitude error quaternions at the nadir of other satellites, and fill the insufficient positions with zero quaternions;

[0032] The normalized target weights and the current angular velocity are concatenated with the extracted attitude error quaternion to form a local observation.

[0033] As an improvement to the above method, the attitude accuracy reward is obtained by summing the logarithmic sum of the real part of the quaternion error of the main tracking satellite for each target, expressed as:

[0034] ;

[0035] in, Indicates attitude accuracy reward; This indicates taking the real part of the quaternion; To participate in the target The attitude error quaternion corresponding to all satellite sets; The total number of targets; the main tracking satellite is the satellite corresponding to the attitude error quaternion with the largest real number part of the attitude error quaternion of all satellites relative to a certain target, which is selected as the main tracking satellite.

[0036] As an improvement to the above method, the expression for the tracking effect reward is:

[0037] ;

[0038] in, Indicates a reward for tracking performance; For the goal An indicator variable indicating whether the tracking was successful; Normalize the weights for the target; This represents the overall target quantity.

[0039] As an improvement to the above method, the expression for the energy consumption penalty is:

[0040] ;

[0041] in, Indicates energy consumption penalty; This is the energy consumption penalty coefficient; The three-axis control torque vector generated for the satellite; The modulus representing the orientation quantity.

[0042] As an improvement to the above method, the formula for calculating the angular velocity penalty is as follows:

[0043] ;

[0044] in, Indicates angular velocity penalty; This represents the satellite's current angular velocity. This is the maximum angular velocity; The modulus representing the orientation quantity.

[0045] This application also provides a satellite cooperative tracking and aiming system for moving targets based on multi-agent reinforcement learning, implemented using the above method, the system comprising:

[0046] The satellite tracking module is used to load the policy network, construct and input local satellite observations, and output the satellite tracking actions at each time step.

[0047] The policy network includes parallel convolutional channels and fully connected channels; after the convolutional channels and fully connected channels are concatenated along the feature dimension, the action probability distribution is output through the fusion layer, and the satellite tracking action is obtained based on the probability sampling.

[0048] Compared with existing technologies, the advantages of this application are:

[0049] 1. Response latency is significantly reduced; each satellite can output attitude control commands in real time based on local observations without waiting for centralized scheduling, which can meet the high-timeliness observation requirements for responding to sudden targets.

[0050] 2. Improved tracking continuity and accuracy; the parallel one-dimensional convolution channel and the weighted reward mechanism work together to enable the strategy to dynamically balance tracking accuracy and energy consumption in multi-satellite and multi-target scenarios.

[0051] 3. Energy consumption and maneuver safety are controlled; energy consumption penalties and angular velocity constraints are introduced into the reward function to guide the strategy to actively suppress ineffective or excessive maneuvers, effectively reducing the dynamic load of the reaction flywheel and power consumption while ensuring attitude safety, and extending the satellite's on-orbit life.

[0052] 4. Excellent scalability and fault tolerance; through the strategy sharing and capability aggregation training framework, a single global strategy can be applied across different formation sizes and diverse mission scenarios; when individual satellites lose contact or the target trajectory changes abruptly, the remaining satellites can still redistribute tasks online and maintain stable tracking. Attached Figure Description

[0053] Figure 1 The diagram shown is a strategy network structure diagram;

[0054] Figure 2 The diagram shown is a framework diagram for capability aggregation training. Detailed Implementation

[0055] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0056] The satellite cooperative tracking and aiming method and system based on multi-agent reinforcement learning provided in this application is a cooperative tracking and aiming technology that enables multiple agile satellites to track and continuously observe moving targets in real time in orbit. By relying only on local observations from each satellite and supplemented by a small amount of inter-satellite information exchange, target priority assessment and attitude control decisions are completed simultaneously, thereby significantly improving the rapid response capability and tracking accuracy for multiple targets, and effectively suppressing ineffective maneuvers to reduce energy consumption.

[0057] Example 1

[0058] The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning provided in this application includes: first, generating different satellites and targets and dividing regions to construct multiple scenarios; then, constructing a convolutional parallel policy network; subsequently, setting local observation and reward functions, and training the network using different scenarios through a capability aggregation training framework to obtain the optimal model; finally, loading the model into the network and running it in the scenario of the task to be performed to obtain the tracking and aiming actions made by the satellite at each time step. Specifically, it includes the following steps:

[0059] Step 1: Generate the scene;

[0060] Step A: Generate satellite and target trajectories and expand the area of ​​interest, filter available satellites and targets, and provide basic data for mission allocation.

[0061] This section lays the foundation for subsequent mission allocation by generating satellite and target trajectories, constructing regions of interest, and then selecting targets and satellites that meet the criteria. The specific process includes:

[0062] Step A1: Generate satellite and target trajectory files, randomly generate weight values ​​for all targets, and normalize the weight values ​​for all targets.

[0063] Step A2: Define the area of ​​interest using latitude and longitude coordinates. This can be done by calling the Polygon class from the PythonShapely library in clockwise / counterclockwise order to initialize the area.

[0064] Step A3: For each target, determine whether it falls within the region of interest based on its latitude and longitude coordinates; only targets within the region are activated and added to the observation queue. This can be done by calling the `Polygon.contains` method.

[0065] Step A4: For each satellite, considering the maneuverability of agile satellites, first expand the area of ​​interest by angle, then determine whether the satellite is located within the expanded area. Specifically, first use Polygon.centroid to obtain the polygon center. Let the original boundary point be... Its distance from the center is Then expand radially outwards. The new vertex is obtained by the degree, as shown in formula (1).

[0066] (1)

[0067] With all The newly formed polygon serves as the expanded region of interest, and the Polygon.contains method is called again; if a satellite is located within this expanded region, it is activated and added to the list of available satellites.

[0068] Step 2: Construct a policy network with convolutional parallelism;

[0069] Step B: In a traditional fully connected strategy network, convolutional channels are connected in parallel, and the two features are fused to output the action probability, which guides the satellite to select the tracking action.

[0070] Based on the traditional multilayer perceptron policy network, an additional convolutional feature channel is added, thereby forming a convolutional network. A fully connected parallel architecture. This architecture can simultaneously mine local correlations in satellite local observation sequences. The overall network diagram is shown below. Figure 1 As shown, after convolutional channels (CNN) and fully connected channels (MLP) are concatenated along the feature dimension, the result is fused through a fusion layer (MIX) to output the action probability distribution. The specific process includes:

[0071] Step B1: Input the satellite's local observations (State) into the policy network, where the local observations are one-dimensional observation states. These observations are simultaneously input into both the convolutional and fully connected channels. The local observations include: the satellite's attitude error quaternion to nearby targets, the satellite's attitude error quaternion to the nadir point of neighboring satellites, the normalized weights of nearby targets, and the satellite's current angular velocity.

[0072] Step B2: The convolutional channel consists of two alternating stacked one-dimensional convolutional layers and max pooling layers. Each convolutional kernel has a size of 3, and the pooling window size is 2. Finally, the convolutional feature vector is obtained through a flattening operation.

[0073] Step B3: The fully connected channel uses a 3-layer fully connected network, with each layer followed by a ReLU activation function, outputting a fully connected feature vector.

[0074] Step B4: Concatenate the convolutional feature vector and the fully connected feature vector to obtain a joint representation, and input it into the fused fully connected layer to map the action probability distribution (Prob) to the action space. The processing is shown in Equation (2).

[0075] (2)

[0076] Step B5: Obtain the satellite tracking action based on probability sampling and transmit it to the satellite actuator to complete the status update.

[0077] Step 3: Model training;

[0078] Step C: Define the local observation and reward functions. The reward functions include: attitude accuracy reward, tracking performance reward, energy consumption penalty, and angular velocity penalty. In the multi-scenario capability aggregation training architecture, each scenario not only maintains its own aggregated satellite capabilities during training but also updates with the global satellite capabilities to obtain a stable cooperative tracking strategy.

[0079] For satellite collaborative tracking and aiming tasks, local observation and reward functions are set up. Satellite capabilities are aggregated using generated scene data, and the tracking and aiming strategy network is trained. Optimal global parameters are obtained through iterative updates. A schematic diagram of the training framework is shown below. Figure 2 As shown. The specific process is as follows:

[0080] Step C1: Randomly initialize the global policy network parameters.

[0081] Step C2: Initialize the interactive scene. Each satellite is randomly assigned an initial attitude quaternion and angular velocity. Then, the global policy network parameters are loaded. In subsequent scene training, all satellites within the scene share the same policy network parameters, hence the term "aggregated policy network." This completes the initialization of the environment and network.

[0082] Step C3: Construct local satellite observations based on the scene conditions. First, calculate the vectors pointing from the satellite to each target and the nadir points of other satellites, and generate multiple desired attitude quaternions accordingly (the "target-pointing" desired attitude quaternion is used to sense the attitude required for surrounding targets, and the "other satellite nadir points" desired attitude quaternion is used to sense the relative positions of surrounding satellites relative to this satellite). Then, multiply the conjugate of these desired attitude quaternions by the current attitude quaternion to obtain multiple attitude error quaternions. Subsequently, extract the three sets of target error quaternions with the largest real parts for each satellite (the largest real part indicates that the current satellite can achieve the desired attitude with the least amount of attitude adjustment). The system uses the successful tracking and aiming of the target satellite and the error quaternions of two other satellites (i.e., each satellite only considers the relative positions of the two closest satellites). Any missing positions are filled with zero quaternions (this application can pre-define that each satellite only considers the nearest N targets and M satellites, but in some small scenarios there may be no N targets or M other satellites. In this case, to keep the local observation length of the input strategy network unchanged, the missing parts are filled with all-zero quaternions). Finally, the normalized target weights and the current angular velocity are concatenated with these error quaternions to form a local observation.

[0083] Step C4: Input the local observations obtained in the above steps into the policy network of step two to obtain actions, and determine the control torque based on the actions. Then, use satellite rigid body dynamics to calculate angular acceleration and update angular velocity. Finally, update the satellite attitude quaternions based on the angular velocity.

[0084] Step C5: For each satellite in the scene, repeat steps C3-C4 to obtain their respective local observations.

[0085] Step C6: Calculate the attitude accuracy bonus. The attitude accuracy bonus is calculated by scaling the real logarithm of the quaternion error of the main tracking satellite for each target and summing the results. The calculation process is shown in formula (3).

[0086] (3)

[0087] in, This indicates taking the real part of the quaternion. To participate in the target The corresponding attitude error quaternion in the entire satellite set, The total number of targets. The primary tracking satellite is the satellite whose error quaternion for a given target is calculated from all satellites. The satellite with the largest real-valued error quaternion is selected as the primary tracking satellite; that is, the satellite whose current attitude is closest to successfully tracking and aiming at the target among all satellites.

[0088] Step C7: Calculate the tracking performance reward. The tracking performance reward is a weighted sum based on the number of successful tracking indications of the target, and the calculation process is shown in formula (4).

[0089] (4)

[0090] in, For the goal An indicator variable indicating whether the tracking was successful. The target is normalized weights.

[0091] Step C8: Calculate the energy consumption penalty for each satellite individually. The energy consumption penalty is taken as the control moment norm, and the calculation process is shown in formula (5).

[0092] (5)

[0093] in, This is the energy consumption penalty coefficient. For the total number of satellites, The three-axis control torque vector generated for the satellite; The modulus representing the orientation quantity.

[0094] Step C9: Calculate the angular velocity penalty for each satellite individually. The angular velocity penalty is a negative reward when the angular velocity exceeds the threshold. The calculation process is shown in formula (6).

[0095] (6)

[0096] in, The current angular velocity of the satellite. This represents the maximum angular velocity.

[0097] Step C10: Weight the above rewards to obtain the overall reward function. Then, use a centralized evaluation network to evaluate the satellite actions based on the overall reward. This centralized evaluation network is the central evaluation network in the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm.

[0098] Step C11: Update the positions of the satellite and the target at the next moment.

[0099] Step C12: Repeatedly execute C3 to C11 within the same scene until all tracking tasks are declared over.

[0100] Step C13: Update the parameters of the aggregation policy network and the centralized evaluation network using a multi-agent proximal policy optimization algorithm. Considering both the overall reward and the evaluation output of the centralized evaluation network, the Adam optimizer is used with a learning rate of 1e-3 to calculate the gradient. .

[0101] Step C14: Update global parameters ,in These are the network parameters before the update. These are the updated network parameters. It is the learning rate.

[0102] Step C15: Repeat steps C2 to C14 until the model can stably complete the cooperative tracking task in various heterogeneous scenarios.

[0103] Step 4: Task execution;

[0104] Step D: Each satellite is loaded with a pre-trained policy network, which outputs satellite attitude and tracking actions in real time in the actual scenario to complete the target cooperative tracking task. Other satellite information and target information are transmitted via inter-satellite / satellite-to-ground communication, and only a small amount of position coordinates are transmitted.

[0105] The trained model is used to perform tracking and aiming tasks in the target scenario. The specific process is as follows:

[0106] Step D1: Initialize the interactive scene. For each satellite, set the initial attitude quaternion to the direction the payload points towards the Earth's center, and set the angular velocity to 0. Then, read the trained model parameters to complete the initialization of the environment and network.

[0107] Step D2: Through the interaction between the model and the environment, continue until the scene ends. Output the satellite tracking action, satellite attitude, target tracking results, etc. at each moment.

[0108] Example 2

[0109] This application also provides a satellite cooperative tracking and aiming system for moving targets based on multi-agent reinforcement learning, implemented using the above method, the system comprising:

[0110] The satellite tracking module is used to load the policy network, construct and input local satellite observations, and output the satellite tracking actions at each time step.

[0111] The policy network includes parallel convolutional channels and fully connected channels; after the convolutional channels and fully connected channels are concatenated along the feature dimension, the action probability distribution is output through the fusion layer, and the satellite tracking action is obtained based on the probability sampling.

[0112] This invention breaks away from the traditional two-stage operational framework of "target allocation-attitude control," proposing an end-to-end policy network based on local observation information. This allows each satellite in the formation to synchronously complete target priority determination and tracking decisions under decentralized conditions, significantly shortening the command loop, improving the rapid response capability to sudden targets, and maintaining high system fault tolerance and robustness without a central node. Addressing the multiple requirements of attitude accuracy, continuous tracking effect, energy consumption control, and attitude safety in multi-satellite collaborative observation, this application provides a multi-agent reward mechanism that balances cooperative game theory and energy-saving constraints. By balancing tracking quality and energy consumption during reinforcement learning, it guides each agent to dynamically adjust its observation attitude to meet the needs of continuous multi-target tracking, while actively suppressing ineffective or excessive attitude maneuvers to reduce energy consumption. To enhance the generalization of the strategy, this application further constructs a heterogeneous multi-scenario capability aggregation training framework: in a real orbital dynamics simulation environment, multiple scenarios covering different satellite sizes, target densities and spatial distributions are introduced. Through strategy sharing and capability aggregation mechanisms, information exchange and collaborative control are achieved, gradually strengthening the adaptability of each agent to complex environments, and on this basis, significantly improving the mission execution efficiency and operational stability of the entire satellite formation.

[0113] This application may also provide a computer device, including: at least one processor, memory, at least one network interface, and a user interface. The various components in this device are coupled together via a bus system. It is understood that the bus system is used to implement communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0114] The user interface can include a display, keyboard, or clicking device. Examples include a mouse, trackball, touchpad, or touchscreen.

[0115] It is understood that the memory in the embodiments disclosed in this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0116] In some implementations, the memory stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0117] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application functions. Programs implementing the methods of the embodiments of this disclosure can be included in the application programs.

[0118] In the above embodiments, the processor can also invoke programs or instructions stored in memory, specifically programs or instructions stored in an application program, for the following purposes:

[0119] Follow the steps described above.

[0120] The above methods can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed above. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the disclosed methods can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0121] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0122] For software implementation, the technology of this application can be implemented by executing the functional modules (e.g., procedures, functions, etc.) of this application. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0123] This application may also provide a non-volatile storage medium for storing a computer program. When the computer program is executed by a processor, it can implement the steps in the above method embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A satellite cooperative tracking method for moving targets based on multi-agent reinforcement learning, comprising: Load the trained policy network onto each satellite and initialize the interaction scenario; Construct local observations of the satellite, input them into the strategy network, and output the satellite tracking actions at each moment; The policy network includes parallel convolutional channels and fully connected channels; after the convolutional channels and fully connected channels are concatenated along the feature dimension, the action probability distribution is output through the fusion layer, and the satellite tracking action is obtained by sampling based on the probability.

2. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 1, characterized in that, The training process of the policy network includes: Step 1: Generate satellite and target trajectories and expand the area of ​​interest, then filter available satellites and targets; Step 2: Set up local observation and reward functions, aggregate satellite capabilities using generated scene data and train the policy network, and obtain the optimal global parameters through iterative updates.

3. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 2, characterized in that, Step 1 includes: Step 1-1: Randomly generate weight values ​​for all targets and normalize them; Step 1-2: Designate targets that fall within the region of interest as targets to be observed; Steps 1-3: For each satellite, considering the maneuverability of the agile satellite, first expand the area of ​​interest by a set angle, then determine whether the satellite is located in the expanded area, and use the satellite located in the expanded area as the observation satellite.

4. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 2, characterized in that, Step 2 includes: Step 2-1: Randomly initialize the global policy network parameters; Step 2-2: Randomly set the initial attitude quaternion and angular velocity for each satellite, load the global policy network parameters, and in subsequent scene training, all satellites in the scene share a single policy network parameter, called the aggregated policy network; Steps 2-3: Calculate local satellite observations; Steps 2-4: Input local observations into the strategy network to obtain satellite tracking actions, and determine control torque based on the actions; calculate the satellite's angular acceleration, update the angular velocity, and then update the satellite attitude quaternion based on the angular velocity; Step 2-5: For each satellite, repeat steps 2-3 and 2-4 to obtain its respective local observations; Steps 2-6: Calculate the attitude accuracy reward and tracking performance reward, as well as the energy consumption penalty and angular velocity penalty for each satellite. Weight the calculation results to obtain the overall reward. Use a centralized evaluation network to evaluate the satellite's actions based on the overall reward. Steps 2-7: Update the positions of the satellite and the target at the next moment; Steps 2-8: Repeat steps 2-3 to 2-7 within the same scene until all tracking tasks are completed; Steps 2-9: The multi-agent proximal policy optimization algorithm is used to update the parameters of the aggregation policy network and the centralized evaluation network. The global parameters are updated by comprehensively considering the overall reward and the evaluation calculation gradient output by the centralized evaluation network. Step 2-10: Repeat steps 2-2 to 2-9 until the policy network supports collaborative tracking tasks in multiple heterogeneous scenarios.

5. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 4, characterized in that, Steps 2-3, which calculate the satellite's local observations, include: Generate multiple desired attitude quaternions based on the vectors pointing to each target and other satellite nadir points; Multiplying the conjugate of these desired attitude quaternions by the current attitude quaternion yields multiple attitude error quaternions; For each satellite, extract the N sets of attitude error quaternions with the largest real part and the M sets of attitude error quaternions at the nadir of other satellites, and fill the insufficient positions with zero quaternions; The normalized target weights and the current angular velocity are concatenated with the extracted attitude error quaternion to form a local observation.

6. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 4, characterized in that, The attitude accuracy bonus is obtained by summing the logarithmic real part of the quaternion error of the main tracking satellite for each target, and the expression is: ； in, Indicates attitude accuracy reward; This indicates taking the real part of the quaternion; To participate in the target The attitude error quaternion corresponding to all satellite sets; The total number of targets; the main tracking satellite is the satellite corresponding to the attitude error quaternion with the largest real number part of the attitude error quaternion of all satellites relative to a certain target, which is selected as the main tracking satellite.

7. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 4, characterized in that, The expression for the tracking effect reward is: ； in, Indicates a reward for tracking performance; For the goal An indicator variable indicating whether the tracking was successful; Normalize the weights for the target; This represents the overall target quantity.

8. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 4, characterized in that, The expression for the energy consumption penalty is: ； in, Indicates energy consumption penalty; This is the energy consumption penalty coefficient; The three-axis control torque vector generated for the satellite; The modulus representing the orientation quantity.

9. The satellite cooperative tracking and aiming method for moving targets based on multi-agent reinforcement learning according to claim 4, characterized in that, The formula for calculating the angular velocity penalty is as follows: ； in, Indicates angular velocity penalty; This represents the satellite's current angular velocity. This is the maximum angular velocity; The modulus representing the orientation quantity.

10. A satellite cooperative tracking and aiming system for moving targets based on multi-agent reinforcement learning, implemented according to the method described in any one of claims 1-9, characterized in that, The system includes: The satellite tracking module is used to load the policy network, construct and input local satellite observations, and output the satellite tracking actions at each time step; and The policy network includes parallel convolutional channels and fully connected channels; after the convolutional channels and fully connected channels are concatenated along the feature dimension, the action probability distribution is output through the fusion layer, and the satellite tracking action is obtained based on the probability sampling.

Images