Satellite scheduling strategy generation method and device, and storage medium

Abstract

The invention relates to the technical field of space remote sensing, in particular to a satellite scheduling strategy generation method and device and a storage medium, and the method comprises the steps: obtaining a plurality of imaging strips according to the imaging characteristics of the imaging load of a satellite related to a target area; wherein the target area comprises a movable target; establishing a POMDP model for the target area; generating an action mask according to the relationship between the imaging strip of the satellite and the target area; wherein the action masks comprise valid action masks and invalid action masks; and training the POMDP model based on the action mask, and generating the satellite scheduling strategy. According to the embodiment of the invention, the motion mask is applied to the solution process of the POMDP model during the multi-fall cooperative scheduling, so that the generation speed of the large-scale complex scheduling strategy of the satellite is remarkably improved, and the robustness of the scheduling strategy is relatively high.

Description

Technical Field

[0001] This specification relates to the field of space remote sensing technology, and in particular to a satellite scheduling strategy generation method, device and storage medium. Background Technology

[0002] Existing algorithms for solving the collaborative scheduling problem of imaging satellites can be broadly categorized into two types: exact search algorithms and approximate algorithms. Exact search algorithms, such as exhaustive search, branch and bound, or dynamic programming, can theoretically find the optimal solution, but their computational complexity increases exponentially with the problem size (e.g., number of satellites, number of missions, planning duration). This makes it difficult to provide a solution within an acceptable timeframe for large-scale satellite imaging planning problems, failing to meet real-time requirements. To overcome the limitations of exact search algorithms, approximate algorithms have gained widespread attention due to their high search efficiency, such as simulated annealing, genetic algorithms, and dynamic path search algorithms. These algorithms can find approximate optimal solutions in a relatively short time, but the quality and stability of their solutions are still not ideal when dealing with complex constraints and dynamically changing environments. Furthermore, these approximate algorithms still fall short in robustness and real-time performance when facing the complex dynamic constraints unique to satellite scheduling. Summary of the Invention

[0003] The purpose of the embodiments in this specification is to provide a satellite scheduling strategy generation method, device and storage medium to solve the problem that in the prior art, when multiple satellites perform imaging under complex constraints such as moving targets, the multi-satellite collaborative scheduling algorithm has a large computational load and slow convergence, resulting in poor imaging real-time performance.

[0004] To achieve the above objectives, on the one hand, embodiments of this specification provide a satellite scheduling strategy generation method, including,

[0005] Based on the imaging characteristics of the imaging payloads of satellites related to the target area, multiple imaging strips are obtained; wherein, the target area includes movable targets;

[0006] A partially visible Markov decision POMDP model is established for the target region;

[0007] An action mask is generated based on the relationship between the satellite's imaging stripe and the target area; wherein the action mask includes a valid action mask and an invalid action mask;

[0008] The POMDP model is trained based on the action mask to generate the satellite scheduling strategy.

[0009] As one aspect of an embodiment of this specification, generating an action mask based on the relationship between the satellite's imaging stripe and the target region further includes,

[0010] Determine whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area;

[0011] If so, the imaging action of the satellite at the predetermined time is identified as a valid action;

[0012] If not, the imaging action of the satellite at the predetermined time is identified as an invalid action;

[0013] The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

[0014] As another aspect of the embodiments of this specification, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0015] Determine whether the satellite has maneuvered to an attitude that satisfies the requirement of covering at least part of the target area at a predetermined time.

[0016] As another aspect of the embodiments of this specification, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0017] Determine whether the imaging strip of the satellite at a predetermined time covers at least one target area with a non-negligible probability of the presence of a movable target.

[0018] As another aspect of the embodiments of this specification, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0019] The target area is divided into multiple grids;

[0020] Determine whether the area ratio of the imaging strip of the satellite covering a grid at a predetermined time is greater than a preset threshold.

[0021] As another aspect of the embodiments of this specification, the preset threshold range includes preset values ​​between (0,1).

[0022] As another aspect of the embodiments of this specification, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0023] Determine whether the predetermined time is within the current or upcoming valid observation time window.

[0024] As another aspect of the embodiments of this specification, generating an action mask based on the relationship between the satellite's imaging stripe and the target region further includes,

[0025] When the satellite performs imaging operations on the target area at a predetermined time, it determines whether the satellite's resources meet the resource constraints.

[0026] If so, the imaging action of the satellite at the predetermined time is identified as a valid action;

[0027] If not, the imaging action of the satellite at the predetermined time is identified as an invalid action;

[0028] The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

[0029] As another aspect of the embodiments of this specification, the resource constraints include at least one of the following conditions: the satellite's energy reserves, the satellite's remaining data storage capacity, the satellite's remaining computing resources, or the satellite's environmental data.

[0030] As another aspect of the embodiments of this specification, the generation action mask further includes,

[0031] The mask vector of the valid action mask is a value that does not affect the probability of selecting the valid action, and the mask vector of the invalid action mask is a value that cancels out the probability of selecting the invalid action.

[0032] As another aspect of the embodiments of this specification, training the POMDP model based on the action mask to generate the satellite scheduling strategy further includes...

[0033] The POMDP model is trained using a reinforcement learning algorithm;

[0034] The satellite scheduling strategy is generated by offsetting the probability of selecting invalid actions in the reinforcement learning algorithm using the action mask.

[0035] As another aspect of the embodiments of this specification, the probability of selecting invalid actions in the reinforcement learning algorithm by canceling the action mask further includes,

[0036] The mask vector of the action mask is added to the probability of selecting an invalid action in the reinforcement learning algorithm, so that the probability of the invalid action corresponding to the invalid action mask being selected approaches 0.

[0037] As another aspect of the embodiments of this specification, after adding the mask vector of the action mask to the probability of selecting an invalid action in the reinforcement learning algorithm, it further includes...

[0038] When evaluating the action using the dominance function, the dominance of the action is normalized.

[0039] As another aspect of the embodiments of this specification, the probability of selecting an invalid action in the reinforcement learning algorithm by canceling out the action mask further includes,

[0040] A new action is output based on the probability of selecting a valid action;

[0041] A new action mask and a new state are generated based on the new action; wherein, the new action mask is used to iteratively offset the probability of selecting an invalid action in the reinforcement learning algorithm, and the new state is used to update the action network and evaluation network in the reinforcement learning algorithm.

[0042] As another aspect of the embodiments of this specification, regenerating a new action mask and a new state based on the new action further includes,

[0043] Generate empirical data including a tuple data structure containing action masks;

[0044] The experience data is stored in the experience pool of the reinforcement learning algorithm.

[0045] The empirical data consists of 7-tuple data (s t a t ,logp(s t |a t ), r t V(s) t ), done, mask), where s t Given the current state, a t For the action I selected based on the old satellite scheduling strategy, logp(s) t |a t ) for s t Choose action a t The logarithmic function of the transition probability, r t To perform action a t The immediate reward obtained afterward, V(s) t To evaluate the network's estimated current state s t The value of is , where 'done' is a boolean flag indicating whether the current round has ended, and 'mask' is the current state 's'. t The corresponding action mask vector. The empirical data consists of 7-tuple data (s t a t ,logp(s t |a t ), r t V(s) t ), done, mask), where s t Given the current state, a t For the action I selected based on the old satellite scheduling strategy, logp(s) t |a t ) for s t Choose action at The logarithmic function of the transition probability, r t To perform action a t The immediate reward obtained afterward, V(s) t To evaluate the network's estimated current state s t The value of is , where 'done' is a boolean flag indicating whether the current round has ended, and 'mask' is the current state 's'. t The corresponding action mask vector.

[0046] On the other hand, embodiments of this specification also provide a satellite scheduling strategy generation apparatus, including,

[0047] A strip division unit is used to obtain multiple imaging strips based on the imaging characteristics of the imaging payloads of satellites related to the target area; wherein, the target area includes movable targets;

[0048] A modeling unit is used to establish a partially visible Markov decision POMDP model for the target region.

[0049] An environment processing unit is used to generate an action mask based on the relationship between the imaging stripe of the satellite and the target area; wherein the action mask includes a valid action mask and an invalid action mask;

[0050] The model training unit trains the POMDP model based on the action mask to generate the satellite scheduling strategy.

[0051] On the other hand, embodiments of this specification also provide an intelligent agent, including:

[0052] At least one processor; and

[0053] At least one memory storing instructions that, when executed individually or jointly by the at least one processor, cause the agent to perform the satellite scheduling strategy generation method described above.

[0054] On the other hand, embodiments of this specification also provide a computer storage medium storing a computer program thereon, which, when run by the processor of a computer device, executes instructions for the above-described method.

[0055] On the other hand, embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the above-described method.

[0056] On the other hand, embodiments of this specification also provide a chip, the chip including a circuit system configured to perform the above-described method.

[0057] As can be seen from the technical solutions provided in the embodiments of this specification above, when performing multi-satellite collaborative scheduling, the embodiments of this specification apply action masks to the solution process of partially visible Markov decision (POMDP) ​​models, which significantly improves the generation speed of large-scale complex satellite scheduling strategies and the scheduling strategies have high robustness; and also improves the success rate of observing moving targets. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0059] Figure 1 The diagram shown is a schematic diagram of multi-star cooperative imaging in an embodiment of this specification;

[0060] Figure 2 The diagram shown is a flowchart of a satellite scheduling strategy generation method according to an embodiment of this specification;

[0061] Figure 3 The diagram shown is a flowchart of a multi-satellite imaging scheduling strategy generation method for mobile target observation according to an embodiment of this specification.

[0062] Figure 4 The diagram shown is a flowchart of the imaging strip division in an embodiment of this specification;

[0063] Figure 5 The diagram shown is a schematic diagram of the calculation of the vertex of the imaging strip in an embodiment of this specification;

[0064] Figure 6 The diagram shown is a schematic representation of the imaging strip division of multiple satellites in an embodiment of this specification.

[0065] Figure 7 The diagram shown is a flowchart of the improved PPO algorithm in the embodiments of this specification.

[0066] Figure 8 The diagram shown is a structural schematic of the improved PPO algorithm in an embodiment of this specification.

[0067] Figure 9 The figure shown is a schematic diagram of the simulation results of the embodiments in this specification;

[0068] Figure 10 The diagram shown is a comparison of the effects of the embodiments in this specification with other solutions;

[0069] Figure 11 The diagram shown is a schematic representation of the average reward during strategy training in an embodiment of this specification.

[0070] Figure 12 The diagram shown is a schematic representation of the average reward of the strategy in the embodiments of this specification.

[0071] Figure 13 The image shown is an embodiment of a satellite scheduling strategy generation device described in this specification. Detailed Implementation

[0072] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0073] In this specification, unless otherwise stated, "and / or" describes an association between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. Furthermore, in this disclosure, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0074] In this specification, the expressions "greater than" or "less than" may be used to determine whether a specific condition is met. However, this is only for illustrative purposes and is not intended to exclude statements of "above" or "below". A condition described as "above" may be replaced by "greater than", a condition described as "below" may be replaced by "less than", and a condition described as "above and less than" may be replaced by "greater than and below". Furthermore, hereinafter, "A" to "B" represent at least one of the elements from A (inclusive) to B (inclusive).

[0075] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0076] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0077] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0078] This application uses terminology used in some communication specifications (e.g., the 3rd Generation Partnership Project, the European Telecommunications Standards Institute, ETSI, Extensible Radio Access Network, ERAN, and Open-Radio Access Network, O-RAN) to describe various embodiments, but this is merely illustrative. The various embodiments of this application can also be readily modified and applied in other communication systems.

[0079] In the embodiments of this application, communication between devices in the communication system can be carried out according to communication protocols at any stage, such as including but not limited to the following communication protocols: 1G (Generation), 2G, 2.5G, 2.75G, 3G, 4G, 4.5G and 5G, New Radio (NR) and / or other currently known or future communication protocols.

[0080] For ease of understanding, the technical terms involved in the embodiments of this application will be explained below.

[0081] (1) Terminal: refers to a device with wireless transceiver function and can cooperate with network-side equipment to provide communication services to users. This terminal can receive NGSO satellite services. That is, the terminal connects to the NGSO satellite in the NGSO satellite service cell and receives NGSO satellite services. The terminal is not connected to the GEO satellite. When the terminal is in the avoidance area where the NGSO satellite interferes with the GEO satellite, it will be affected by the NGSO satellite service interruption. For example, terminal devices can be mobile phones, tablets, laptops, wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless communication devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, Internet of Things (IoT) devices, narrowband Internet of Things (NB-IoT) devices, vehicle-to-everything (V2X) devices, devices in device-to-device communication (D2D), enhanced machine-type communication (eMTC) devices, and reduced-capacity devices. Capability (RedCap), cellular phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), clients, handheld devices with wireless communication capabilities, vehicle-mounted devices, or shipboard devices, etc.

[0082] In scenarios such as the Internet of Things (IoT), the terminal can also be a machine or device for monitoring or measurement, such as including but not limited to: machine-type communication (MTC) terminals, vehicle communication terminals, device-to-device terminals, machine-to-machine (M2M) terminals, and so on.

[0083] (2) Network equipment: refers to network-side equipment capable of communicating with terminal equipment. Network equipment can be located on satellites or the ground. Network equipment can also be called space base stations, satellite-borne base stations, satellites, satellite communication nodes, satellite network terminal equipment, satellite communication modules, or base stations, etc. This network-side equipment can also be called access network equipment or wireless access network equipment. Network-side equipment can be a base station (BTS) in a satellite-borne Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) communication system, a base station (NodeB, NB) in a satellite-borne Wideband Code Division Multiple Access (WCDMA) system, an evolved base station (eNB or eNodeB) in a satellite-borne LTE system, a base station in a terrestrial network or non-terrestrial network (NTN), such as a base station (gNB) in a satellite-borne 5G network, a base station in a future network after 5G (e.g., 6G network), a base station in a future evolved Public Land Mobile Network (PLMN) network, a Transmission Reception Point (TRP) in a satellite, or a Cloud Radio Access Network in a satellite. In the context of Networks (CRAN), wireless controllers can also be satellite-borne city base stations, micro base stations, pico base stations, or femtobase stations. Base stations can also be ground-based base stations capable of satellite communication, and can be referred to as Access Points (APs), 5G nodes (5th generation nodes), wireless points, or Transmission / Reception Points (TRPs), the latter being other terms with equivalent technical meanings. Network equipment can also refer to base station equipment carried by High Altitude Platform Stations (HAPS) with loiter capabilities, such as large balloons or airships, base station equipment in Roadside Units (RSUs), or base station equipment in vehicle-to-everything (V2X) networks.

[0084] Furthermore, the term "network side" or "network equipment side" refers to one side of the network, which can be a base station or include one or more network devices as described above. The term "terminal side" or "terminal equipment side" refers to the side of the user or terminal, which can be a UE or include one or more terminal devices as described above.

[0085] like Figure 1 The diagram shown is a schematic of multi-satellite collaborative imaging in an embodiment of this specification. The diagram illustrates the imaging area of ​​the ground by multiple satellites at the same time. Based on the analysis of the imaging angle and swath width of each satellite's imaging payload, a model is built for movable targets (with uncertainty in target movement) in the target area. Then, by analyzing the effectiveness of satellite scheduling imaging actions, a real-time satellite scheduling strategy can be obtained, which can significantly improve the speed and quality of scheduling decisions, thereby maximizing the dynamic observation benefits of moving or uncertain targets on the ground.

[0086] like Figure 2 The diagram shows a flowchart of a satellite scheduling strategy generation method according to an embodiment of this specification. The diagram illustrates the combination of satellite-corresponding imaging strips and a POMDP model describing collaborative satellite imaging. By using action masks for satellite scheduling actions to cancel out invalid actions during model training, the model training can converge quickly, yielding the optimal satellite scheduling strategy. This specification provides the operational steps of the method described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the method can be executed sequentially or in parallel according to the embodiments or accompanying drawings. The method for execution on the NGSO satellite side specifically includes:

[0087] Step 201: Based on the imaging characteristics of the imaging payloads of satellites related to the target area, obtain multiple imaging strips; wherein, the target area includes movable targets;

[0088] Step 202: Establish a partially visible Markov decision (POMDP) ​​model for the target region;

[0089] Step 203: Generate an action mask based on the relationship between the imaging stripe of the satellite and the target area; wherein the action mask includes a valid action mask and an invalid action mask;

[0090] Step 204: Train the POMDP model based on the action mask to generate the satellite scheduling strategy.

[0091] In the embodiments described in this specification, multiple imaging strips corresponding to ground areas are obtained by combining the imaging characteristics of the imaging payloads of multiple satellites. Some of these imaging strips cover the target area, some cover a portion of the target area, and some do not cover the target area. For the imaging problem of movable targets, a POMDP model is used for modeling, and an action mask is used to cancel the probability of invalid actions, thereby enabling the rapid generation of satellite scheduling strategies.

[0092] As one embodiment, generating an action mask based on the relationship between the satellite's imaging stripes and the target region further includes:

[0093] Determine whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area;

[0094] If so, the imaging action of the satellite at the predetermined time is identified as a valid action;

[0095] If not, the imaging action of the satellite at the predetermined time is identified as an invalid action;

[0096] The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

[0097] In this embodiment, the imaging payloads (such as optical cameras and synthetic aperture radar sensors) carried by low-orbit imaging satellites typically have a fixed instantaneous field of view (FOV), which determines that the swath width of their Earth imaging at a specific orbital altitude is limited. When the target area to be observed is large, or the satellite's transit trajectory does not completely cover the target center, a single imaging may not be able to completely cover the target area. Therefore, it is necessary to decompose the satellite's potential observation capability of the target area into a series of feasible imaging strips with defined coverage. For the problem of multi-satellite collaborative observation of mobile targets, the precise location of the mobile target (such as ships on the sea or vehicles on the ground) is often unknown or uncertain at the time of planning and decision-making. By modeling this problem using the Partially Observable Markov Decision Process (POMDP) ​​framework in the embodiments of this specification, the complex satellite coverage problem can be transformed into a discrete, manageable imaging strip selection problem. The imaging strips will serve as the basic elements of the action space in the POMDP model. The POMDP model can include tuple data, such as six-tuple data (S, A, T, R, Ω, O), where S is the state space, A is the action space, T is the state transition function, R is the reward function, Ω is the observation set, and O is the observation function.

[0098] An action mask is generated based on the coverage relationship between the satellite's imaging stripe at a predetermined time and the target area. The predetermined coverage relationship means that the imaging stripe covers at least a portion of the target area at the current time or at some future time. It is then determined whether the satellite's imaging stripe at the predetermined time can form a coverage relationship with the target area. If the determination result is yes, the satellite's action of generating the imaging stripe is a valid action, and the generated action mask is a valid action mask; if the determination result is no, the satellite's action of generating the imaging stripe is an invalid action, and the generated action mask is an invalid action mask.

[0099] In one embodiment, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0100] Determine whether the satellite has maneuvered to an attitude that satisfies the requirement of covering at least part of the target area at a predetermined time.

[0101] In this embodiment, based on the satellite's ephemeris information and attitude parameters, such as field of view, latitude and longitude of the satellite's nadir point, orbital altitude, and side angle range, it can be calculated whether the satellite's imaging stripe at a predetermined time can cover at least a portion of the target area. This step can be referred to as the physical reachability judgment of the satellite's imaging payload to at least a portion of the target area. In other words, it determines whether the satellite can acquire an image of at least a portion of the target area at a predetermined time under physical conditions such as mechanical structure, imaging capability, and spatial position. If the satellite can maneuver to an attitude that covers at least a portion of the target area at the predetermined time, the satellite imaging action that forms the imaging stripe is identified as a valid action; otherwise, the satellite imaging action that forms the imaging stripe is identified as an invalid action.

[0102] In one embodiment, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0103] Determine whether the imaging strip of the satellite at a predetermined time covers at least one target area with a non-negligible probability of the presence of a movable target.

[0104] In this embodiment, the movement of the movable target within the target area is unknown; that is, the possible location of the movable target within the target area at any given time is uncertain. Probability is needed to describe the possible appearance of the movable target in a specific grid within the target area. Therefore, the imaging strip must not only cover at least one grid within the target area, but the probability of the movable target appearing in the covered grid must also be non-zero. In other words, at least one non-negligible target area includes grids within the target area where the probability of the movable target appearing is non-zero (or the probability is less than a set threshold). If the imaging strip covers at least one target area with a non-negligible probability of the movable target's presence, the satellite imaging action forming the imaging strip is identified as a valid action; otherwise, the satellite imaging action forming the imaging strip is identified as an invalid action.

[0105] In one embodiment, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0106] The target area is divided into multiple grids;

[0107] Determine whether the area ratio of the imaging strip of the satellite covering a grid at a predetermined time is greater than a preset threshold.

[0108] In this embodiment, the target area can be divided into multiple grids, for example, the target area can be divided into 9 grids. When the target area covered by the imaging strip is smaller than a certain proportion of the grid, for example, when the target area covered by the imaging strip is smaller than 50% of the grid, the satellite imaging action that forms the imaging strip is identified as an invalid action; when the target area covered by the imaging strip is larger than a certain proportion of the grid, for example, when the target area covered by the imaging strip is larger than 50% of the grid, the satellite imaging action that forms the imaging strip is identified as a valid action.

[0109] The preset threshold can be 50% of the above embodiment, or it can be other preset values ​​between (0,1), which will not be elaborated here.

[0110] In one embodiment, determining whether the imaging stripe of the satellite at a predetermined time covers at least a portion of the target area further includes,

[0111] Determine whether the predetermined time is within the current or upcoming valid observation time window.

[0112] In this embodiment, the satellite moves in orbit, and the time window for observing the target area is discrete. The current or upcoming effective observation time window in this embodiment can be calculated using the satellite's ephemeris and imaging capabilities (including angle, field of view, etc.).

[0113] Generating an action mask based on the relationship between the satellite's imaging stripes and the target region further includes:

[0114] When the satellite performs imaging operations on the target area at a predetermined time, it determines whether the satellite's resources meet the resource constraints.

[0115] If so, the imaging action of the satellite at the predetermined time is identified as a valid action;

[0116] If not, the imaging action of the satellite at the predetermined time is identified as an invalid action;

[0117] The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

[0118] In this embodiment, when the satellite performs an imaging action, it is determined whether the satellite's resources support the imaging action. If the satellite's resources do not support the imaging action, the imaging action is identified as invalid. If the satellite's resources support the imaging action, the imaging action is identified as valid.

[0119] The resource constraints may include, but are not limited to, one of the following: the satellite's energy reserves, the satellite's remaining data storage capacity, the satellite's remaining computing resources, or the satellite's environmental data.

[0120] In one embodiment, the generation action mask further includes,

[0121] The mask vector of the valid action mask is a value that does not affect the probability of selecting the valid action, and the mask vector of the invalid action mask is a value that cancels out the probability of selecting the invalid action.

[0122] In this embodiment, the effective action mask can be, for example, 0, and the invalid action mask can be, for example, a very large negative number, such as -∞. This ensures that the probability of selecting an effective action is not changed, thus not affecting the subsequent model training process, and can also offset the probability of selecting an invalid action, thereby significantly reducing the computation convergence time and reducing the complexity of model training.

[0123] In one embodiment, training the POMDP model based on the action mask to generate the satellite scheduling policy further includes...

[0124] The POMDP model was trained using the Proximal Policy Optimization (PPO) algorithm;

[0125] The satellite scheduling strategy is generated by offsetting the probability of selecting invalid actions in the PPO algorithm using the action mask.

[0126] In this embodiment, the PPO algorithm mainly includes an action network and an evaluation network, as well as activation function and loss function modules. In the embodiment of this specification, it also includes a mask function. The probability of selecting an imaging action output by the action network is canceled out by the action mask of the mask function, and the probability of selecting an invalid action is obtained after passing through the activation function. The evaluation network is used to evaluate the state corresponding to the action. Then, the POMDP model is iteratively trained through the loss function to finally obtain the optimal satellite scheduling strategy, that is, which satellites are selected to perform imaging actions, so that the combination of imaging strips of multiple satellites can meet the imaging requirements of the target area.

[0127] In one embodiment, offsetting the probability of selecting an invalid action in the PPO algorithm by using the action mask further includes,

[0128] The mask vector of the action mask is added to the probability of selecting an invalid action in the PPO algorithm, so that the probability of the invalid action corresponding to the invalid action mask being selected approaches 0.

[0129] In this embodiment, the Mask function is used to calculate the probability of selecting the action (including valid actions and invalid actions) output by the action network with the action mask. For example, the probability of selecting the action is added to the action mask corresponding to the action, so that the probability of invalid actions being selected approaches 0, thereby offsetting the probability of selecting invalid actions.

[0130] After the activation function outputs a new action based on the probability of selecting a valid action, the environment processing unit generates a new action mask and a new state, updates the Mask function using the new action mask, and updates the action network and evaluation network using the new state.

[0131] The environment processing unit generates empirical data, including an action mask, based on the new action output by the activation function; this empirical data is then stored in the experience pool of the PPO algorithm. During subsequent iterative training of the POMDP model using a loss function, empirical data in a tuple structure is retrieved from the experience pool for iterative training. The empirical data consists of 7-tuple data (s...). t a t ,logp(s t |a t ), r t V(s) t ), done, mask), where s t Given the current state, a t For the action I selected based on the old satellite scheduling strategy, logp(s)t |a t ) for s t Choose action a t The logarithmic function of the transition probability, r t To perform action a t The immediate reward obtained afterward, V(s) t To evaluate the network's estimated current state s t The value of is , where 'done' is a boolean flag indicating whether the current round has ended, and 'mask' is the current state 's'. t The corresponding action mask vector.

[0132] In one embodiment, after adding the mask vector of the action mask to the probability of selecting an invalid action in the PPO algorithm, the method further includes...

[0133] When evaluating the action using the dominance function, the dominance of the action is normalized.

[0134] In this embodiment, during the iterative training of the POMDP model using the loss function, the relative merits of actions are quantified through the dominance function, providing encouragement or inhibition for policy updates, thus achieving stable training and accuracy adjustment. In this embodiment, the output dominance of the dominance function is normalized to have zero mean and unit standard deviation, thereby further eliminating the influence of rewards at different scales, reducing the variance of gradient updates, making the training process more stable, facilitating the adjustment of hyperparameters such as the learning rate, and improving the convergence efficiency and final performance of the PPO algorithm.

[0135] The method described in this specification enables the coordinated scheduling of multiple satellites to conduct real-time observations of mobile targets at uncertain locations, rapidly determining satellite scheduling strategies and improving the success rate of observations. Furthermore, by combining the coverage relationship between the satellite's imaging stripe and the target area, as well as the satellite's own resource constraints, the effectiveness of the satellite's imaging actions is ensured, enhancing system reliability. Through an end-to-end reinforcement learning framework, the current state, future coverage capabilities, and target uncertainty information of multiple satellites can be comprehensively considered. By learning collaborative strategies, the optimal allocation of resources across the entire satellite constellation can be achieved. This avoids the limitations of traditional heuristic methods or single-satellite decision-making, enabling the discovery of better collaborative observation patterns, thereby improving the overall observation efficiency and mission completion rate of the constellation.

[0136] like Figure 3The diagram shows a flowchart of a multi-satellite imaging scheduling strategy generation method for mobile target observation according to an embodiment of this specification. The diagram illustrates the modeling and training process for mobile targets. The order of steps may not necessarily follow the example shown in the diagram; other orders are also possible. The scheduling strategy generation method of this embodiment specifically includes:

[0137] Step 301: Generate a multi-star imaging reinforcement learning model for movable targets.

[0138] In this step, the generation of a multi-star imaging reinforcement learning model for a movable target can be divided into two sub-steps. The first sub-step includes the precise division of multi-star imaging strips, and the second sub-step includes a POMDP model that can handle the uncertainty of the target's location.

[0139] Imaging payloads (such as optical cameras and SAR sensors) carried by low-Earth orbit imaging satellites typically have a fixed instantaneous field of view (FOV), which limits the swath width of their Earth imaging at a given orbital altitude. When the target area to be observed is large, or when the satellite's transit trajectory does not completely cover the target center, a single imaging session may not be able to fully cover the target area. Therefore, it is necessary to decompose the satellite's potential observation capability of the target area into a series of feasible imaging strips with defined coverage areas.

[0140] For the first sub-step, you can refer to... Figure 4 The diagram shown is a flowchart of imaging strip division according to an embodiment of this specification. This diagram describes the precise division of imaging strips for multi-star imaging, specifically including:

[0141] Step 401: Obtain data for dividing imaging strips based on satellite information.

[0142] In this step, for example, the task time window [T] can be obtained. start ,T end The data includes the predetermined time period, the geographical range of the target area (such as a rectangular area defined by latitude and longitude), the orbital elements of the satellite constellation participating in the scheduling, and the latitude and longitude of the corresponding endpoints of the imaging strip can also be obtained through the latitude and longitude of the nadir point, the satellite's coverage angle to the ground, and the orbital inclination. Among these, the latitude and longitude of the nadir point and the orbital inclination are usually known, while the satellite's coverage angle to the ground can be obtained from the satellite's side view and the satellite's field of view.

[0143] For each satellite in the constellation, the sequence of visible time windows during its flyby of the target region within the mission time window is predicted using an orbital dynamics model (such as SGP4 / SDP4). For each visible time window, the nadir coordinates at the start and end times are calculated, along with the corresponding orbital altitude h at the start time. start and the orbital height h at the end timeend .

[0144] Step 402: Generate discrete satellite-side view sequences.

[0145] In this step, a discrete sequence of side-view angles (angles of the side-view angles) is generated based on the available maximum side-view angle range of the satellite imaging payload and the user-defined side-view angle discrete step size. The maximum side-view angle range can be θ from the nadir point to both sides. max Degree, the θ max The range can be, for example, [-24°, 24°], where the side angle is a distance from the walk length Δθ. side For example, it can be 6°, and the discrete side view sequence is θ. side,1 θ side,2 ,…θ side,N These discrete side views represent the different directions in which the satellite can point the sensors of the imaging payload.

[0146] Step 403: Calculate the edge angle of the imaging strip.

[0147] In this step, for each discrete side view θ side,j Combined with the field of view θ of the satellite imaging payload FOV (For example, 12°), calculate the pointing angles of the two edges that make up an imaging strip relative to the nadir point. For example, using θ side,j If the direction is the center line of the imaging strip, then the angle between the two edges of the imaging strip can be approximated as α. edge1,j =θ side,j -θ FOV / 2 and α edge2,j =θ side,j +θ FOV / 2.

[0148] At the same time, the curvature of the Earth and the attitude of the satellite must also be considered:

[0149] Satellite ground coverage angle d σ The angle between the line connecting the Earth and the Earth's surface and the normal to the Earth can be given by the following formula:

[0150]

[0151] Where, α σ R is the satellite imaging angle (the angle between the line connecting the satellite and the ground and the satellite's orientation towards the sky). e Let be the Earth's radius, and h be the satellite's instantaneous altitude. For example... Figure 5 The diagram shown is a schematic diagram of the calculation of the vertex of the imaging strip in an embodiment of this specification. Points A and B in the diagram correspond to the ground projections of the two edge points of the short side of the imaging strip at a certain instant, i is the orbital inclination angle of the satellite, and S' is the nadir point. The satellite's coverage angle at point A. This is the satellite's coverage angle at point B. The latitude and longitude coordinates of points A and B can be determined based on the latitude and longitude coordinates of S' and... and And i is calculated.

[0152] Step 404: Calculate the geographic coordinates of the endpoints of the imaging strip.

[0153] In this step, for each discrete time t within each visible time window (or within a time period, assuming the orbital parameters do not change significantly), and the pair of short-side edge angles of the imaging strip calculated in step 403, Here, edge1 and edge2 are the two short sides of an imaging strip.

[0154] Obtain the latitude and longitude (L) of the nadir point at that moment. s (t),B s (t) and orbital height h(t).

[0155] The satellite's orbital inclination is known to be i.

[0156] The edge angles of the two short sides of the imaging strip Sub-satellite latitude and longitude (L) s (t),B s Substituting the orbital height h(t) and orbital inclination i into the spherical trigonometric transformation formula (as shown in the generalized form below), the latitude and longitude coordinates of the four vertices of the instantaneous imaging strip on the ground are calculated:

[0157]

[0158] Where P represents the endpoint of the imaging strip; Az is the azimuth angle of the ground point viewed from the nadir point, which is related to the satellite's flight direction and side view direction; Lp is the longitude of the imaging strip endpoint; Ls is the longitude of the nadir point; Bp is the latitude of the imaging strip endpoint; and Bs is the latitude of the nadir point. These represent the satellite imaging angles at the corresponding endpoints. For the four endpoints of the imaging strip, the start and end times of the imaging strip need to be considered.

[0159] By iteratively processing all discrete side views and all visible time windows, the set of all possible imaging stripes for the satellite during the entire mission is obtained, denoted as C. strips Each imaging strip is defined by the geographic coordinates of its four endpoints, the corresponding satellite, and the imaging time window. For example... Figure 6 The diagram shown is a schematic diagram of the imaging strip division of multiple satellites in an embodiment of this specification. In the diagram, the target area (red square area) is divided into 9 grids, and the slanted rectangular area is the imaging strip of multiple satellites. Points A and B are the endpoints of the short side of the aforementioned imaging strip.

[0160] The algorithm described above can transform the complex satellite coverage problem into a discrete, manageable imaging strip selection problem, which will serve as the basic elements of the action space in the subsequent POMDP model.

[0161] In some embodiments, the mission time window can be set to a full calendar day, such as Coordinated Universal Time (UTC) from 04:00:00 on 2022 / 12 / 12 to 04:00:00 on 2022 / 12 / 13.

[0162] The target region can be defined as a rectangular geographical area, such as a longitude range [Lon]. min Lon max Latitude range min ,Lat max In this embodiment, the target area is set to a region of 1.5°*1.5°.

[0163] The target area can be divided into grid partitions. To facilitate the description of the location probability distribution (belief state) of the moving target, the target area is uniformly divided into grid granularities of 0.5°*0.5°, forming 3*3=9 discrete geographic grids, each with its unique number.

[0164] The generation of the visible time window sequence is based on a pre-defined constellation of low-Earth orbit imaging satellites (e.g., satellites numbered Sat1, Sat2, ..., Sat16, whose orbital elements are known) and specialized orbital analysis software (such as the open-source Orekit). This calculates the visible time window for each satellite over the pre-defined target area within the mission's time window. The results are shown in Table 1. These time windows are a fundamental prerequisite for satellites to observe the target area.

[0165] Table 1 shows the time window sequence.

[0166]

[0167] The imaging characteristics of a satellite's imaging payload can be simulated. For example, key parameters of the imaging payload's imaging characteristics include, but are not limited to, those shown in Table 2:

[0168] Table 2 Parameters of Imaging Load Imaging Characteristics

[0169]

[0170] Side-Looking Angle Range: For example, [-24°, +24°], which represents the maximum angle at which the satellite sensor can swing to both sides from the nadir point.

[0171] Side-Looking Angle Discretization Step: For example, 6° means that within the side-looking angle range, a series of discrete pointing angles are taken at 6-degree intervals to generate different imaging strips.

[0172] Field of View (FOV): For example, 12°, which indicates the angular range that the sensor can cover in a certain direction, directly affecting the width of the imaging strip.

[0173] Detection Probability (P) d For example, it can be 0.9, which means that if the target actually exists within the imaging strip and the satellite successfully images the imaging strip, there is a 90% probability that the target can be successfully detected.

[0174] False Alarm Probability (P) fa For example, 0.1 can be applied to complex observation models or belief updates.

[0175] To address the problem of multi-satellite collaborative observation of moving targets in the second sub-step, a model is constructed. Since the precise location of moving targets (such as ships on the sea or vehicles on the ground) is often unknown or uncertain at the time of planning and decision-making, it can only be probabilistically estimated through historical information, prior knowledge, or incomplete sensor data. In the embodiments of this specification, the Partially Observable Markov Decision Process (POMDP) ​​framework is used to model this problem.

[0176] The resulting POMDP model consists of a six-tuple data structure, where each six-tuple comprises (S, A, T, R, Ω, O), and:

[0177] S is the state space, used to define the true state of the system. Here, the state s∈S is not only the physical state of the satellite, but also includes a probabilistic description of the position of the movable target.

[0178] Wherein, state s t This is a composite vector, including belief state, satellite coverage, and the current time:

[0179] Belief state (b) t): Describes the probability distribution of the location of a moving target at time t. The entire target area is divided into Ng discrete geographic grids C1, C2, ... C Ng Belief state b t ={p t (C1),p t (C2),…p t (C ng )}, where p t (C g ) is the location of the movable target at time t in grid C. g The probability within, and The update of belief states follows the principle of Bayesian filtering:

[0180]

[0181] in, It is a prior probability predicted based on the target motion model. In obtaining observation o t The posterior probability after observation (o) at the current time t. P(o) t |s t (C g )) is an observation model, It is the prior probability, P(o t |s t (C k )) is the observation model.

[0182] Satellite coverage: Detailed status parameters of each satellite at the current moment, such as: (a) operational status (e.g., normal, fault, maintenance, etc.); (b) remaining on-board resources (e.g., available energy, remaining storage capacity, etc.) and their expected consumption rate; (c) current attitude parameters and maximum attitude maneuverability (e.g., maximum angular velocity, maximum angular acceleration limit, etc.); (d) a list of visible time windows for the target area within the next N decision cycles (or within the preset planning time domain). Each window must specify its start and end times and be associated with a reference or identifier of a specific set of imaging strips that can be generated based on the current satellite status.

[0183] Current time (t): a timestamp used in the decision-making process.

[0184] As can be seen, the state space S is a high-dimensional hybrid space containing continuous probability values ​​and discrete satellite state information. The beginning of each visible time window can be regarded as a decision time slot, corresponding to a state s. t .

[0185] A represents the action space: at each decision time t (corresponding to a state s) tThe intelligent agent (or scheduling system, which may be located at a ground station or on one or more satellites) needs to select one of all currently available satellites and instruct it to perform an imaging action on a specific imaging strip and observe that imaging strip, or choose not to perform any observation (i.e., wait).

[0186] Action a t ∈A is defined as: Selecting satellite Sat k The imaging strip that can be formed within the current or immediately adjacent visible time window. kj To perform imaging actions.

[0187] Action Space A t (s t ) is in state s t The set of all effective imaging strips from all satellites, plus a "Do Nothing" action, i.e., A = U k,j {Strip kj}∪{DoNothing}, when selecting the imaging strip kj When the set of grids it covers is denoted as G(Strip), the total number of grids it covers is denoted as G(Strip). kj ).

[0188] T is the state transition function: P(s) t+1 =s'|s t =s,a t =a), which represents the probability that the system will transition to the next state s' after the action a is performed in the current state s.

[0189] In the embodiments described in this specification, the state transition includes a deterministic transition portion and a stochastic transition portion:

[0190] Among them, deterministic transitions are: the passage of time and the evolution of satellite orbits (leading to changes in the visible time window and available imaging strips) are deterministic.

[0191] Random transitions: These primarily manifest in the positional evolution of movable targets. Movable targets move between grid points within a target region according to their own motion model (which may be random or follow a certain pattern). This part is typically modeled as a probability transition matrix P. trans (C g’ |C g ), indicating that the target is from grid C. g Move to C g’ The probability of the belief state prior update. This transfer model will be used.

[0192] Conditional transition probability P(w) t+1 |w t ,at In the multi-star imaging scheduling problem, the agent, within each visible time window w... t Making decisions involves transitions between states (referring to the next decision-making time window w). t+1 The occurrence of [a satellite] is deterministic and independent of the agent's current state and the chosen action (unless the action affects the satellite's future availability, such as energy consumption). The finite set of visible time windows for all satellites passing over the target area during the mission time can be expressed as: W = {w1, w2, ... w} M}

[0193] R is the reward function R(s,a): it defines the immediate reward that the agent receives after performing action a in state s, and is the target of reinforcement learning algorithm optimization.

[0194] The reward function should be designed to guide the agent to learn a strategy that maximizes the benefits of observing moving targets. The reward function defined in the embodiments of this specification is: selecting imaging strip a. t =Strip kj The reward obtained is the total number of grids C covered by the imaging strip. g ∈G(Strip kj The sum of the expected observation benefits of the satellite is multiplied by the observation duration of the satellite (or a weighting factor related to the duration).

[0195]

[0196] in, Before performing an action, the movable target is located in grid C. g The prior probability (from belief state b) t ).

[0197] P d (g,a t ) for grid C g Inside, through action a t The probability of successfully detecting (or confirming) a mobile target (can be set as a constant, or related to sensor type, weather, target characteristics, etc.).

[0198] V g It is the observation grid C g The value or priority (can be set to 1 if all grids are equally important).

[0199] t start (a t ) and t end (a t ) are the actions to be performed. t The start and end times of (i.e., the actions of imaging the imaging strips).

[0200] If you choose not to take any action, the reward is 0 or a small negative value (representing the opportunity cost).

[0201] Ω is the set of observations: Since the state is partially observable, the agent cannot directly obtain the true state s, but rather perceives it indirectly by observing o∈Ω.

[0202] Observe o t It can include:

[0203] Results after performing the imaging action: for example, successful image acquisition, image quality, whether the target was detected within the coverage area, etc.

[0204] Updates to target information from other information sources;

[0205] Feedback on the satellite's own status (such as energy consumption, fault information, etc.).

[0206] A key observation in the embodiments of this specification is the impact of imaging results on the probability distribution of the location of movable targets.

[0207] O is the observation function (O(o|s',a)): P(o t =o|s t+1 =s',a t =a) represents the probability that the agent gains observation o after performing action a and causing the system to transition to state s'. For example, if the real movable target is in grid C g The information is contained within (state s'), and action a covers C. g Then, observing that "the movable target is in C" g The probability P(o=”target in C) is... g "|s',a) will be higher. If action a does not cover C." g Then it is observed that "the movable target is in C" g The probability of "is low".

[0208] The observation function is used to update the belief state b. t .

[0209] The model constructed using the POMDP described above can provide a model foundation for solving this complex scheduling problem using deep reinforcement learning algorithms (such as the PPO algorithm).

[0210] Step 302: Train the POMDP model using the improved PPO algorithm.

[0211] In this step, training the POMDP model using the improved PPO algorithm is just one example. Other reinforcement learning algorithms with action networks and evaluation networks can be used, such as the A2C (Advantage Actor-Critic) algorithm, the A3C (Asynchronous Advantage Actor-Critic) algorithm, and the TRPO (Trust Region Policy Optimization) algorithm.

[0212] In satellite imaging mission planning, the coverage of target areas by each satellite varies across different time windows, causing the effective imaging strips that can be generated to dynamically change with the state (time, satellite position, probability distribution of movable targets). For many states, performing certain actions (selecting certain imaging strips) is ineffective (e.g., the imaging strip does not cover high-probability target areas, the satellite cannot point to the imaging strip at that moment, or selecting the imaging strip would violate energy / storage constraints, etc.). Ineffective action masks can offset the probability of selecting these ineffective actions, thus avoiding exploring them. Traditional PPO indirectly handles these constraints through reward function design (providing extremely poor negative rewards for ineffective actions). This approach cannot guarantee that the agent completely avoids ineffective actions during training, potentially failing to ensure that the finally learned policy always satisfies all hard constraints. Furthermore, exploring a large number of ineffective actions can lead to excessively long training times, making it difficult to meet the requirements of online real-time scheduling.

[0213] like Figure 7 The diagram shown is a flowchart of the improved PPO algorithm according to an embodiment of this specification. The improved PPO algorithm is described in this diagram, and see also... Figure 8 The diagram shown is a schematic representation of the improved PPO algorithm according to an embodiment of this specification. It comprises two parts: a first PPO network and an updated second PPO network. These two parts form an iterative loop for the PPO algorithm, used to solve the POMDP model. This improved PPO algorithm includes...

[0214] Step 701: Initialize the various functional modules in the PPO algorithm.

[0215] In this step, the action network and evaluation network in the first PPO network and the second PPO network are initialized. The action network and evaluation network can share some underlying parameters using the same network structure, or they can be independent networks.

[0216] In the embodiments of this specification, the action network output layers in the first PPO network and the second PPO network do not directly use activation functions (or use linear activation), but instead directly output the raw, unnormalized probabilities logits of all actions to the Mask module. The evaluation network output state value is then used.

[0217] In the initialization step, the experience pool D can also be initialized. Each experience stored in the experience pool D is a tuple data structure, such as a 7-tuple (s t a t ,logp(s t |a t ), r t V(s) t ), done, mask), where s t Given the current state, a t For the action I selected based on the old satellite scheduling strategy, logp(s) t |a t ) for s t Choose action a t The logarithmic function of the transition probability, r t To perform action a t The immediate reward obtained afterward, V(s) t To evaluate the network's estimated current state s t The value of is , where 'done' is a boolean flag indicating whether the current round has ended, and 'mask' is the current state 's'. t The corresponding action mask vector.

[0218] You can also set hyperparameters, such as the learning rate η. θ η φ Discount factor γ, GAE (Generalized Advantage Estimation) parameter λ, PPO clipping factor ε, batch size M batch PPO update round number K epoch wait.

[0219] Step 702: Train the first PPO network.

[0220] In this step, at the beginning of each training iteration, the action network of the first PPO network is based on the initial state s input by the environment processing unit. t Output the unnormalized probability of choosing the original action, logits(s). t The evaluation network evaluates the initial state s input from the environmental processing unit. t Output the value of this state.

[0221] The environmental processing unit also generates a corresponding action mask for each action and inputs the action mask into the Mask module. The action mask can be generated based on the coverage relationship between the satellite's imaging stripe at a predetermined time and the target area. The predetermined coverage relationship means that the imaging stripe covers at least part of the target area at the current time or at some future time. It is determined whether the satellite's imaging stripe at the predetermined time can form a coverage relationship with the target area. If the determination result is yes, the satellite's action of generating the imaging stripe is a valid action, and the generated action mask is a valid action mask; if the determination result is no, the satellite's action of generating the imaging stripe is an invalid action, and the generated action mask is an invalid action mask. In the embodiments of this specification, for example, the target area is divided into 9 grids. If the imaging strip of a satellite at a preset time fails to cover more than 50% of any grid, the imaging action of that satellite's imaging strip is identified as an invalid action, and the invalid action mask generated accordingly can be -∞. If the imaging strip of a satellite at a preset time can cover more than 50% of any grid, the imaging action of that satellite's imaging strip is identified as a valid action, and the valid action mask generated accordingly can be 0. Specific judgment methods can also refer to the aforementioned methods for identifying actions as valid actions and generating valid action masks, and for identifying actions as invalid actions and generating invalid action masks, which will not be repeated here. It should be noted that the methods for determining the validity of signing actions and generating valid and invalid action masks can be executed individually, or some of the aforementioned methods can be combined into a composite judgment and identification method. Such combinations and arrangements are all within the scope of protection of this application.

[0222] The action network outputs logits(s) t The input is fed into the Mask module, where it is processed, as described in the previous embodiment. This allows the logits(s) output by the action network to be processed. t Adding the action mask to the action mask can cancel out the probability of selecting an invalid action, since the invalid action mask has a value of -∞, without affecting the probability of selecting a valid action.

[0223] After processing by the Mask module, the processed action probability, i.e., the probability of selecting a valid action, is the logits(s) output by the action network. t The probability of selecting an invalid action is -∞. This value is input into an activation function, such as the Softmax function (or the Tanh activation function), to calculate the normalized probability distribution π of selecting that action. θ (·|s t ), where · represents the probability of choosing that action.

[0224] Choose an action a from the probability distribution of choosing that action. t .

[0225] This action is performed in the environmental processing unit. t Obtain partial observations of the next state, update the new year state to get s t+1 And receive an instant reward r t The log function of the state transition probability is logp(a) t |s t ) and the completion marker "done".

[0226] The environmental processing unit outputs the state value V(s) of the current state st from the evaluation network of the first PPO network. t ), forming a 7-tuple of empirical data (s t a t ,logp(s t |a t ), r t V(s) t ), done, mask).

[0227] The environmental processing unit stores the experience data into the experience pool D.

[0228] Step 703: Select a batch of experience data from the experience pool to train the POMDP model.

[0229] In this step, from including M batch A batch of empirical data is randomly collected from a random pool of empirical data. The empirical data in this batch includes the empirical data seven-tuples as described in the steps above.

[0230] Step 704: Train the second PPO network.

[0231] In this step, the acquired batch of empirical data is input into the action network, evaluation network, loss function module, and mask module of the second PPO network. The action network generates an unnormalized probability of selecting an imaging strip for imaging action based on the state in the empirical data. The evaluation network generates a state value V'(batch_s) for that state based on the state in the empirical data. t The probabilities output by the action network are input into the Mask module to process invalid actions. Then, the probabilities of canceling invalid actions are input into an activation function, such as the Softmax function in this example. The activation function calculates a normalized probability distribution for selecting the action, and the probability distribution output by the activation function is input into the loss function module. The state value output by the evaluation network is input into the loss function module, where generalized advantage estimation (GAE) is used to calculate the advantage. The dominance score is then normalized to obtain the normalized dominance score.

[0232] The loss function module includes three loss functions for each empirical data point in the current batch:

[0233] (1) Action network loss function L CLIP (θ):

[0234] Current strategy π θ The probability of choosing action a in state s (after processing with the Mask module): π θ (a|s).

[0235] Probability ratio: in, The probability of choosing action 'a' in the previous state (after processing with the Mask module).

[0236] The loss of the network at the current time t is: Where min(·) is used to select the minimum value. The normalized dominance is represented by clip(·,·,·), which is a clipping function; ε is the clipping range hyperparameter.

[0237] The loss function of the action network is: in, This is the expected value.

[0238] (2) Evaluate the network loss function L VF (φ):

[0239] The evaluation network loss at time t is: Where φ represents the evaluation network parameters, and V φ (s) Evaluate the network output, R t For cumulative returns;

[0240] The loss function of the evaluation network is: in, The expected value is used to evaluate the network's loss function, which enables the network to estimate the state value more accurately.

[0241] (3) Entropy regularization term L S (θ):

[0242] The This entropy regularization term can encourage policy exploration.

[0243] in, Let S[π] be the expected value. θ ](s)=-Σ a π θ(a|s)logπ θ (a|s) represents the entropy of the policy in state s.

[0244] The entropy regularization term is optional, and in some embodiments, it may be omitted.

[0245] The weighted sum of the above three loss functions is used as the loss function module for the second PPO network:

[0246] L CLIP+VF+S (θ+φ)=c1L CLIP (θ)+c2L VF (φ)+c3S[π θ ];

[0247] Where c1, c2, and c3 are weighting coefficients, and L CLIP (θ) is the action network loss function, L VF (φ) is the evaluation network loss function, S[π] θ ] represents the entropy of the strategy.

[0248] Based on the calculated loss function L CLIP+VF+S (θ+φ) are used to update the action network parameters and evaluation network parameters respectively using gradient descent (e.g., Adam optimizer), thereby minimizing the corresponding loss function.

[0249] Step 705: Based on the updated parameters, iterate continuously until the predetermined conditions are met.

[0250] In this step, the parameters of the first PPO network are updated with the updated parameters of the second PPO network, the experience pool is updated based on the current policy, and the process returns to step 702 until convergence or the predetermined number of training steps is reached.

[0251] The embodiments in this specification introduce an action masking mechanism, which directly eliminates a large number of physically infeasible or inefficient actions early in the policy exploration process, greatly reducing the effective search space. This allows the reinforcement learning agent to focus more on promising action regions, thereby significantly accelerating the convergence speed of the learning process. Figure 9The figure shown is a simulation result diagram of the embodiments of this specification. The convergence speed of the method in the embodiments of this specification is illustrated in the figure. The figure shows the curves of the average cumulative reward of three different algorithms (PPO+mask+advNor (normalized dominance), PPO+advNor, and standard PPO) under the same training environment and task, as a function of the number of training episodes. PPO+mask+advNor (red curve) shows a faster convergence speed and a higher final average reward, proving the effectiveness of the method in the embodiments of this specification. Compared to the standard PPO algorithm (blue curve), its average reward curve is significantly improved after approximately 500 training episodes. The horizontal axis represents episodes, and the vertical axis represents reward. The standard PPO algorithm did not reach the same level after 3000 episodes, showing a significant improvement in convergence speed (e.g., the speed of reaching a certain reward threshold is increased by approximately X times, or the average reward is about Y% higher under the same number of training episodes). Combined with the stabilizing effect of dominance normalization on the training process, the algorithm is more likely to find effective strategies with high returns, and the final average reward (e.g., ...) is significantly higher. Figure 9 As shown, approximately 240) is significantly higher than the comparison algorithm.

[0252] The method described in this specification incorporates an action masking mechanism, fundamentally ensuring that the selected imaging actions always meet key physical constraints (such as satellite attitude reachability, energy limitations, etc., as long as these constraints are incorporated into the masking logic) during training and final decision-making. This avoids generating impractical or unexecutable scheduling instructions, greatly enhancing the operability and reliability of the scheduling results in actual satellite systems.

[0253] To address the challenges of dynamic environmental changes and model uncertainties in satellite scheduling, the methods described in this specification demonstrate excellent robustness and adaptability. For example... Figure 10 The diagram illustrates a comparison of the effectiveness of embodiments of this specification with other solutions. Training was conducted under three different initial target probability distributions and transition probability models (prob1, prob2, prob3). The method described in the embodiments of this specification can learn effective scheduling strategies and obtain comparable high average rewards (e.g., approximately 275 for prob1, approximately 230 for prob2, and approximately 345 for prob3), demonstrating its adaptability to changes in the prior target model. Furthermore, as... Figure 11 The diagram illustrates the average reward during strategy training in an embodiment of this specification. In more challenging online training scenarios (where the initial target position probability and transition probability are randomly initialized at the start of each training round), the method (PPO+mask+advNor) in this embodiment can still continuously learn and maintain a high average reward level (stable above 320), demonstrating its strong online learning and adaptability in dynamically changing environments. Ultimately, as... Figure 12 The diagram shown illustrates the average reward of the strategy in the embodiment of this specification. The results of 50 independent tests on the trained online strategy (with initial and transition probabilities randomly initialized in each test) show that the average reward value remains at a high and relatively stable level under different random scenarios (for example, the average reward value fluctuates between 295 and 335 with a small standard deviation). This fully demonstrates the robustness and generalization ability of the generated strategy, thereby effectively meeting the task requirements of online real-time optimization scheduling and solving the problem that traditional methods are difficult to adapt to dynamic environments.

[0254] like Figure 13 The illustration shows a satellite scheduling strategy generation device according to an embodiment of this specification. It can be located at a ground station or in a network, or it can be integrated onto a satellite. It is implemented using a dedicated chip or a general-purpose chip. Some functional units and modules can be implemented by a single chip or hardware device, or a single functional unit or module can be implemented by several chips or multiple hardware devices. The device in this embodiment includes:

[0255] The strip division unit 1301 is used to obtain multiple imaging strips based on the imaging characteristics of the imaging payloads of satellites related to the target area; wherein the target area includes movable targets;

[0256] Modeling unit 1302 is used to establish a partially visible Markov decision POMDP model for the target region;

[0257] The environment processing unit 1303 is used to generate an action mask based on the relationship between the imaging strip of the satellite and the target area; wherein the action mask includes a valid action mask and an invalid action mask;

[0258] The model training unit 1304 trains the POMDP model based on the action mask to generate the satellite scheduling strategy.

[0259] Corresponding to the above-mentioned communication measurement configuration method on the base station side, this specification also provides an intelligent agent, including: at least one processor; and at least one memory storing instructions thereon, wherein when the instructions are executed individually or jointly by the at least one processor, the intelligent agent performs the above-mentioned satellite scheduling strategy generation method.

[0260] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0261] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0262] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0263] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0264] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0265] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0266] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0267] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0268] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0269] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0270] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0271] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0272] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating satellite scheduling strategies, characterized in that, include, Based on the imaging characteristics of the imaging payloads of satellites related to the target area, multiple imaging strips are obtained; wherein, the target area includes movable targets; A partially visible Markov decision POMDP model is established for the target region; An action mask is generated based on the relationship between the satellite's imaging stripe and the target area; wherein the action mask includes a valid action mask and an invalid action mask; The POMDP model is trained based on the action mask to generate the satellite scheduling strategy.

2. The method according to claim 1, characterized in that, Generating an action mask based on the relationship between the satellite's imaging stripes and the target region further includes: Determine whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area; If so, the imaging action of the satellite at the predetermined time is identified as a valid action; If not, the imaging action of the satellite at the predetermined time is identified as an invalid action; The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

3. The method according to claim 2, characterized in that, Determining whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area further includes, Determine whether the satellite has maneuvered to an attitude that satisfies the requirement of covering at least part of the target area at a predetermined time.

4. The method according to claim 2, characterized in that, Determining whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area further includes, Determine whether the imaging strip of the satellite at a predetermined time covers at least one target area with a non-negligible probability of the presence of a movable target.

5. The method according to claim 2, characterized in that, Determining whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area further includes, The target area is divided into multiple grids; Determine whether the area ratio of the imaging strip of the satellite covering a grid at a predetermined time is greater than a preset threshold.

6. The method according to claim 5, characterized in that, The preset threshold range includes preset values ​​between (0, 1).

7. The method according to claim 2, characterized in that, Determining whether the imaging stripe of the satellite at a predetermined time covers at least part of the target area further includes, Determine whether the predetermined time is within the current or upcoming valid observation time window.

8. The method according to claim 1, characterized in that, Generating an action mask based on the relationship between the satellite's imaging stripes and the target region further includes: When the satellite performs imaging operations on the target area at a predetermined time, it determines whether the satellite's resources meet the resource constraints. If so, the imaging action of the satellite at the predetermined time is identified as a valid action; If not, the imaging action of the satellite at the predetermined time is identified as an invalid action; The valid action mask is generated based on the valid action, and the invalid action mask is generated based on the invalid action.

9. The method according to claim 8, characterized in that, The resource constraints include at least one of the following: the satellite's energy reserves, the satellite's remaining data storage capacity, the satellite's remaining computing resources, or the satellite's environmental data.

10. The method according to claim 1, characterized in that, The generated action mask further includes, The mask vector of the valid action mask is a value that does not affect the probability of selecting the valid action, and the mask vector of the invalid action mask is a value that cancels out the probability of selecting the invalid action.

11. The method according to claim 10, characterized in that, Training the POMDP model based on the action mask to generate the satellite scheduling strategy further includes: The POMDP model is trained using a reinforcement learning algorithm; The satellite scheduling strategy is generated by offsetting the probability of selecting invalid actions in the reinforcement learning algorithm using the action mask.

12. The method according to claim 11, characterized in that, The probability of selecting invalid actions in the reinforcement learning algorithm by canceling out the action mask further includes... The mask vector of the action mask is added to the probability of selecting an invalid action in the reinforcement learning algorithm, so that the probability of the invalid action corresponding to the invalid action mask being selected approaches 0.

13. The method according to claim 12, characterized in that, After adding the mask vector of the action mask to the probability of selecting an invalid action in the reinforcement learning algorithm, it also includes... When evaluating the imaging action using the dominance function, the dominance of the action is normalized.

14. The method according to claim 12, characterized in that, The probability of selecting an invalid action in the reinforcement learning algorithm by canceling out the action mask further includes... A new action is output based on the probability of selecting a valid action; A new action mask and a new state are generated based on the new action; wherein, the new action mask is used to iteratively offset the probability of selecting an invalid action in the reinforcement learning algorithm, and the new state is used to update the action network and evaluation network in the reinforcement learning algorithm.

15. The method according to claim 14, characterized in that, Based on the new action, a new action mask and a new state are generated, which further include... Generate empirical data including a tuple data structure containing action masks; The experience data is stored in the experience pool of the reinforcement learning algorithm.

16. The method according to claim 15, characterized in that, The empirical data consists of 7-tuple data (s t a t ,logp(s t |a t ), r t V(s) t ), done, mask), where s t Given the current state, a t For the action I selected based on the old satellite scheduling strategy, logp(s) t |a t ) for s t Choose action a t The logarithmic function of the transition probability, r t To perform action a t The immediate reward obtained afterward, V(s) t To evaluate the network's estimated current state s t The value of is , where 'done' is a boolean flag indicating whether the current round has ended, and 'mask' is the current state 's'. t The corresponding action mask vector.

17. A satellite scheduling strategy generation device, characterized in that... include, A strip division unit is used to obtain multiple imaging strips based on the imaging characteristics of the imaging payloads of satellites related to the target area; wherein, the target area includes movable targets; A modeling unit is used to establish a partially visible Markov decision POMDP model for the target region. An environment processing unit is used to generate an action mask based on the relationship between the imaging stripe of the satellite and the target area; wherein the action mask includes a valid action mask and an invalid action mask; The model training unit trains the POMDP model based on the action mask to generate the satellite scheduling strategy.

18. An intelligent agent, characterized in that, include: At least one processor; as well as At least one memory storing instructions that, when executed individually or jointly by the at least one processor, cause the agent to perform the method according to any one of claims 1 to 16.

19. A computer storage medium storing instructions thereon, characterized in that, When the instructions are executed individually or jointly by at least one processor of a computer device, the computer device performs the method according to any one of claims 1 to 16.

20. A computer program product, comprising instructions, characterized in that, When the instructions are executed individually or jointly by at least one processor of a computer device, the computer device performs the method according to any one of claims 1 to 16.

21. A chip, characterized in that, The chip includes a circuit system configured to perform the method according to any one of claims 1 to 16.

Images