ViT online scheduling and sequence parallel reasoning method for low-orbit satellite constellation

Abstract

The invention discloses a low earth orbit satellite constellation-oriented ViT online scheduling and sequence parallel reasoning method, and belongs to the technical field of satellite communication and artificial intelligence fusion calculation. The method comprises the following steps: in response to task state update, triggering online scheduling; determining an optimal patch granularity and sequence allocation proportion for each task through alternative optimization iteration of patch granularity selection (discrete decision) and cross-satellite sequence allocation proportion (continuous decision); and dividing the task token sequence into subsequences according to an optimization result, distributing the subsequences to a plurality of satellites, and performing multi-hop key value pair exchange by each satellite through a ring topology so as to cooperatively complete global self-attention calculation. According to the method, the problem that the ViT model is difficult to realize high-efficiency and low-delay distributed online reasoning under the condition that the low-orbit satellite constellation computing power and communication resources are limited is solved, the task average processing delay is remarkably reduced, and the adaptive capacity of the system to the dynamic load and the network topology is improved.

Description

Technical Field

[0001] This invention relates to the field of satellite communication and artificial intelligence fusion computing technology, specifically to a ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations. Background Technology

[0002] Low Earth Orbit (LEO) satellite constellations have been rapidly deployed in recent years, forming a distributed network through inter-satellite links (ISL) to provide continuous coverage and low-latency services for tasks such as remote sensing, target identification, and environmental awareness. However, the computing power, storage capacity, and energy budget of a single LEO satellite are very limited, making it difficult to independently complete real-time inference of complex deep neural network (DNN) models. On the other hand, ground-based offloading methods are limited by visibility windows, long-distance propagation delays, and bandwidth jitter, making it difficult to meet the requirements of high real-time tasks.

[0003] Traditional satellite collaborative inference research has primarily focused on models with local convolutional structures such as Convolutional Neural Networks (CNNs), reducing the load on individual satellites by offloading data across satellites or between satellites and ground stations. However, CNNs only focus on local region features, and their inference performance is limited for long-range dependencies, small target detection, and complex scene understanding in high-resolution remote sensing images. The Visual Transformer (ViT), which has emerged in recent years, significantly improves the performance of tasks such as image classification, remote sensing scene recognition, and change monitoring by utilizing a global self-attention mechanism. However, it requires repeated exchange of large-scale Key / Value (K / V) representations across multiple layers during operation, resulting in a computational and communication load far exceeding that of CNNs.

[0004] Deploying Vit in a constellation environment presents the following key challenges:

[0005] (1) ViT's global attention mechanism requires the model to acquire the complete sequence of key-value information at each layer. If this information is transmitted directly between multiple satellites, it will result in huge bandwidth overhead. ISL's bandwidth resources are limited and dynamically changing, making it impossible to apply traditional CNN offloading strategies.

[0006] (2) Remote sensing missions in the LEO constellation are characterized by suddenness and uncertainty, with random arrival times and variable workloads between missions. Most existing methods are based on offline decision-making, which cannot immediately calculate the optimal strategy when a mission arrives, nor can they cope with the deadline requirements of different missions.

[0007] (3) Existing distributed Transformer inference methods are difficult to adapt to constellation environments.

[0008] (4) The scheduling problem is essentially a hybrid discrete-continuous NP-hard problem.

[0009] Therefore, a new technical solution is needed to enable online granularity selection, dynamic sequence allocation, and cross-satellite sequence parallel inference for ViT inference tasks in the LEO constellation environment, thereby improving overall inference efficiency and task completion rate while meeting real-time constraints. Summary of the Invention

[0010] The purpose of this invention is to provide an online scheduling and sequential parallel inference method for ViT for low-Earth orbit satellite constellations, which solves the technical problem that existing technologies cannot efficiently and in real-time execute distributed inference of visual Transformer models in low-Earth orbit satellite constellation environments. Specifically, it addresses the challenges of limited computing power of constellation nodes, limited bandwidth of inter-satellite links, random arrival of multiple tasks and strict time delay constraints, and provides an online scheduling and parallel inference collaborative optimization method.

[0011] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0012] A ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations, executed by the gateway satellite, includes the following steps:

[0013] Step S1, Task Status Update: In response to the arrival of a new inference task or the completion of an existing task, update the current set of tasks to be processed;

[0014] Step S2, Alternating Optimization Decision: For each task in the task set, perform alternating optimization of patch granularity selection and cross-satellite sequence allocation ratio until the iteration termination condition is met, and output the optimal patch granularity and optimal sequence allocation ratio for each task; wherein, the patch granularity is selected from a predefined discrete set to determine the total number of tokens in the input image; the sequence allocation ratio is a continuous variable used to define the allocation ratio of task token sequences among different satellites; the patch granularity selection is achieved by traversing the predefined discrete set and evaluating the standardized hysteresis estimate, and the sequence allocation ratio is optimized by an optimization algorithm under the conditions of satisfying satellite computing power constraints and inter-satellite link bandwidth constraints;

[0015] Step S3, Distributed Inference Execution: Obtain the token sequence based on the input image of the task according to the optimal patch granularity, and divide the token sequence into sub-sequences according to the optimal sequence allocation ratio and distribute them to multiple satellites in the constellation; each satellite exchanges key-value pair information by passing local key vectors and value vectors hop by hop based on a preset ring communication topology to collaboratively complete global self-attention calculation, and the results are aggregated by the gateway satellite.

[0016] Furthermore, the alternating optimization decision in step S2 specifically includes:

[0017] Step S21, Initialization: Randomly initialize the patch granularity and sequence allocation ratio for all tasks;

[0018] Step S22, solving the first subproblem: Under the condition of fixing the current sequence allocation ratio, for each task, select the patch granularity that minimizes the standardized hysteresis estimate of the task from the discrete set of its corresponding patch granularity, and prune the granularities that do not meet the constraints, and update the optimal patch granularity.

[0019] Step S23, solving the second subproblem: Under the condition of fixing the current optimal patch granularity, the particle swarm optimization algorithm is used to jointly optimize the sequence allocation ratio of all tasks. Under the premise of satisfying the satellite computing power constraint and inter-satellite link bandwidth constraint, the sequence allocation ratio that minimizes the total standardized hysteresis estimate of all tasks is found, and the optimal sequence allocation ratio is updated.

[0020] Step S24, Iteration judgment: Repeat steps S22 and S23 until the preset number of iterations or time budget is reached.

[0021] Furthermore, the standardized hysteresis estimate, Tardiness... i The calculation formula is:

[0022]

[0023] Among them, W i R represents the waiting time for task i. i D represents the estimated task runtime based on the current patch granularity and sequence allocation ratio. i This indicates the deadline for task i.

[0024] Furthermore, in step S22, the method for selecting the patch granularity for each task is as follows: traverse each candidate granularity in its predefined discrete set of patch granularities, calculate the task runtime estimate when using the candidate granularity based on the current sequence allocation ratio, then calculate the standardized hysteresis estimate, and select the candidate granularity with the smallest standardized hysteresis estimate.

[0025] Furthermore, in step S23, when using the particle swarm optimization algorithm for optimization, the following methods are used to ensure that the solution satisfies the constraints:

[0026] For each iteration, a proportion vector is assigned to the candidate sequence. First, a simplex projection is performed to ensure that each proportion value is non-negative and sums to 1. The simplex projection refers to the mathematical operation of mapping any real vector to the standard simplex (i.e., the set of vectors whose components are all non-negative and sum to 1), which can be achieved, for example, by iterative scaling or sorting truncation.

[0027] Subsequently, based on the candidate sequence allocation ratio vector and the current optimal patch granularity, it is verified whether the upper limit constraint of the computing power of each satellite and the upper limit constraint of the bandwidth of each inter-satellite link are met.

[0028] For candidate vectors that violate constraints, constraint processing is performed by adding a penalty term to the optimization objective function; the penalty term is proportional to the degree of constraint violation.

[0029] Furthermore, the distributed inference execution in step S3 specifically includes:

[0030] Step S31, Sequence partitioning and distribution: The gateway satellite partitions the token sequence of a task into subsequences corresponding to the number of satellites based on the optimal patch granularity and optimal sequence allocation ratio of each task, and distributes them through inter-satellite links;

[0031] Step S32, Local key-value pair generation and circular exchange: Each satellite calculates the received local token subsequence to generate the corresponding local key vector and value vector; each satellite sends the local key vector and value vector to the neighboring satellites hop by hop according to the preset circular topology order, and receives the key vector and value vector from other satellites. After multiple rounds of exchange, each satellite obtains the global key vector set and value vector set.

[0032] Step S33, Attention Calculation and Result Convergence: Each satellite performs self-attention calculation based on the local query vector and the global key vector set and value vector set to obtain the output of the local subsequence; each satellite returns the local output to the gateway satellite, which converges the results to obtain the final inference result of the task.

[0033] Furthermore, the ring topology is a unidirectional ring, and the key vector and value vector of each satellite are transmitted along the unidirectional ring. After the number of satellites minus one round of hops, each satellite obtains the key vector and value vector of all satellites.

[0034] Furthermore, the predefined discrete set of patch granularities includes multiple granularity levels that divide the input image into different numbers of tokens, and the number of tokens is negatively correlated with the size of the patch granularity.

[0035] Furthermore, the method also includes an adaptive step: when the number of satellite nodes participating in collaborative inference within the constellation changes or the inter-satellite link status changes significantly, steps S1 to S3 are automatically triggered to re-optimize scheduling and reassign inference tasks.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] This invention decomposes the computational task into two core decisions: patch granularity selection and sequence allocation ratio optimization. It also designs a matching cyclic key-value pair exchange attention computation mechanism. For the first time, this invention systematically solves the fundamental contradiction between the global attention requirement of ViT and the discretization of satellite constellations and limited resources, making it possible to deploy high-performance ViT models in the LEO environment with dual constraints of computing power and bandwidth.

[0038] The online alternating optimization framework constructed in this invention can respond quickly when tasks arrive randomly. By decoupling and iteratively alternating discrete enumeration and continuous optimization (particle swarm optimization), a high-quality scheduling scheme is generated within a finite time budget. It directly optimizes the standardized hysteresis objective, thereby effectively meeting the task deadline constraints, significantly reducing the average processing latency and timeout rate of tasks, and ensuring the real-time performance of the system.

[0039] The multi-hop key-value pair exchange mechanism based on ring topology employed in this invention transforms the communication mode required to achieve global attention from traditional full broadcast to sequential relay transmission, fundamentally alleviating the pressure on scarce inter-satellite link bandwidth. This communication mechanism is linearly related to the number of satellites, making the communication overhead controllable when adding cooperative inference satellite nodes, demonstrating excellent horizontal scalability.

[0040] The entire scheduling and inference process of this invention relies heavily on real-time satellite computing load and link status information for decision-making. When network topology changes (such as satellite nodes joining / leaving) or link communication quality fluctuates, the method can automatically trigger re-optimization and dynamically adjust task partitioning and allocation strategies, thereby ensuring the continuity and performance stability of inference services in a dynamically changing constellation environment. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 To propose the scenario model upon which this method is based, a low-Earth orbit satellite inter-satellite collaborative reasoning model is proposed.

[0043] Figure 2 This is a framework diagram of the ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations proposed in this invention.

[0044] Figure 3 This is a flowchart of the ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations proposed in this invention.

[0045] Figure 4 This is a graph showing the performance of the present invention compared to a comparative method when handling different numbers of tasks.

[0046] Figure 5 The graph shows the performance of the present invention compared to the comparative method under different link communication delays.

[0047] Figure 6 This diagram illustrates the performance of the present invention compared to a comparative method when different numbers of satellites are involved in the operation.

[0048] Figure 7 The image shows an ablation experiment of the algorithm proposed in this invention. Detailed Implementation

[0049] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0050] In the description of the embodiments of the present invention, it should be understood that the terms "length", "vertical", "horizontal", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0052] In this embodiment of the invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this embodiment of the invention according to the specific circumstances.

[0053] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0054] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] Example 1: This example discloses a ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations, including the following steps:

[0057] Based on a network environment consisting of multiple low-Earth orbit satellites with ISL (Inter-Satellite Link), the satellites are connected through a fixed or dynamic grid topology. Each satellite has a processing unit with a certain computing power, inter-satellite links for mission transmission, and communication modules. The system allows multiple missions to arrive simultaneously and queue for scheduling.

[0058] The ViT input image is divided into sequence tokens according to different patch granularities. These token sequences are then proportionally distributed to different satellites. Each satellite performs the following steps: local forward feature extraction, local query / key / value generation, passing key / value features using a ring-shaped exchange mechanism, reconstructing the global context in a multi-hop loop, performing local attention and subsequent network layers, and returning a local output. The main satellite then aggregates these outputs to obtain the final prediction result. This parallel sequence structure reduces cross-satellite full broadcast operations, significantly lowering communication overhead.

[0059] When a task arrives, the online scheduler performs patch granularity selection (discrete decision) and cross-satellite sequence allocation (continuous decision). Granularity selection involves choosing the optimal granularity from a predefined set based on current satellite computing power, link bandwidth, task deadlines, and other metrics, ensuring the overall token quantity best matches the computational load to the current constellation state. Sequence allocation involves distributing the obtained token sequence across multiple satellites in a specific ratio, constructing a continuous variable vector to describe the allocation ratio for each satellite. Both processes jointly affect the total computational latency and the total communication latency.

[0060] To address the hybrid discrete-continuous optimization problem, an alternating optimization framework is designed, comprising subproblem 1 (granularity selection) and subproblem 2 (sequence allocation). Specifically, subproblem 1 involves a pruned finite enumeration of candidate granularities, calculating their corresponding lower bound delay estimates to quickly select feasible granularities and improve online response speed. Subproblem 2 employs Particle Swarm Optimization (PSO) to solve for continuous variables. Granularity selection and sequence allocation are performed alternately until convergence or the upper limit of the number of iterations is reached, resulting in the final scheduling decision.

[0061] Once scheduling is complete, the system enters the actual inference phase, which includes the following steps: distributing tokens based on the scheduling results; each satellite performing local self-attention computation; performing ring-based multi-hop key-value feature exchange; reconstructing the complete context at each layer; aggregating the inference results from all satellites; and outputting the final prediction. The entire inference process requires no central server and can be triggered on any master satellite.

[0062] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described below in conjunction with specific embodiments.

[0063] See Figure 1 This embodiment is based on a collaborative inference system consisting of n low-Earth orbit satellites. The satellites are interconnected via ISL to form a network topology and jointly perform the ViT inference task. Each satellite includes a communication module, a computing processing module, a local storage module, and a status monitoring module, and can independently perform forward computation of a portion of the Transformer layer.

[0064] Any satellite in the system can serve as the mission entry point. When the ViT inference mission arrives at the entry satellite (also known as the gateway satellite), this embodiment initiates the scheduling process through the online scheduling optimizer set on that satellite.

[0065] See Figure 2The online scheduling optimizer is responsible for generating two core scheduling results in real time based on the constellation's current resource status, including the computing load, available bandwidth, link hop count, and task deadlines of each satellite: a patch granularity selection result (selecting the most suitable patch granularity from a preset set of granularities to determine the token sequence length of the task); and a cross-satellite sequence allocation vector (a set of lightweight sequence allocation ratio vectors A). k ={A k,1 A k,2 ,...,A k,n} is used to describe the proportion of token subsequences undertaken by each satellite in the parallel computation of the sequence, so as to ensure load balancing under computing power constraints.

[0066] After generating the scheduling results, the gateway satellite slices the ViT input sequence according to the sequence allocation vector and allocates each subsequence to the corresponding satellite via ISL.

[0067] In the second stage, namely the sequential parallel inference stage, all satellites participating in the inference process perform self-attention layer computations in parallel.

[0068] Specifically:

[0069] (1) Each satellite first performs embedding and linear transformation on the local token subsequence and calculates the local Query, Key and Value;

[0070] (2) Subsequently, each satellite forms a unidirectional loop in topological order {S1→S2→···→S n →S1};

[0071] (3) All satellites transmit local key / value hop by hop in the loop. After n-1 hops, each satellite can collect K / V information from all other satellites, thus achieving consistent construction of the global attention context.

[0072] (4) After obtaining the complete context, each satellite performs self-attention operation on the local query;

[0073] (5) The token output after attention processing is returned to the gateway satellite in reverse path, and the gateway satellite performs MLP (Feed-Forward Network) aggregation or final classification head to generate the final ViT inference result.

[0074] This ring-based K / V switching architecture eliminates the need for full broadcasting between satellites and context aggregation by a central node, effectively reducing communication pressure and enabling stable and efficient Transformer inference in LEO constellations with limited bandwidth and high link hop counts.

[0075] In this embodiment, each visual inference task entering the constellation system is processed based on the ViT structure. To adapt to the computing power and communication bandwidth constraints in the LEO constellation, this invention adopts a multi-granularity patch-token approach, and uses patch granularity as one of the key optimization variables of the online scheduler.

[0076] For any time t, let the set of active tasks be Q(t) = {1,2,...,K}, where each task k∈Q(t) divides the input image into patches and maps them to a sequence of tokens.

[0077] The predefined set of optional patch granularities is G. k ∈{g1,g2,...,g M}. Granularity G k The total number of tokens for task k is determined to be N. k (G k ).

[0078] To achieve parallel execution across satellites, this embodiment defines a sequence allocation vector A for each task. k ={A k,1 A k,2 ,...,A k,n}, which satisfies A k ≥0 and

[0079] According to the allocation vector, satellite S i The number of tokens received is N k,i =A k,i N k (G k The token partitioning result will be used for subsequent Query / Key / Value calculations and the ring-based K / V exchange mechanism.

[0080] At any time t, the system's operating state is:

[0081] The set of activity tasks Q(t);

[0082] The cumulative waiting time W for each task k ;

[0083] The deadline for each task is D. k ;

[0084] The upper limit of computing power C of each satellite i ;

[0085] Bandwidth constraints of inter-satellite links

[0086] To achieve real-time scheduling, this invention predicts the computation and communication latency of each task.

[0087] Calculate the time delay: for satellite S i The number of tokens N for processing task k k,i Its computational delay is estimated to be Where, τ c The computation latency is calculated per token and per layer (μs / token / layer), where D is the number of layers in the Transformer network.

[0088] Communication latency: This invention employs a ring-based Key / Value exchange mechanism, estimating the communication overhead based on the number of tokens. Where L0 is the link startup delay, τ n Here, denoted as token, and denoted as link latency per hop (μs / token / hop), where s is the conversion factor between K / V size and number of tokens.

[0089] Total runtime delay prediction for the mission: Due to the parallel execution of the ViT sequence requiring waiting for all satellites to complete local computation and K / V exchange, the total runtime of mission k is... This latency model reflects the characteristics of computing, communication, and network topology, and is the core basis for scheduler execution optimization.

[0090] Scheduling optimization objective: At each scheduling time, this invention aims to minimize the normalized tardiness of all tasks.

[0091] The optimization problem is If W k +R k ≤D k If the task is completed within the specified timeframe, no penalty is incurred; otherwise, a normalized delay cost is applied. This objective function avoids using additional integer variables and focuses on tasks that are "close to meeting the deadline."

[0092] The scheduling must satisfy the following constraint: the patch granularity must come from a predefined set G. k ∈{g1,g2,...,g M},k-1,...,K;Assignment vectors follow Satellite computing capacity limitations Link bandwidth limitation

[0093] This embodiment constructs a long-running online optimization and scheduling module at the constellation gateway satellite to continuously update the task set Q(t) during the dynamic arrival and completion of tasks, and to patch the granularity G of each task. kand the sequence assignment vector A across satellites k Joint optimization is performed. The entire process follows the approach of "window update - alternating optimization - real-time inference", enabling the system to achieve low latency and high throughput ViT sequence parallel inference under strict LEO constellation resource constraints.

[0094] Since each task has M discrete granularity choices and an n-dimensional continuous assignment vector, joint optimization is a hybrid discrete-continuous optimization problem. Enumerating all granularity combinations for the K tasks has a time complexity of O(M). K The computation is not feasible. Therefore, this embodiment uses an alternating optimization framework to solve the granularity selection subproblem and the sequence allocation subproblem separately, solving for discrete and continuous variables respectively. See [link to relevant documentation]. Figure 3 .

[0095] With one type of variable fixed, the optimal solution for the other type of variable can be obtained using a low-cost method. Therefore, this embodiment employs an alternating iterative structure:

[0096] (1) Fixed allocation vector {A} k}, solve for the patch granularity {G} of each task. k};

[0097] (2) Fixed patch granularity {G k}, solve for the sequence assignment vector {A} k}

[0098] The two steps described above are performed alternately for several rounds, with each round significantly reducing the overall standardized hysteresis objective function, thereby continuously improving the solution.

[0099] After termination, the algorithm outputs the patch granularity G for each task. k And the token allocation ratio A for each mission on each satellite k The gateway immediately segments the token sequence according to the above parameters and distributes it to each satellite, where the constellation performs parallel ViT inference.

[0100] Subproblem 1:

[0101] Patch granularity selection corresponds to the discrete variable G k The core objective of this optimization is to allocate vector {A} in the current sequence. k Under fixed conditions, select the optimal patch granularity for each task individually to minimize the standardized hysteresis loss for that task.

[0102] Specifically, the granularity of each task ranges from a finite set {g1, g2, ..., g...} M Choose from}. This invention adopts a strategy of "task-by-task enumeration + constraint filtering": First, for task k, test all candidate granularities g one by one. m; Calculate the number of tokens N for each granularity k (G k Then, based on the current allocation vector A... k Calculate the number N of tokens at this granularity on each satellite. k,i =A k,i N k (G k Finally, calculate the estimated runtime delay for this granularity. and its standardization lag Finally, the granularity with the smallest standardized latency is selected for each task. The results are then fed into the next stage of continuous optimization.

[0103] Subproblem 2:

[0104] Sequence assignment optimization involves continuous variable A k Its function is to determine the allocation ratio of task tokens across n satellites. Since the objective function is non-convex and non-differentiable, and multiple resource constraints exist, this invention employs Particle Swarm Optimization (PSO) as the primary solution strategy.

[0105] In this embodiment, the sequence allocation variables of all tasks are concatenated into a single particle state vector, allowing each particle to represent a global solution of the system. The particle moves through the search space, iteratively searching for a low-hysteresis allocation scheme.

[0106] After PSO updates the particle positions, this invention projects the particles to satisfy all simplex constraints, all satellite computing power constraints, and all link bandwidth constraints. This process performs fast repair on each particle that violates the constraints, avoids the propagation of infeasible solutions, and ensures that the particle swarm always works within the effective search area.

[0107] The target value for each particle is determined by the formula. The algorithm performs calculations. PSO updates the global optimum based on fitness. The algorithm terminates with a fixed number of iterations or a fixed time budget to ensure linearity. Upon completion, it outputs the current optimal assignment vector.

[0108] To verify the technical advantages of the online scheduling and distributed sequential parallel inference method proposed in this invention, this invention compares and analyzes this method with three representative existing Transformer inference acceleration schemes, covering typical technical paths such as single-node inference, two-node pipelined parallel inference, and multi-node dynamic sharding inference. The comparison includes key performance dimensions such as average normalized latency, timeout rate, link latency sensitivity, and constellation node scalability.

[0109] To evaluate the performance advantages of the present invention, the following three existing methods were selected as baselines:

[0110] Local-only: All Transformer computations are performed locally on a single gateway satellite or edge node, without using any task offloading or multi-node collaboration mechanisms. This method has high computational latency and limited throughput, but it is link-independent.

[0111] The ParaTra method is a parallel Transformer inference framework that divides the model into two parts, deployed on terminal devices and edge servers respectively, increasing throughput through pipelined batch processing. However, its parallel granularity is fixed and cannot be dynamically adjusted according to input size or system load, and it requires hierarchical synchronization, resulting in significant coupling between communication and computation.

[0112] The Voltage method achieves distributed inference across multiple devices by dynamically partitioning the input token sequence, emphasizing reduced communication and improved scheduling flexibility. However, its token partitioning strategy is primarily based on communication optimization and does not include an online scheduling mechanism for time-deadline tasks, leading to performance degradation in high-concurrency scenarios.

[0113] refer to Figure 4 Under different numbers of concurrent Transformer tasks, the method of the present invention exhibits significant deadline-keeping capability: when the number of tasks is low (approximately 30 or less), the method of the present invention can stably maintain zero lag; when the number of tasks increases to 100, the method of the present invention produces only minimal lag, and the growth trend is gradual.

[0114] In contrast, the Voltage method exhibits significant degradation after the number of tasks exceeds 20, with latency reaching high levels under high concurrency scenarios. The ParaTra method experiences rapidly increasing latency with a large number of tasks, and long-tail tasks suffer severe delays. The Local-only method exhibits the highest latency under high load and cannot meet arbitrarily strict deadline requirements. The results demonstrate that this invention, through adaptive patch granularity selection and PSO resource allocation mechanism, can effectively maintain the stability of Transformer inference latency under multi-task impact.

[0115] refer to Figure 5Among all the compared methods, the timeout rate of this invention is consistently the lowest. When the task size ranges from 10 to 40, this invention achieves zero timeout; even with 100 concurrent tasks, the timeout rate is still significantly lower than Voltage, ParaTra, and Local-only. Other methods, however, exhibit significant timeout accumulation under high concurrency conditions. Voltage's timeout rate increases dramatically in scenarios with a large task load; ParaTra, limited by its two-node model partitioning strategy, experiences a significant timeout rate under high load; and Local-only approaches complete timeout when the number of tasks is large. This demonstrates that this invention effectively guarantees the strict real-time requirements of the LEO constellation inference system.

[0116] refer to Figure 6 The method of this invention exhibits excellent scalability under different numbers of satellites. When the number of satellite nodes increases from 2 to 4, the average normalization latency and timeout rate of this invention decrease significantly, approaching optimal performance. When the number of nodes continues to increase to 6 or more, performance slightly declines as multi-hop communication costs increase, but it is still better than all comparison baselines. Although Voltage also benefits from an increase in the number of nodes, its performance degrades more rapidly. ParaTra and Local-only lack scalability across multiple nodes, and their performance remains essentially unchanged or significantly worse than that of this invention. This invention achieves maximum benefit with a small number of nodes and maintains high efficiency even as the number of nodes increases.

[0117] To verify the necessity of each core module in this invention, the following two ablation schemes were further compared:

[0118] Fixed patch granularity: No granularity adaptation is performed; only patches of a fixed size are selected.

[0119] Uniform sequence distribution: PSO optimization is not used; a uniform distribution strategy is directly adopted.

[0120] refer to Figure 7 The comparison results show that the fixed patch granularity scheme lags significantly as the number of tasks increases, failing to adapt to changes in computational pressure caused by different task sizes. The uniform allocation scheme experiences early performance degradation under high task loads and cannot effectively utilize the differences in multi-satellite resources. The complete architecture of granularity adaptation + PSO sequence allocation adopted in this invention exhibits optimal performance, especially with significant advantages in high-density task scenarios.

[0121] The significant advantages of this invention are:

[0122] The first to propose a Vision Transformer distributed inference system for the LEO constellation; supports online scheduling of dynamic tasks; the ring-type key-value exchange method reduces the global self-attention communication volume from O(N) to O(N) 2The computational complexity is reduced to O(N), effectively alleviating the pressure on inter-satellite links; through an alternating optimization framework, complex joint decisions are decomposed into low-overhead sub-problems, improving real-time feasibility; the scheduling process utilizes the current link status and computing resources for dynamic updates, effectively improving task completion rate and meeting deadline requirements; when multiple satellites participate in collaborative inference, the total latency and throughput are significantly improved, demonstrating good scalability.

[0123] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0124] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations, characterized in that, Performed by the gateway satellite, including the following steps: Step S1, Task Status Update: In response to the arrival of a new inference task or the completion of an existing task, update the current set of tasks to be processed; Step S2, Alternating Optimization Decision: For each task in the task set, perform alternating optimization of patch granularity selection and cross-satellite sequence allocation ratio until the iteration termination condition is met, and output the optimal patch granularity and optimal sequence allocation ratio for each task; wherein, the patch granularity is selected from a predefined discrete set to determine the total number of tokens in the input image; the sequence allocation ratio is a continuous variable used to define the allocation ratio of task token sequences among different satellites; the patch granularity selection is achieved by traversing the predefined discrete set and evaluating the standardized hysteresis estimate, and the sequence allocation ratio is jointly optimized by an optimization algorithm under the conditions of satisfying satellite computing power constraints and inter-satellite link bandwidth constraints; Step S3, Distributed Inference Execution: Obtain the token sequence based on the input image of the task according to the optimal patch granularity, and divide the token sequence into sub-sequences according to the optimal sequence allocation ratio and distribute them to multiple satellites in the constellation; each satellite exchanges key-value pair information by passing local key vectors and value vectors hop by hop based on a preset ring communication topology to collaboratively complete global self-attention calculation, and the results are aggregated by the gateway satellite.

2. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 1, characterized in that, The alternating optimization decision in step S2 specifically includes: Step S21, Initialization: Randomly initialize the patch granularity and sequence allocation ratio for all tasks; Step S22, solving the first subproblem: Under the condition of fixing the current sequence allocation ratio, for each task, select the patch granularity that minimizes the standardized hysteresis estimate of the task from the discrete set of its corresponding patch granularity, and prune the granularities that do not meet the constraints, and update the optimal patch granularity. Step S23, solving the second subproblem: Under the condition of fixing the current optimal patch granularity, the particle swarm optimization algorithm is used to optimize the sequence allocation ratio of all tasks. Under the premise of satisfying the satellite computing power constraint and inter-satellite link bandwidth constraint, the sequence allocation ratio that minimizes the total standardized hysteresis estimate of all tasks is found, and the optimal sequence allocation ratio is updated. Step S24, Iteration judgment: Repeat steps S22 and S23 until the preset number of iterations or time budget is reached.

3. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 2, characterized in that, The standardized hysteresis estimate Tardiness i The calculation formula is: Among them, W i R represents the waiting time for task i. i D represents the estimated task runtime based on the current patch granularity and sequence allocation ratio. i This indicates the deadline for task i.

4. The ViT online scheduling and sequential parallel inference method for low-Earth orbit satellite constellations according to claim 2 or 3, characterized in that, In step S22, the method for selecting the patch granularity for each task is as follows: traverse each candidate granularity in its predefined discrete set of patch granularities, calculate the task runtime estimate when using the candidate granularity based on the current sequence allocation ratio, then calculate the standardized hysteresis estimate, and select the candidate granularity with the smallest standardized hysteresis estimate.

5. The ViT online scheduling and sequential parallel inference method for low-Earth orbit satellite constellations according to claim 2, characterized in that, In step S23, when using the particle swarm optimization algorithm for optimization, the following methods are used to ensure that the solution satisfies the constraints: For each iteration, a proportional vector is assigned to the candidate sequence. First, a simplex projection is performed to ensure that each proportional value is non-negative and the sum is 1. Subsequently, based on the candidate sequence allocation ratio vector and the current optimal patch granularity, it is verified whether the upper limit constraint of the computing power of each satellite and the upper limit constraint of the bandwidth of each inter-satellite link are met. For candidate vectors that violate constraints, constraint processing is performed by adding a penalty term to the optimization objective function; the penalty term is proportional to the degree of constraint violation.

6. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 1, characterized in that, The distributed inference execution in step S3 specifically includes: Step S31, Sequence partitioning and distribution: The gateway satellite partitions the token sequence of a task into subsequences corresponding to the number of satellites based on the optimal patch granularity and optimal sequence allocation ratio of each task, and distributes them through inter-satellite links; Step S32, Local key-value pair generation and circular exchange: Each satellite calculates the received local token subsequence to generate the corresponding local key vector and value vector; each satellite sends the local key vector and value vector to the neighboring satellites hop by hop according to the preset circular topology order, and receives the key vector and value vector from other satellites. After multiple rounds of exchange, each satellite obtains the global key vector set and value vector set. Step S33, Attention Calculation and Result Convergence: Each satellite performs self-attention calculation based on the local query vector and the global key vector set and value vector set to obtain the output of the local subsequence; each satellite returns the local output to the gateway satellite, which converges the results to obtain the final inference result of the task.

7. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 6, characterized in that, The ring topology is a one-way ring. The key vector and value vector of each satellite are transmitted along the one-way ring. After the number of satellites minus one round of hops, each satellite obtains the key vector and value vector of all satellites.

8. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 1, characterized in that, The predefined discrete set of patch granularity contains multiple granularity levels that divide the input image into different numbers of tokens, and the number of tokens is negatively correlated with the size of the patch granularity.

9. The ViT online scheduling and sequence parallel inference method for low-Earth orbit satellite constellations according to claim 1, characterized in that, The method also The adaptive steps include: when the number of satellite nodes participating in collaborative inference within the constellation changes or the inter-satellite link status changes significantly, steps S1 to S3 are automatically triggered to re-optimize scheduling and reassign inference tasks.

Images