Federal large model robust fine tuning method and system for remote sensing task heterogeneous scene

Abstract

The invention discloses a federal remote sensing large model robust fine tuning method and system for a task heterogeneous remote sensing scene, and the method comprises the steps: a server integrates a task similarity graph and an auxiliary LoRA expert pool driven by task semantics, and schedules and issues auxiliary LoRA experts matched with a client task through an expert scheduling mechanism based on the task similarity graph; on the premise that the weight of the large model is frozen, the client optimizes local LoRA expert parameters by adopting a multi-task distribution robust optimization strategy through assisting the gradient of the LoRA experts and uploads the parameters; and the server also performs task-oriented parameter aggregation on the local LoRA expert parameters uploaded by the client to update auxiliary LoRA expert parameters, and after federated learning is finished, a federated remote sensing large model composed of the original large model weight and the local LoRA expert parameters is obtained. According to the method, the problem of model conflict caused by task heterogeneity can be effectively solved, and the unification of cross-task knowledge migration and local robustness can be realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing data processing, and particularly relates to a federated large model robust fine-tuning method and system for a heterogeneous scene of a remote sensing task. BACKGROUND

[0002] With the continuous improvement of space information infrastructure, remote sensing observation capability continues to improve, and multi-source, multi-temporal and multi-scale remote sensing data are emerging on an unprecedented scale. These data play a key role in supporting smart cities, ecological assessment, disaster emergency and other national major needs. However, the collection and management of remote sensing data naturally presents a distributed pattern. Different institutions, regions and even countries operate independent observation systems, and data are difficult to share across domains due to sovereignty, security or commercial sensitivity, forming widely existing data islands. Traditional methods need to transmit raw images between ground stations, which poses privacy leakage and data security risks.

[0003] Federated learning realizes data immobility and model mobility by exchanging only model parameters, providing a feasible path for collaborative modeling under privacy protection. In recent years, research has attempted to introduce federated learning into remote sensing image interpretation tasks, protecting data privacy while preliminarily verifying the feasibility of model collaborative training. These explorations show that federated learning can alleviate the data island problem to some extent and achieve preliminary results in classification accuracy and model convergence.

[0004] However, when facing real complex remote sensing application scenarios, the existing federated learning framework still faces serious challenges: on the one hand, the downstream tasks undertaken by each institution are highly heterogeneous, and there is a lack of effective multi-task collaborative optimization mechanism, leading to conflicts in model update direction and restricting the generalization ability of remote sensing large models under heterogeneous tasks; on the other hand, remote sensing data have strong regional and temporal dependence, and local training is prone to overfitting to specific regional or seasonal features, while existing fine-tuning strategies often fail to effectively alleviate the accumulation of local bias, resulting in insufficient robustness of the global model. Therefore, how to realize robust federated fine-tuning with multi-task collaborative optimization under the premise of protecting data privacy has become a key problem to be broken through.

[0005] Current federated learning methods for remote sensing have some explorations in dealing with heterogeneity and large model challenges, but there are still obvious limitations. On the one hand, the solutions for heterogeneity (such as patent application CN120433832A discloses a low-orbit satellite internet heterogeneous federated learning method based on dynamic aggregation, and patent application CN115345316A discloses a federated learning training system and method for heterogeneous task requirements) mainly focus on data distribution offset or client system state difference, and alleviate the field offset by sharing correction data, dynamic weighted aggregation or client scheduling. However, these methods still assume that all participants perform the same or highly similar downstream tasks, and do not consider the task target heterogeneity that is prevalent in remote sensing applications, resulting in conflicting optimization objectives for different tasks when aggregating models.

[0006] On the other hand, attempts to lightweight large models (such as patent application CN120182700A discloses a dynamic weight adjustment method and system for pest monitoring based on a multi-modal remote sensing large model, and patent application CN118735006A discloses a training method, device, equipment and storage medium for federated large models) introduce multi-modal fusion or general adapter structures to reduce communication overhead, but their adaptation mechanisms are task-independent static designs that cannot dynamically adjust model behavior according to specific task semantics.

[0007] In addition, some methods (such as patent application CN119672531A discloses a federated learning-based remote sensing image processing method, system and related equipment) attempt to align multi-source data through physical corrections such as radiation and geometry, which improves feature consistency, but their optimization objectives are still limited to the data level, lack the ability to model task semantic similarity, and cannot guide knowledge transfer between tasks with similar semantics.

[0008] In summary, existing technologies face core bottlenecks such as insufficient multi-task collaborative optimization and weak model robustness in real multi-task remote sensing federated scenarios. SUMMARY

[0009] In view of the above, the purpose of the present application is to provide a federated large model robust fine-tuning method for task-heterogeneous remote sensing scenarios, aiming to address the model conflict challenges caused by task heterogeneity in federated remote sensing learning, designing a task similarity graph based on task natural language description, implementing an expert scheduling mechanism based on the task similarity graph, and achieving effective utilization of the LoRA expert pool driven by task semantics. Here, task conflict specifically refers to the case where each client performs remote sensing tasks with related semantics but different optimization objectives, and their local LoRA update directions cancel each other out under traditional federated averaging aggregation, resulting in a decline in the global model's generalization ability.

[0010] To achieve the above object, the embodiment provides a federated large model robust fine-tuning method for a task-heterogeneous remote sensing scene, and comprises the following steps. The server is integrated with a task similarity graph and a task semantic driven auxiliary LoRA expert pool, and an auxiliary LoRA expert adapted to the task of the client is dispatched and delivered through an expert scheduling mechanism based on the task similarity graph; The client optimizes the local LoRA expert parameters by using a multi-task distribution robust optimization strategy through the gradient of the auxiliary LoRA expert on the premise of freezing the original large model weight and uploads the local LoRA expert parameters; The server further updates the auxiliary LoRA expert parameters by performing parameter aggregation on the local LoRA expert parameters uploaded by the client in a task-oriented manner, and obtains a federated remote sensing large model composed of the original large model weight and the local LoRA expert parameters after the federated learning is completed.

[0011] Preferably, the task similarity is constructed in the following manner: First, natural language descriptions of all remote sensing task types are collected, a pre-trained text encoder is used to map each task description into a semantic feature vector, the cosine similarity between each two tasks is calculated based on the semantic feature vectors, and a full task similarity matrix is constructed, then, is subjected to Top- sparsification and trimming, so as to construct a sparse task semantic similarity graph, , is a node set, is an edge set, the node corresponds to a remote sensing task type, and the edge represents the semantic association strength between the task and , and the remote sensing task type includes crop classification, change detection and building extraction.

[0012] Preferably, in the task semantic driven auxiliary LoRA expert pool, each auxiliary LoRA expert corresponds to a group of low-rank matrices, and is indexed by a task label and is adapted to and belongs to a certain type of remote sensing task.

[0013] Preferably, the auxiliary LoRA expert adapted to the task of the client is dispatched and delivered through an expert scheduling mechanism based on the task similarity graph, comprising: The server determines the Top- neighbor set of the node corresponding to the remote sensing task type of the client from the task similarity graph according to the remote sensing task type of the client, and retrieves the corresponding A number of auxiliary LoRA experts are distributed to the client. These auxiliary LoRA experts participate in forward inference and gradient calculation during local training on the client, but their parameters remain frozen and are only used as knowledge references for semantically similar tasks.

[0014] Preferably, the client, while freezing the weights of the original large model, employs a multi-task distributed LoRA optimization strategy to optimize the local LoRA expert parameters by using the gradient of the auxiliary LoRA expert, including: Multiple auxiliary LoRA experts and one local LoRA expert are inserted in parallel within the original large model with a single frozen parameter. Each expert forms an independent adaptation path, and multiple prediction outputs are obtained based on local remote sensing data. A sub-LoRA bar optimization loss is independently constructed for each path, and its gradient with respect to its own LoRA expert parameters is calculated. Subsequently, the gradients of all LoRA expert parameters are treated as optimization signals in the same vector space. A unified Pareto optimal update direction is solved using a multi-gradient descent algorithm. This Pareto optimal update direction is then applied only to the parameter updates of the local LoRA expert, enabling fine-tuning guided by cross-task knowledge.

[0015] Preferably, the independent Bruker bar optimization loss for each path is based on the Wasserstein distance and is expressed as: in, Indicates the first k The prediction output of a LoRA expert, This represents the input remote sensing data and the corresponding real-world labels for the task. For local experience distribution, Therefore Centered on, with radius Wasserstein ball, For task-related loss functions, Expressing expectations, Indicates belonging to the distribution neighborhood The probability distribution, Indicates the first k The LoRA expert corresponds to the path of the Bruker optimization loss.

[0016] Preferably, the Approximated by duality: in, This is the index for remote sensing data samples, where n is the number of samples. Let the radius be the Wasserstein sphere. It is the L2 norm. To derivation.

[0017] Preferably, a unified Pareto optimal update direction is solved by a multi-gradient descent algorithm, comprising: wherein is a d-dimensional probability simplex, and the optimal solution is The Pareto optimal update direction is: The update direction Fuses the gradient information of all experts, and the direction ensures that when updated along the direction, the local task can be robustly optimized through the distributed robust optimization loss, and the knowledge of the semantically similar task implicitly guides the local update through the gradient coordination mechanism.

[0018] Preferably, the local LoRA expert parameters uploaded by the client are task-oriented for parameter aggregation update auxiliary LoRA expert parameters, comprising: The server aggregates all local LoRA expert parameters under the same task label to update the auxiliary LoRA expert parameters of the corresponding task label.

[0019] To achieve the above-mentioned purposes, the embodiments of the present application also provide a federated remote sensing large model robust fine-tuning system for task-heterogeneous remote sensing scenes, comprising a plurality of clients and a server; The server is used to integrate a task similarity graph and a task semantic driven auxiliary LoRA expert pool, schedule and issue auxiliary LoRA experts adapted to the task of the client through an expert scheduling mechanism based on the task similarity graph, and is also used to aggregate and update the auxiliary LoRA expert parameters of the local LoRA expert parameters uploaded by the client in a task-oriented manner. Under the premise of freezing the original large model weight, the client optimizes the local LoRA expert parameters by the gradient of the auxiliary LoRA expert using a multi-task distributed robust optimization strategy and uploads them, and after the federated learning is completed, a federated remote sensing large model composed of the original large model weight and the local LoRA expert parameters is obtained.

[0020] Compared with the prior art, the present application has at least the following beneficial effects: 1. The present application effectively solves the model conflict problem caused by task heterogeneity: by scheduling semantically related experts through a task similarity graph, the client can locally utilize related task knowledge in a collaborative manner, avoiding the performance degradation caused by the inconsistency of traditional federated averaging task objectives.

[0021] 2. The application realizes the unification of cross-task knowledge transfer and local robustness: a multi-LoRA expert parallel architecture is constructed on the client side, combined with distributed robust optimization to improve the robustness to distribution bias, and through a multi-task optimization mechanism, implicit knowledge transfer between semantically related tasks is realized, and the generalization ability of the model is improved under the premise of only updating local parameters. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is the flowchart of the federated remote sensing large model robust fine-tuning method for task-heterogeneous remote sensing scenes provided by the embodiment; Figure 2 is the detailed flowchart of the federated remote sensing large model robust fine-tuning method for task-heterogeneous remote sensing scenes provided by the embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0025] The inventive concept of the present application is: in order to solve the technical problems of insufficient multi-task cooperative optimization and weak model robustness in existing federated learning for remote sensing data, the present application proposes a federated large model robust fine-tuning scheme for task-heterogeneous remote sensing scenes. By constructing an expert scheduling mechanism based on task similarity graph, a task semantic driven LoRA expert pool is maintained, so that the client can cooperatively call the trainable LoRA expert corresponding to its own task and the frozen LoRA expert of several semantically similar tasks in each training round. In each training round, the client uses a multi-task distributed robust optimization strategy to realize the federated remote sensing large model cooperative architecture with task-specific adaptation, cross-task knowledge transfer and model robustness enhancement under the premise of protecting the privacy of remote sensing data.

[0026] The application provides a federated large model robust fine-tuning scheme for a task-oriented heterogeneous remote sensing scene, which is composed of a server and a client. Under the premise of freezing the original large model weight, a set of task-oriented LoRA experts are trained collaboratively through a federated learning framework, and an expert scheduling mechanism based on a task similarity graph, a task semantic driven LoRA expert pool, and a multi-task distribution robust optimization strategy on the client side are integrated to cope with the model conflict and generalization challenges brought by remote sensing multi-task heterogeneity.

[0027] Specifically, assume that the federated system contains remote sensing agencies (i.e., clients), each client owns a private remote sensing dataset and its corresponding downstream task type , wherein ={crop classification, change detection, building extraction, …}. The application divides the model adaptation knowledge into three components: (1) basic remote sensing large model parameters : a frozen pre-trained backbone network providing general representation ability, not participating in training and communication; (2) auxiliary LoRA expert pool : a set of auxiliary LoRA experts maintained by the server, each expert corresponds to a set of low-rank matrices , which is specifically adapted to a certain type of remote sensing task and belongs to a certain type of remote sensing task . The server constructs a task similarity graph based on the natural language description of the task, and allocates a subset of experts related to the task semantics of each client through an expert scheduling mechanism based on the task similarity graph; (3) local adaptation of the client: contains a trainable local LoRA expert and auxiliary LoRA experts downloaded from the server. The client optimizes locally using a multi-task distribution robust optimization strategy, which consists of two parts: through a distributionally robust optimization (DRO) loss function, the robustness of the model to local data distribution deviation is improved; a multi-task optimization mechanism based on a multiple gradient descent algorithm (MGDA) coordinates the gradient direction of the local task loss and the auxiliary task loss, avoids conflict, and solves the Pareto optimal update direction.

[0028] Instead of minimizing the single total loss explicitly, the client coordinates the gradients of multiple task losses through a multi-task optimization mechanism and only uses the resulting update direction to fine-tune the local LoRA expert, thereby achieving collaborative improvement of cross-task knowledge transfer and robust generalization ability while ensuring privacy.

[0029] In the federated remote sensing large model robust fine-tuning scheme, the server is used for task semantic graph guided LoRA expert management. In order to effectively deal with the model conflict problem caused by the heterogeneity of remote sensing tasks, the application constructs a task semantic driven LoRA expert pool on the server side, and designs a task similarity graph based expert scheduling mechanism, forming a "semantic awareness-expert scheduling-collaborative fine-tuning" architecture.

[0030] Specifically, the server first collects the natural language descriptions of all remote sensing task types , and uses a pre-trained text encoder (such as Sentence-BERT or a remote sensing field fine-tuned CLIP text encoder) to map each task description to a semantic feature vector . Based on these semantic vectors, the cosine similarity between tasks is calculated, and a full task similarity matrix is constructed. Subsequently, the is Top- sparsified and pruned, thereby constructing a sparse task semantic similarity graph . The node set is , the edge set is , the node corresponds to the remote sensing task type , and the edge represents the semantic association strength between tasks .

[0031] Based on this graph structure, the server maintains a task semantic driven auxiliary LoRA expert pool , where each expert is indexed by the task label and is specifically adapted to the corresponding remote sensing task. The implementation of all LoRA experts follows the standard low-rank decomposition form: for the weight matrix of a certain layer of the large model , its adaptive incremental representation is , where , is the rank.

[0032] At the beginning of each round of federated training, the server assigns expert resources to the client (the task of which is ) through a task similarity graph based expert scheduling mechanism: (1) The client first obtains the task Correspondingly, trainable local LoRA experts The expert will participate in local optimization and upload the results to the server for aggregation after training is complete. (2) The server determines the node corresponding to the remote sensing task type from the task similarity map based on the client's remote sensing task type. Top- Neighbors set From the LoRA expert pool Searching for the corresponding A LoRA expert The data is then distributed to the client. These auxiliary experts participate in forward inference and gradient calculation during local training on the client, but their parameters remain frozen, serving only as a knowledge reference for semantically similar tasks.

[0033] After each training round, the server receives local LoRA expert parameter updates uploaded by all clients, corresponding to their task labels, and updates the parameters for the same task. All uploaded expert parameters are aggregated (e.g., weighted average) to update the global LoRA expert pool. Through this mechanism, the server continuously maintains an expert system aligned with the semantic structure of the task and capable of cross-task knowledge evolution, providing a structured and scalable model adaptation foundation for heterogeneous remote sensing federated learning.

[0034] In the robust fine-tuning scheme of the federated remote sensing large model, the client mainly adopts a local multi-task distributed robust optimization strategy to update the parameters of the local LoRA expert. To achieve robust fine-tuning of the local task while preserving cross-task knowledge, this invention designs a multi-task distributed robust strategy on the client side. This strategy consists of two parts working together: distributed robust optimization (DRO) and multi-task optimization based on the multiple gradient descent algorithm (MGDA). It aims to balance local task performance, out-of-distribution generalization ability, and cross-task knowledge guidance.

[0035] The core of this strategy lies in the parallel insertion of multiple LoRA experts within a single frozen original large model. Each expert forms an independent adaptation path, ultimately yielding multiple predicted outputs. A sub-Bruker loss is independently constructed for each path, and its gradient with respect to its own parameters is calculated. Subsequently, these gradients are treated as optimization signals in the same vector space, and a unified Pareto optimal update direction is solved using a multi-gradient descent algorithm. Finally, this direction is applied only to the parameter updates of the LoRA expert corresponding to the local task, achieving cross-task knowledge-guided fine-tuning. The specific implementation process is as follows: (1) Parallel forward process of multiple LoRA experts Client Receive the trainable local LoRA expert sent by the server and A LoRA expert to assist with input remote sensing imagery. In the frozen large model During the forward propagation process, parallel insertions are made. Set up a LoRA adapter. Specifically, at each layer of the model, the original weights are computed simultaneously. Low-rank increments with various LoRA experts The superposition effect forms There are several independent computational paths. These paths share most of the intermediate computation results, but each maintains the independence of its LoRA adaptation, ultimately yielding... Each predicted output .in The corresponding LoRA expert can be trained locally, while the others correspond to frozen auxiliary experts.

[0036] (2) Calculation of independent blue bar optimization loss For each predicted output Calculate the optimization loss of each Bruker bar independently. The optimization loss of the Bruker bar based on the Wasserstein distance can be expressed as: in, This represents the input remote sensing data and the corresponding real-world labels for the task. For local experience distribution, Therefore Centered on, with radius Wasserstein ball, For task-related loss functions, Expressing expectations, Indicates belonging to the distribution neighborhood The probability distribution, Indicates the first k The LoRA expert corresponds to the path of the sub-LoRA bar optimization loss. In actual implementation, this... It can be approximated by dual form as follows: in This is the index for remote sensing data samples, where n is the number of samples. Let the radius be the Wasserstein sphere. It is the L2 norm. To Differentiation. This form enhances the model's robustness to input perturbations, making it particularly suitable for common imaging condition variations and annotation noise in remote sensing data.

[0037] (3) Multi-task optimization based on multi-gradient descent algorithm For each loss , backpropagation is performed respectively to obtain the gradient of the corresponding LoRA expert parameter : Since all LoRA experts adopt the same low-rank structure, the parameter vector dimensions are consistent, so the gradient can be regarded as an optimization signal in the same vector space. Subsequently, the gradients are passed through a multi-gradient descent algorithm to solve the following convex optimization problem to obtain a Pareto optimal update direction: where is an -dimensional probability simplex. Let the optimal solution be , then the Pareto optimal update direction is: The update direction combines the gradient information of all experts. This direction ensures that when updated along it, the local task can be robustly optimized through distributed robust optimization loss, and the knowledge of semantically similar tasks is implicitly guided to the local update through the gradient coordination mechanism.

[0038] (4) Local parameter update and auxiliary expert freezing Finally, only the update direction is applied to the locally trainable LoRA expert, while all auxiliary LoRA expert parameters remain frozen, achieving gradient participation coordination and strict parameter freezing.

[0039] Based on the above client-server functions, the federal training process of the present application adopts a three-stage mechanism of "task semantic driven expert distribution - local fine-tuning - expert upload aggregation", to realize robust collaborative learning in the task heterogeneous remote sensing scene. As Figure 1 shown, it specifically includes: S1, the server integrates a task similarity graph and a task semantic driven auxiliary LoRA expert pool, and schedules and distributes auxiliary LoRA experts adapted to the client task through an expert scheduling mechanism based on the task similarity graph; S2, the client optimizes the local LoRA expert parameters using the gradient of the auxiliary LoRA expert under the premise of freezing the original large model weights, and uploads them; S3, the server further updates the auxiliary LoRA expert parameters by aggregating the local LoRA expert parameters uploaded by the client in a task-oriented manner, and obtains a federal remote sensing large model composed of the original large model weights and the local LoRA expert parameters after the federal learning is completed.

[0040] AsFigure 2 As shown, the detailed process of the robust fine-tuning method for the federated remote sensing large model is as follows: (1) Initialization phase The server loads the backbone parameters of the pre-trained remote sensing large model. This data is then globally frozen and not used for subsequent training and communication. Subsequently, all natural language descriptions of the tasks are collected, and task semantic features are constructed using a text encoder. A sparse task semantic similarity map is then built based on cosine similarity. Based on this, for each task type Randomly initialize a global LoRA expert This constitutes a task semantic-driven LoRA expert pool. Each client does not need to initialize its local LoRA instance separately.

[0041] (2) Federal training process for each round At the start of each training round, the server determines the client's task tags. Execute an expert scheduling mechanism based on task similarity graphs: obtain task labels from the global LoRA expert pool. Corresponding experts; and based on the task semantic graph Search Top- Given semantically similar tasks, obtain the corresponding... A LoRA expert. The server will... Each LoRA expert is sent to its corresponding client. Upon receiving the expert, the client marks the auxiliary expert parameters as frozen but differentiable (i.e., participating in forward and gradient calculations but not updating the parameters), and performs end-to-end fine-tuning only on the local LoRA expert.

[0042] Next, the client executes a multi-task split-bar optimization strategy. Specifically, for each LoRA expert, the client independently calculates its split-bar optimization loss on its local private remote sensing dataset. Backpropagation is performed on each split-bar optimization loss to obtain the corresponding LoRA parameter gradient; this gradient is then input into a multi-gradient descent algorithm to solve for the Pareto optimal update direction; only this direction is used to update the local LoRA expert, while the parameters of other experts are frozen. After training, the client only uploads the updated local LoRA expert to the server.

[0043] (3) Server aggregation stage The server determines the client's task tag. The data is categorized into the corresponding task categories, and a weighted average aggregation is performed on all local LoRA experts uploaded by clients under the same task. This aggregation process continuously optimizes the knowledge representation capabilities of global LoRA experts, gradually integrating robust task knowledge.

[0044] The method of the present application is based on a low-rank adaptive architecture, which realizes privacy-protected multi-task robust collaborative learning under the premise of freezing the pre-training backbone of a remote sensing large model.

[0045] The method of the present application maintains multi-task expert knowledge on the server side through the construction of a task semantic driven LoRA expert pool and an expert scheduling mechanism based on a task similarity graph; and introduces a multi-task distribution robust strategy on the client side to fuse the distribution robust optimization and multi-task optimization mechanism, thereby realizing the unification of local task-specific optimization and cross-task knowledge guidance and effectively alleviating task conflicts and robustness challenges.

[0046] In summary, the method of the present application cooperatively calls trainable LoRA experts corresponding to the task itself and frozen LoRA experts of several semantically similar tasks, and combines a multi-task distribution robust optimization strategy to realize the unification of cross-task knowledge transfer and model robustness under the premise of only updating the trainable LoRA expert parameters of the corresponding task, thereby significantly improving the overall performance and adaptability of the federated remote sensing large model in a heterogeneous multi-task scenario.

[0047] The above detailed description of the specific embodiments of the present application has described the technical solutions and beneficial effects of the present application, and it should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes, characterized in that, The method comprises the following steps: The server integrates a task similarity graph and a task semantic driven auxiliary LoRA expert pool, schedules and issues an auxiliary LoRA expert suitable for a task of the client through a task similarity graph based expert scheduling mechanism; The client optimizes local LoRA expert parameters through the gradient of the auxiliary LoRA expert under the premise of freezing original large model weights, and uploads the local LoRA expert parameters; The server also aggregates and updates the auxiliary LoRA expert parameters based on the task of the local LoRA expert parameters uploaded by the client, and obtains a federal remote sensing large model composed of original large model weights and local LoRA expert parameters after the federal learning is completed.

2. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 1, wherein, The task similarity is constructed in the following manner: Firstly, collect all the natural language descriptions of remote sensing task types , map each task description to a semantic feature vector using a pre-trained text encoder, compute the cosine similarity between each pair of tasks based on these semantic feature vectors, construct a full task similarity matrix , then perform Top- pruning to obtain a sparse task semantic similarity graph , , is a set of nodes, is a set of edges, node corresponds to a remote sensing task type , edge represents the semantic association strength between tasks and . Remote sensing task types include crop classification, change detection, and building extraction.

3. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 1, wherein, In the task semantic driven auxiliary LoRA expert pool, each auxiliary LoRA expert corresponds to a group of low-rank matrices, and is indexed by a task label and belongs to a certain type of remote sensing task.

4. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 1, wherein, The auxiliary LoRA expert suitable for the task of the client is scheduled and issued through the task similarity graph based expert scheduling mechanism, which comprises: The server determines a remote sensing task type corresponding node from a task similarity graph according to a remote sensing task type of the client Top- neighbor set , and retrieves corresponding auxiliary LoRA experts from an auxiliary LoRA expert pool and delivers to the client, these auxiliary LoRA experts participate in forward inference and gradient calculation in local training of the client, but their parameters remain frozen, only as a knowledge reference for semantically similar tasks.

5. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 1, wherein, The client optimizes the local LoRA expert parameters through the gradient of the auxiliary LoRA expert under the premise of freezing the original large model weights, which comprises: A plurality of auxiliary LoRA experts and a local LoRA expert are inserted into a single frozen parameter original large model in parallel, each expert forms an independent adaptation path, and a plurality of prediction outputs are obtained based on local remote sensing data, a distribution robust optimization loss is independently constructed for each path, and the gradient of the LoRA expert parameters for each path is calculated; Subsequently, the gradients of all LoRA expert parameters are regarded as optimization signals in the same vector space, a unified Pareto optimal update direction is solved through a multi-gradient descent algorithm, and the Pareto optimal update direction is only applied to the parameter update of the local LoRA expert, thereby realizing fine tuning under the guidance of cross-task knowledge.

6. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 5, characterized in that, The distribution robust optimization loss independently constructed for each path is based on the Wasserstein distance and is expressed as: where, denotes the prediction output of the k th LoRA expert, denotes the input remote sensing data and the corresponding task ground truth label, is the local empirical distribution, is the Wasserstein ball centered at with radius is the Wasserstein ball centered at is the task-dependent loss function, denotes the expectation, denotes the probability distribution belonging to the neighborhood of the distribution, denotes the distribution robust optimization loss of the k th LoRA expert corresponding path.

7. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 6, characterized in that, The By duality, we have: wherein, is the Wasserstein ball radius, is the Wasserstein ball radius, is the L2 norm, is the Wasserstein ball radius, is the Wasserstein ball radius.

8. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 5, characterized in that, The unified Pareto optimal update direction is solved through the multi-gradient descent algorithm, which comprises: where is - the probability simplex, let the optimal solution be then the Pareto optimal update direction is: , the update direction Fuses the gradient information of all experts, this direction ensures that when updating along it, the local task can be robustly optimized through the distributed robust optimization loss, and the knowledge of semantically similar tasks implicitly guides the local update through the gradient coordination mechanism.

9. The federated remote sensing large model robust fine-tuning method for task-oriented heterogeneous remote sensing scenes according to claim 1, wherein, The local LoRA expert parameters uploaded by the client are aggregated and updated to update the auxiliary LoRA expert parameters, which comprises: The server aggregates all local LoRA expert parameters under the same task label to update the auxiliary LoRA expert parameters of the corresponding task label based on the task label of the local LoRA expert.

10. A federated remote sensing large model robust fine-tuning system for task-oriented heterogeneous remote sensing scenes, characterized in that, The method comprises the following steps: The server integrates a task similarity graph and a task semantic driven auxiliary LoRA expert pool, schedules and issues an auxiliary LoRA expert suitable for a task of the client through a task similarity graph based expert scheduling mechanism, and aggregates and updates the auxiliary LoRA expert parameters based on the task of the local LoRA expert parameters uploaded by the client. The client adopts a multi-task distribution robust optimization strategy to optimize the local LoRA expert parameters by the gradient of the auxiliary LoRA expert under the premise of freezing the original large model weight and uploads, and obtains a federal remote sensing large model composed of the original large model weight and the local LoRA expert parameter after the federal learning is finished.

Images