A clustering fine-tuning and parameter transmission method of hybrid expert model under satellite-ground network

Abstract

This invention discloses a clustered fine-tuning and parameter transmission method for a hybrid expert model under a satellite-to-ground network, comprising the following steps: Step S1: Estimate the correlation between each device cluster and the experts in the hybrid expert model to obtain the expert correlation probability; Step S2: Perform threshold truncation based on the expert correlation probability and calculate the expert allocation probability; Step S3: Perform federated updates of experts and gating during the connection phase; Step S4: Perform asynchronous local expert training based on the admission results during the disconnection phase; Step S5: Repeat the connection and disconnection phase training within the orbital cycle until a preset stopping condition is reached, and output the fine-tuned global gating parameters and all expert parameters; Step S6: Encapsulate the clustered fine-tuned parameters and transmit them between satellite and ground. This invention enables distributed large model fine-tuning under intermittent satellite-to-ground connection conditions, reduces the resource consumption of unassigned experts and shared parts in local training, and reduces terminal-side memory pressure and uplink communication overhead.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of satellite communication, federated learning, and large model fine-tuning, specifically to a method for clustered fine-tuning and parameter transmission of a hybrid expert model in a satellite-ground network. Background Technology

[0002] As the scale of large model parameters continues to expand, centralized training and fine-tuning increasingly rely on high-performance data centers. In reality, a large amount of high-value data is scattered across ground terminals, edge nodes, and industry private networks. Directly aggregating raw data faces challenges related to privacy, bandwidth, compliance, and latency. Federated learning can support distributed training by exchanging only model updates rather than raw data, thus becoming an important direction for collaborative edge training of large models.

[0003] However, traditional federated learning largely assumes that servers and terminals can periodically and stably perform model broadcasting, local training, parameter uploading, and aggregation. In satellite-to-ground communication scenarios, especially those where a single low-Earth orbit satellite temporarily covers several ground equipment clusters, the relative motion between the satellite and the ground naturally leads to intermittent connections: a ground cluster maintains a high-quality link with the satellite only within a limited connection window, remaining disconnected the rest of the time. This makes it difficult for traditional federated learning schemes, which require strict synchronization, to fully utilize local computing power during periods of disconnection.

[0004] On the other hand, the MoE model, by activating only a small number of experts in each forward propagation, makes the characteristic of "some parameters being frequently activated while others participate less in the calculation of the current sample" more pronounced. This characteristic provides a natural entry point for clustering and expert-based federated fine-tuning: if the data of a certain ground cluster is mainly associated with a few experts, then it is not necessary to load and train the complete MoE model on all terminals. Instead, local updates can be performed only around experts highly correlated with the data of that cluster, and the corresponding local updates can be uploaded to the satellite for aggregation when the connection window arrives.

[0005] Therefore, in satellite-to-ground communication scenarios, there are problems such as training interruptions caused by intermittent connections, difficulty in loading the complete model on the terminal side, and excessive pressure on uplink parameter uploading. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a clustering fine-tuning and parameter transmission method for hybrid expert models under satellite-ground network conditions. This method can realize distributed large model fine-tuning under intermittent satellite-ground connection conditions, reduce the resource occupation of unassigned experts and shared parts in local training, and reduce terminal-side memory pressure and uplink communication overhead.

[0007] The objective of this invention is achieved through the following technical solution: a method for cluster fine-tuning and parameter transmission of a MoE model under a satellite-to-ground network, comprising the following steps:

[0008] Step S1: Divide the ground equipment into multiple equipment clusters according to geographical location and data distribution, construct the MoE model, and estimate the correlation between each equipment cluster and the experts in the hybrid expert model to obtain the expert correlation probability;

[0009] Step S2: Perform threshold truncation based on the relevant expert probabilities and calculate the expert assignment probabilities;

[0010] Step S3: Perform federated updates for experts and gates during the connection phase;

[0011] Step S4: Perform asynchronous local expert training based on the admission results during the disconnection phase;

[0012] Step S5: Repeat the connection and disconnection training within the orbital cycle until the preset stopping condition is reached, and output the fine-tuned global gating parameters and all expert parameters;

[0013] Step S6: Parameter encapsulation and satellite-to-ground transmission after cluster fine-tuning: After completing the cluster fine-tuning in steps S1 to S5 and obtaining the expert update, gated update or local update branch corresponding to the device cluster, the device cluster encapsulates the parameters to be uploaded into a lightweight parameter transmission packet and completes the uplink transmission within the satellite-to-ground connection window.

[0014] The beneficial effects of this invention are:

[0015] 1. By using a single satellite as a model aggregation and coordination node, this invention enables distributed large model fine-tuning under intermittent satellite-to-ground connectivity conditions, avoiding reliance solely on terrestrial cellular network coverage.

[0016] 2. By using expert-driven model splitting, this invention enables each device cluster to train and upload only experts that are highly relevant to its data, reducing the resource consumption of unassigned experts and shared parts in local training, and reducing terminal-side memory pressure and uplink communication overhead.

[0017] 3. By continuing the local training of the assigned experts during the disconnection phase, this invention can make full use of the local computing power that was originally idle due to the disconnection, thereby improving training efficiency. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention;

[0019] Figure 2 This is a schematic diagram of fine-tuning the large model;

[0020] Figure 3 This is a schematic diagram of the instantaneous coverage area of ​​a low-Earth orbit satellite.

[0021] Figure 4This is a schematic diagram of asynchronous federated learning across clusters.

[0022] Figure 5 This diagram illustrates the variation of test accuracy of EMS-FL and the benchmark scheme with the number of orbital cycles. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0024] This invention uses a satellite (in this application, a low-Earth orbit satellite) as a federated coordination server, dividing ground equipment into multiple equipment clusters based on geographical location and data distribution. It utilizes a trial dataset to statistically analyze the correlation between each expert and the data of each equipment cluster, forming expert correlation probabilities, truncation probabilities, and allocation probabilities. Based on this, it performs non-overlapping cluster allocation of experts in the MoE model, ensuring that each equipment cluster focuses on training only the set of experts highly correlated with its data. Furthermore, it designs an asynchronous local training and connection window aggregation mechanism based on satellite connection / disconnection phases, continuously training local experts during disconnection and uploading aggregated experts and gating updates during connection. Simultaneously, this invention further proposes an expert update admission and controlled merging mechanism based on connection windows and expert stability. During the connection phase, the satellite does not necessarily directly merge expert updates uploaded by equipment clusters into the global expert pool. Instead, it comprehensively considers the upload completion feasibility within the connection window, the consistency of expert update direction, the degree of expert parameter drift, and performance changes on the trial dataset to determine the admission of expert updates. For expert updates that meet the criteria, the satellite performs controlled merging according to preset weights; for expert updates that do not meet the criteria, the satellite performs delayed merging, pruned merging, rejected merging, or retains them as local update branches. This reduces the risks of upload failures, expert conflicts, and destructive aggregation. Regarding parameter transmission, the device cluster does not upload complete MoE model parameters. Instead, it encapsulates the allocated expert LoRA adaptation parameters, gated adaptation parameters, and necessary version / verification information obtained from cluster fine-tuning into parameter transmission packets. These packets are then uploaded, verified, admitted, merged, and distributed within the satellite-to-ground connection window according to expert number, model version, and priority. This allows the parameters after "cluster fine-tuning" to be transmitted to the satellite-to-ground in a lightweight manner. Specifically:

[0025] like Figure 1 As shown, a clustering fine-tuning and parameter transfer method for a hybrid expert model under a satellite-to-ground network includes the following steps:

[0026] Step S1: As Figure 2 As shown, ground equipment is divided into multiple equipment clusters according to geographical location and data distribution, a hybrid expert model is constructed, and the correlation between each equipment cluster and the experts in the hybrid expert model is estimated to obtain the expert correlation probability.

[0027] Using satellites as federal coordination servers, ground equipment is divided into multiple equipment clusters based on geographical location and data distribution;

[0028] A complete MoE model is constructed at the satellite. The MoE model refers to a hybrid expert model, comprising multiple MoE layers. Each MoE layer contains multiple expert networks and a gating network. The expert networks are simply referred to as experts. A low-rank adaptation matrix is ​​inserted into each expert. ,

[0029] For each device cluster Building a trial dataset The data is then sequentially input into the current MoE model, and the trial samples in the statistical trial dataset are routed to experts in each MoE layer. The number of times this occurs yields the relevant probabilities from expert analysis:

[0030]

[0031] The higher the probability of expert relevance, the stronger the expert's expertise. With device cluster The more relevant the data, the better. This indicates that the trial dataset is in the entire Each MoE layer and all Each sample was routed to an expert. The total number of times, m=1,2,…,M, where M represents the total number of experts.

[0032] Step S2: Perform threshold truncation based on the relevant expert probabilities and calculate the expert assignment probabilities;

[0033] For each Perform threshold truncation; values ​​less than the threshold The probability of each expert is set to zero, retaining only highly relevant experts. The remaining probabilities are then normalized to obtain the expert assignment probabilities. Subsequently, each expert is assigned to only one device cluster, with non-overlapping constraints applied so that different device clusters are responsible for different expert sets, resulting in the expert groups corresponding to each device cluster. .

[0034] , when Otherwise, it is 0.

[0035]

[0036] Through the above processing, experts who are highly relevant to local data can be prioritized and assigned to the corresponding device clusters, while avoiding multiple device clusters from repeatedly undertaking the main training tasks of the same experts.

[0037] All devices in each device cluster load the expert group corresponding to the current cluster, as well as the global gating network parameters.

[0038] like Figure 3 The diagram illustrates the instantaneous coverage area of ​​a satellite. When a satellite provides coverage, the covered devices / device clusters are connected to the satellite; other devices / device clusters are disconnected from the satellite. Cross-cluster asynchronous federated learning is illustrated below. Figure 4 As shown, specifically:

[0039] Step S3: Perform federated updates for experts and gates during the connection phase;

[0040] The currently connected device cluster first uploads its locally trained expert updates within the connection window to the satellite. These expert updates include the expert update results of each device in the cluster. Upon receiving the expert update, the satellite does not necessarily perform global aggregation directly. Instead, it first uses an expert update admission and controlled merging mechanism. If the expert update meets the admission criteria, the satellite performs a merge update on the corresponding expert within the cluster and sends the updated expert parameters and the current global gating parameters to the cluster. If the expert update does not meet the admission criteria, the satellite performs one of the following processing steps: postpone merging, prune and merge, refuse to merge, or retain it as a local update branch. Subsequently, the currently connected device cluster uploads its locally trained gating parameter updates within the connection window to the satellite. These updates include the local gating parameters of each device in the cluster. The satellite aggregates these to obtain new global gating parameters and sends them to each device in the cluster. After the expert update passes the admission criteria, the satellite can perform expert aggregation and gating aggregation as follows:

[0041]

[0042]

[0043] in, Represents device cluster In training rounds The aggregated results corresponding to the expert panel, Indicates training round t device cluster The expert update results for the j-th device are as follows: This represents the global update result after aggregating the gated parameters in training round t+1. The set represents the local gating parameter update result of the j-th device in training round t+1. Represents device cluster The set of devices in the set, where J represents the device cluster. The number of devices in the system.

[0044] The expert update admission and controlled merging based on connection windows and expert stability proceeds as follows: During the connection phase, after the device cluster uploads its local expert update to the satellite, the satellite does not necessarily merge the update directly into the global expert model. Instead, it first performs an expert update admission judgment. This judgment determines whether a particular expert update is suitable for merging into the global model within the current orbital period.

[0045] For device clusters Uploaded by experts In the update, the satellite calculates the upload feasibility index based on the estimated upload capacity of the next connection window and the size of the expert update package: .in, Represents device cluster The estimated amount of data that can be uploaded within the connection window. Represents device cluster Uploaded by experts The size of the update package. When When this is the case, it indicates that the expert update is expected to be uploaded within the next connection window. If the expert update fails to upload completely within the connection window, the satellite will temporarily suspend the global merging of the expert update and notify the device cluster to retain the expert update as a local update branch.

[0046] For expert updates that have been fully uploaded, the satellite further determines whether the update meets the following conditions:

[0047] 1. The expert update is basically consistent with the direction of the most recent global expert update on the satellite side; in the embodiments of this application, the cosine similarity between the expert update and the most recent global expert update on the satellite side is calculated. If the cosine similarity is greater than a set threshold, the update direction is considered to be basically consistent.

[0048] 2. The change in the expert's updated parameters relative to the current global expert parameters does not exceed a preset upper limit;

[0049] 3. After temporarily merging the experts, the model's loss on the trial dataset does not increase, or the increase in loss does not exceed the preset tolerance threshold.

[0050] When the above conditions are met, the satellite will treat the expert update as a valid update and merge it into the global experts according to the preset merging weight. During merging, if the size of the expert update exceeds the preset trust range, the update will be scaled or pruned before merging. .in, Indicates the current global expert parameters. Represents device cluster Uploaded by experts Update Indicates the combined weight. This indicates that shadow expert updates will be limited to a preset range. When the above conditions are not met, the satellite will not directly merge the expert update, but will perform one of the following actions: if the update direction differs significantly from the global expert update direction, it will be postponed until the next connection window for re-evaluation; if the update range exceeds the preset range but still has a positive training effect, it will be pruned and merged; if the update causes a significant increase in the loss of the trial dataset, it will be rejected from merging.

[0051] Step S4: Perform asynchronous local expert training based on the admission results during the disconnection phase;

[0052] The execution entity is the non-connected device cluster. For device clusters currently not connected to the satellite, they continue local training based on the expert update processing results fed back by the satellite. For expert updates that have been admitted and merged, the device cluster continues training starting from the updated expert parameters; for expert updates that have been temporarily suspended from merging or retained as local update branches, the device cluster can continue to accumulate the update locally and upload it again in the next connection window for the satellite to judge; for expert updates that have been rejected from merging, the device cluster stops the global merging path of the update and rolls back to the last stable expert parameters or restarts local training.

[0053] During the disconnection phase, the device clusters continue to use local data to conduct continuous local training on the assigned expert groups, while the gating parameters and the remaining unassigned model parts remain frozen. In this way, within a complete orbital cycle, each device cluster can complete the necessary synchronization during its own connection phase and continue to utilize local computing power during the remaining disconnection time, as shown in Figure 5.

[0054] Repeatedly execute the connection and disconnection phases within the orbital cycle until the preset stopping condition is met, and output the fine-tuned global gating parameters and all expert parameters.

[0055] In the embodiments of this application, the scheme is referred to as standard EMS-FL, or simply EMS-FL. If it is necessary to further reduce the model loading on the terminal side, an enhanced EMS-FL can also be used in the embodiments of this application. Its execution body consists of a device cluster local training module and a satellite-side model management module. The enhanced EMS-FL still uses the expert allocation results obtained in steps S1 and S2 to perform training on the device cluster. Only retain the expert groups that have already been assigned. The corresponding local trainable experts are used, and the local gating is masked to ensure that the gating only extends to the expert group. The system routes data to experts within the cluster, while gating outputs for unassigned experts are set to zero or masked. Thus, during the connection and disconnection phases of this device cluster, local training revolves solely around the expert group. Unassigned experts do not need to participate in forward routing and gradient updates, nor do they need to be fully loaded locally. After the expert group has been trained for several orbital cycles, the unmasked global gating is restored, and the gating is further fine-tuned or synchronously updated using standard EMS-FL. Thus, the enhanced EMS-FL can further reduce the model loading and local training memory overhead on the terminal side, and improve the activation probability and training sufficiency of assigned experts.

[0056] Step S5: Repeat the connection and disconnection training within the orbital cycle until the preset stopping condition is reached, and output the fine-tuned global gating parameters and all expert parameters;

[0057] Step S6: Parameter encapsulation and satellite-to-ground transmission after cluster fine-tuning: After completing the cluster fine-tuning in steps S1 to S5 and obtaining the expert update, gated update or local update branch corresponding to the device cluster, the device cluster encapsulates the parameters to be uploaded into a lightweight parameter transmission packet and completes the uplink transmission within the satellite-to-ground connection window.

[0058] After completing the cluster fine-tuning in steps S1 to S5 and obtaining the expert update, gated update, or local update branch corresponding to the device cluster, the device cluster encapsulates the parameters to be uploaded into a lightweight parameter transmission packet and completes the uplink transmission within the satellite-to-ground connection window.

[0059] To reduce transmission volume, the device cluster preferably uses LoRA and other parameter-efficient fine-tuning methods for transmission: freezing the basic expert weights and uploading only the low-rank adaptation matrix of the assigned experts. , or its increment For gated networks, only the gated LoRA adaptation parameters or gated increments are uploaded, not the complete expert weights and complete gated weights. The parameter transmission packet must include at least the device cluster number, expert number, MoE layer number, orbital period number, global model version number, and LoRA rank. The data includes quantization bit width, assigned expert mask, LoRA matrix or incremental parameters, gating adaptation parameters or gating increment, data length, local loss change information required for admission judgment, and checksum.

[0060] During the uplink transmission phase, the device cluster estimates the upload capacity based on the connection window. and parameter transmission packet size The transmission order is arranged, prioritizing the upload of parameter packets with high expert relevance, significant training gains, and expected complete transmission. If the current connection window is insufficient to transmit all parameter packets, the untransmitted portions are retained in the local update branch and transmitted in the next connection window. If a single parameter packet cannot be transmitted completely at once, it is divided into parameter fragments with fragment numbers and checksums. After satellite reception, these fragments are cached according to their fragment numbers and reassembled. The parameter transmission packet size can be expressed as: ,in, Represents device cluster The assigned expert group, Indicates the quantization bit width. Denotes the rank of LoRA. and They represent experts respectively. The input and output dimensions of the weight matrix to be adapted. Indicates the gating adaptation parameters or the gating increment size. This indicates the size of metadata such as version number, expert number, mask, fragment sequence number, and checksum. Since LoRA only transmits... A single adaptation parameter, not the complete weights. Therefore, in much smaller and This can significantly reduce the amount of parameter transmission.

[0061] After receiving parameter transmission packets or parameter fragments, the satellite first performs integrity verification and version consistency checks. If the verification passes, the parameters are written to the corresponding expert cache according to the device cluster number, expert number, and model version, and the expert update admission and controlled merging mechanism is input. Parameters that pass the admission judgment are used to update global expert or gated parameters; parameters that fail the admission judgment are processed by the satellite, such as being postponed, pruned, rejected, or retained as a local update branch.

[0062] During the downlink transmission phase, the satellite only sends the merged expert LoRA adaptation parameters or expert increments, gated adaptation parameters or gated increments, expert version numbers, expert masks, and merged weights required for subsequent training to the relevant device clusters, without broadcasting the complete MoE model. Upon receiving the data, the device clusters combine the sent adaptation parameters with the locally frozen basic expert weights to obtain locally usable expert parameters. This serves as the starting point for the next round of training in either the connection or disconnection phase. Thus, the parameters obtained through cluster fine-tuning can complete the closed-loop transmission process of "device cluster upload—satellite verification and merging—satellite distribution."

[0063] In the embodiments of this application, a convergence analysis is performed on the training scheme. Under the premise that the global loss function satisfies the smoothness condition, the expert gradient and the gated gradient satisfy the bounded condition, and the local stochastic gradient error satisfies the bounded condition, the orbital periodic-scale convergence performance of the standard EMS-FL scheme and the geostationary baseline scheme is analyzed.

[0064] For the standard EMS-FL, after training for T orbital cycles, its average gradient norm satisfies the following upper bound:

[0065]

[0066] For the synchronous baseline scheme, after training for T orbital cycles, its average gradient norm satisfies the following upper bound:

[0067]

[0068] in, This represents the amplification factor of the density of expert-related data in the local device cluster relative to the global density. Indicates the number of device clusters. Indicates the number of orbital periods. , Let represent the bounded constants of the expert gradient and the gated gradient, respectively. , These represent the upper bounds of the variances of the expert gradient and the gated gradient errors, respectively. This represents the additional variance term introduced by data heterogeneity in the synchronous baseline.

[0069] From the above two equations, it can be seen that the main convergence term of the standard EMS-FL is: The main convergence term of the synchronous baseline is Therefore, when At that time, EMS-FL exhibits a faster theoretical convergence trend; when data heterogeneity increases and the correspondence between experts and device clusters becomes clearer, For larger sizes, the advantages of EMS-FL become more apparent.

[0070] The convergence analysis results can be used to preset or adjust the expert assignment threshold, training step size, expert reallocation cycle, expert update admission threshold, merging weight, and pruning range in controlled merging. The expert update admission and controlled merging mechanisms, as stability enhancement mechanisms in the connection window aggregation phase, do not change the main training process of EMS-FL; therefore, the above convergence analysis can still be used to describe the main convergence trend of standard EMS-FL.

[0071] In summary, by using a single low-Earth orbit satellite as a model aggregation and coordination node, this invention enables distributed large-scale model fine-tuning under intermittent satellite-to-ground connectivity, avoiding reliance solely on terrestrial cellular network coverage. Through expert-driven model splitting, this invention allows each device cluster to train and upload only experts highly relevant to its data, reducing resource consumption by unassigned experts and shared components during local training, thus lowering terminal-side memory pressure and uplink communication overhead. By continuing local training of assigned experts during disconnection phases, this invention fully utilizes local computing power idled due to disconnection, improving training efficiency. EMS-FL requires approximately 400 training rounds to achieve 70% test accuracy, while synchronous baseline requires over 1600 rounds, reducing training time by approximately 75%. Figure 5 As shown.

[0072] In the experimental setup using LoRA (r=4), the uplink overhead of EMS-FL was 14.16 MB, while the synchronization baseline was 43.25 MB; the peak GPU memory usage of EMS-FL was approximately 23.23 GB, while the synchronization baseline was approximately 46.00 GB, indicating that the present invention has significant advantages in lightweight training and parameter uploading, as shown in Table 1.

[0073]

[0074] Table 1: Communication overhead and peak GPU memory usage when using LoRA (r = 4)

[0075] This invention employs an expert update admission and controlled merging mechanism based on connection windows and expert stability. This mechanism prevents expert updates uploaded from device clusters from being directly merged into the global model. For expert updates that cannot be completed, have inconsistent update directions, exhibit excessive expert drift, or suffer from degraded trial dataset performance, the satellite can perform delayed merging, pruned merging, rejection of merging, or retention as a local update branch. This reduces the risks of upload failures, expert conflicts, and destructive aggregation, improving the stability of MoE federated fine-tuning under intermittent satellite-to-ground connections. Through LoRA-adapted parameter packets and a satellite-to-ground transmission process based on expert fragmentation, this invention only uploads low-rank incremental parameters, gated adaptation parameters, and necessary metadata of assigned experts during parameter transmission, without uploading the complete MoE model or parameters of unassigned experts. Simultaneously, it prioritizes parameter packets according to connection window capacity, performs fragmented uploads, and resumes interrupted transmissions, further reducing uplink and downlink transmission volume on the satellite-to-ground link and improving the parameter transmission success rate within a limited connection window.

[0076] The foregoing description illustrates and describes a preferred embodiment of the present invention. However, as previously stated, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for clustering fine-tuning and parameter transmission of hybrid expert model under satellite-to-ground network, characterized in that: Includes the following steps: Step S1: Divide the ground equipment into multiple equipment clusters according to geographical location and data distribution, construct the MoE model, estimate the correlation between each equipment cluster and the experts in the MoE model, and obtain the expert correlation probability; Step S2: Perform threshold truncation based on the relevant expert probabilities and calculate the expert assignment probabilities; Step S3: Perform federated updates for experts and gates during the connection phase; Step S4: Perform asynchronous local expert training based on the admission results during the disconnection phase; Step S5: Repeat the connection and disconnection training within the orbital cycle until the preset stopping condition is reached, and output the fine-tuned global gating parameters and all expert parameters; Step S6: Parameter encapsulation and satellite-to-ground transmission after cluster fine-tuning: After completing the cluster fine-tuning in steps S1 to S5 and obtaining the expert update, gated update or local update branch corresponding to the device cluster, the device cluster encapsulates the parameters to be uploaded into a lightweight parameter transmission packet and completes the uplink transmission within the satellite-to-ground connection window. 2.The clustering fine-tuning and parameter transmission method of hybrid expert model under satellite-to-ground network according to claim 1, characterized in that: Step S1 includes: Using satellites as federal coordination servers, ground equipment is divided into multiple equipment clusters based on geographical location and data distribution; A complete MoE model is constructed at the satellite, consisting of multiple MoE layers. Each MoE layer contains multiple experts and a gating network. A low-rank adaptation matrix is ​​inserted into each expert. , ; For each device cluster Constructing the probe dataset , and inputting it into the current MoE model in turn, counting the number of times that each probe sample in the probe dataset is routed to an expert in each MoE layer, to obtain the expert-related probability: ​ ； The higher the probability of expert relevance, the stronger the expert's expertise. With device cluster The more relevant the data, the better. This indicates that the trial dataset is in the entire Each MoE layer and all Each sample was routed to an expert. The total number of times, m=1,2,…,M, where M represents the total number of experts. 3.The method of claim 2, wherein: Step S2 includes: For each The threshold cut-off is performed, probabilities less than the threshold are directly set to zero, i.e.: When , the value is unchanged, otherwise let be 0; Again After normalization, the expert allocation probability is obtained: ； Subsequently, for each expert, a device cluster is assigned according to the expert assignment probability, and a non-overlapping constraint is imposed so that different device clusters are responsible for different expert sets, to obtain the expert groups corresponding to each device cluster ; All devices in each device cluster load the expert group corresponding to the current cluster, as well as the global gating network parameters.

4. The clustering fine-tuning and parameter transmission method of hybrid expert model under satellite-to-ground network according to claim 1, characterized in that: Step S3 includes: S301. The currently connected device cluster first uploads its locally trained expert updates within the connection window to the satellite, the expert updates including the expert update results of each device in the currently connected device cluster; S302. After receiving the expert update, the satellite first inputs the expert update into the expert update admission and controlled merging mechanism: If the expert update meets the admission criteria, the satellite will perform a merge update on the corresponding expert for that cluster and send the updated expert parameters and the current global gating parameters to the current connected device cluster. If the expert update does not meet the admission criteria, the satellite will perform one of the following actions on the expert update: postpone merging, merge after trimming, refuse to merge, or retain it as a local update branch; S303. The currently connected device cluster uploads its trained local gating parameter updates within the connection window to the satellite, including the local gating parameter update results of each device in the currently connected device cluster. The satellite aggregates these to obtain new global gating parameters, which are then distributed to each device in the currently connected device cluster.

5. The clustering fine-tuning and parameter transmission method of hybrid expert model under satellite-to-ground network according to claim 4, characterized in that: The expert update access and controlled merging mechanism includes: During the connection phase, after the device cluster uploads local expert updates to the satellite, the satellite first performs an expert update admission judgment. This judgment is used to determine whether a particular expert update is suitable for incorporation into the global model within the current orbital period. A1. For device cluster Uploaded expert The satellite calculates the upload feasibility index according to the estimated upload capacity of the next connection window and the size of the expert update package : ； wherein, representing a cluster of devices estimated uploadable data amount within the connection window, representing a cluster of devices uploading experts size of the update package; A2, when the expert update is expected to be completed within the next connection window; when the expert update is at risk of failing to be uploaded completely within the connection window, the satellite postpones the global merge of the expert update and notifies the device cluster to keep the expert update as a local update branch. A3. For expert updates that have been fully uploaded, the satellite further determines whether the update meets the following conditions: (1) The expert update is basically consistent with the direction of the most recent global expert update on the satellite side: calculate the cosine similarity between the expert update and the most recent global expert update on the satellite side. If the cosine similarity is greater than the set threshold, the update direction is considered to be basically consistent. (2) The change in the expert update relative to the current global expert parameters does not exceed the preset upper limit; (3) After the expert is temporarily merged, the loss of the model on the trial dataset does not increase, or the increase in loss does not exceed the preset tolerance threshold. When conditions (1) to (3) above are met, the satellite will treat the expert update as a valid update and merge it into the global experts according to the preset merging weight; during merging, if the magnitude of the expert update exceeds the preset trust range, the update will be scaled or pruned first, and then the merging will be performed: ； wherein, denotes the current global expert parameter, denotes the device cluster uploaded expert update, denotes the merging weight, denotes limiting the shadow expert update within a pre-set range of magnitude; When the above conditions (1) to (3) are not met, the satellite does not directly merge the expert update, but performs one of the following processes: If the update direction differs significantly from the global expert update direction, it will be postponed until the next connection window for re-evaluation; If the update magnitude exceeds the preset range but still has a positive training effect, then the cropped and merged versions will be used. If the update results in a significant increase in the loss of the trial dataset, then the merge should be rejected.

6. The clustering fine-tuning and parameter transmission method of hybrid expert models under a satellite-to-ground network according to claim 5, characterized in that: Once the experts have updated their admission criteria, the satellite performs expert aggregation as follows: ； In step S303, the satellite performs gated aggregation in the following manner: ； in, Represents device cluster In training rounds The aggregated results corresponding to the expert panel Indicates training round t device cluster The expert update results for the j-th device are as follows: This represents the global update result after aggregating the gated parameters in training round t+1. The set represents the local gating parameter update result of the j-th device in training round t+1. Represents device cluster The set of devices in the set, J represents the device cluster. The number of devices in the system.

7. The method of claim 1, wherein the method further comprises: Step S4 includes: S401. The main body executing the disconnection phase is the cluster of devices that are not currently connected to the satellite: For the cluster of devices that are not currently connected to the satellite, they continue to perform local training based on the expert update processing results fed back by the satellite in step S3. For expert updates that have been admitted and merged, the device cluster continues training starting with the updated expert parameters. For expert updates that are temporarily suspended from merging or retained as local update branches, the device cluster can continue to accumulate the update locally and upload it again in the next connection window for satellite judgment. For expert updates that are rejected for merging, the device cluster stops the global merging path for that update and rolls back to the last stable expert parameters or restarts local training. S402. During the disconnection phase, the device cluster continues to use local data to conduct continuous local training on the assigned expert group, while the gating parameters and the remaining unassigned model parts remain frozen; within a complete orbital cycle, each device cluster completes the necessary synchronization during its own connection phase and continues to utilize local computing power during the remaining disconnection time; a complete orbital cycle refers to the cycle in which the satellite orbits all device clusters once.

8. The clustering fine-tuning and parameter transmission method of hybrid expert models under a satellite-to-ground network according to claim 1, characterized in that: Step S6 includes: S601. The device cluster uses LoRA parameter high-efficiency fine-tuning for transmission: Freeze the basic expert weights and only upload the low-rank adaptation matrix of the assigned experts. , or its increment For gated networks, only the gated LoRA adaptation parameters or gated increments are uploaded, not the complete expert weights and complete gated weights; the parameter transmission packet should include at least the device cluster number, expert number, MoE layer number, orbital period number, global model version number, and LoRA rank. Quantization bit width, assigned expert mask, LoRA matrix or incremental parameters, gating adaptation parameters or gating increment, data length, local loss change information required for admission judgment, and check code; S602. In the uplink transmission phase, the device cluster estimates the uplink capacity according to the connection window and parameter transmission packet size Arrange the transmission order, preferentially upload the parameter packet with higher expert-related probability or greater training benefit and expected complete transmission If the current connection window is insufficient to transmit all parameter packets, the untransmitted portion is retained in the local update branch and transmitted in the next connection window; if a single parameter packet cannot be transmitted in one go, it is divided into parameter fragments with fragment sequence numbers and check codes, which are then cached and reassembled by the satellite according to the fragment sequence numbers after reception. The parameter transmission packet size is expressed as: ,in, Represents device cluster The assigned expert group, Indicates the quantization bit width. Denotes the rank of LoRA. and They represent experts respectively. The input and output dimensions of the weight matrix to be adapted. Indicates the gating adaptation parameters or the gating increment size. This indicates the size of the metadata, including version number, expert number, mask, fragment sequence number, and checksum; since LoRA only transmits... A single adaptation parameter, not the complete weights. Therefore, in much smaller and This can significantly reduce the amount of parameter transmission; After receiving the parameter transmission packet or parameter fragment, the satellite first performs integrity verification and version consistency check. If the verification passes, the parameter is written into the corresponding expert cache according to the device cluster number, expert number, and model version, and the expert update admission and controlled merging mechanism is input. Parameters that pass the admission judgment are used to update global expert or gated parameters. Parameters that fail the admission judgment are processed by the satellite by being postponed, pruned, rejected, or retained as a local update branch. S603. During the downlink transmission phase, the satellite only sends the merged expert LoRA adaptation parameters or expert increments, gating adaptation parameters or gating increments, expert version numbers, expert masks and merging weights required for subsequent training to the relevant equipment clusters, without broadcasting the complete MoE model; After the device cluster receives, the adaptive parameters issued are combined with the local frozen basic expert weights to obtain the local available expert parameters and as a starting point for the next round of connection phase or disconnection phase training.

Images