Multi-edge device collaborative reasoning method and system oriented to hybrid expert large model

Abstract

The invention discloses a multi-edge device collaborative reasoning method and system for a hybrid expert large model, and the method comprises the steps: collecting and analyzing system data, and adjusting the expert layout according to the system data; in the first stage, the number of experts required by each layer of a server is dynamically determined by balancing activation diversity and memory resource limitation; in the second stage, according to the expert number and the activation mode of each layer obtained in the first stage, low-time-delay reasoning is achieved by minimizing the calling times of remote experts, and the experts are distributed to all the servers. The system includes a global scheduler and an edge server participating in the system. By using the method, edge multi-machine joint reasoning is achieved, the deployment method is optimized, online adjustment deployment can be performed according to data changes, and the reasoning speed is increased compared with other deployment technologies. The method can be widely applied to the technical field of distributed machine learning.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning technology, and in particular to a multi-edge device collaborative inference method and system for hybrid expert large models. Background Technology

[0002] The Mixture-of-Experts (MoE) architecture has become a core design choice for large-scale training of large language models (LLMs). By integrating multiple specialized sub-models (called experts), MoE achieves significant performance improvements without increasing computational costs.

[0003] Current mainstream methods primarily rely on cloud server clusters for MoE inference, fully utilizing their powerful computing and memory resources. However, this approach often comes with high cloud service costs and significantly increased response times due to network latency. Many small and medium-sized organizations (such as schools, government departments, and enterprises) already possess a certain level of GPU computing resources; however, as the parameter size of large models continues to grow, single-point resources are insufficient to meet the deployment requirements of complete models. Through cross-node collaborative inference, users can integrate and efficiently utilize these dispersed existing resources, achieving a balance between economics and feasibility for large MoE models.

[0004] In this application scenario, due to memory limitations, each participant in collaborative inference deploys only a subset of model parameters. Participants may be from different organizations with significantly different data characteristics and task types. For example, schools might handle subject-specific tasks (such as Q&A, research summaries, etc.), government departments might focus on public service tasks, while enterprises are more geared towards business operations. The collaborative inference system needs to be adaptable and able to schedule different tasks and data distributions to achieve efficient matching of heterogeneous resources and diverse inference needs while protecting local data privacy. Summary of the Invention

[0005] In view of this, in order to address the technical problem that most existing collaborative reasoning methods rely on the computing resources of cloud server clusters, resulting in high costs, this invention proposes, firstly, a multi-edge device collaborative reasoning method for hybrid expert large models, which includes the following steps:

[0006] Collect and analyze system data, and adjust the expert layout accordingly;

[0007] Phase 1: Dynamically determine the number of experts required at each layer of the server by balancing activation diversity and memory resource constraints;

[0008] Phase Two: Based on the number of experts and activation modes at each level obtained in Phase One, minimize the number of remote expert calls to achieve low-latency inference by distributing experts to each server.

[0009] In some embodiments of the first aspect, it further includes:

[0010] An expert migration mechanism is established to accurately quantify and weigh the long-term communication cost-benefit ratio against the instantaneous migration overhead, and the expert deployment strategy is updated regularly.

[0011] In addition to the aforementioned multi-edge device collaborative reasoning method for hybrid expert large models, this invention also proposes a multi-edge device collaborative reasoning system for hybrid expert large models. This system consists of two main components that work together to achieve efficient distributed reasoning of the MoE model. These components include a global scheduler and edge servers participating in the system. The global scheduler is used to collect and analyze system data and adjust the expert layout accordingly. The edge servers are used to execute the online reasoning process.

[0012] Based on the above scheme, this invention provides a multi-edge device collaborative inference method and system for hybrid expert large models. By using multiple machines to collaboratively deploy the MoE model in edge scenarios, leveraging activation sparsity and local workload characteristics, it minimizes cross-server communication and performs joint inference on resource-constrained devices. Furthermore, deployment optimization is performed based on the inference properties of edge servers to improve inference efficiency. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the steps of a multi-edge device collaborative reasoning method for a hybrid expert large model according to the present invention.

[0014] Figure 2 This is a comparison chart showing the trend of local computation ratio of different methods under different model-dataset configurations as a function of inference runtime.

[0015] Figure 3 This is a diagram illustrating the difference in the average ratio calculated locally with and without migration operations in scenarios where the dataset changes.

[0016] Figure 4 This is a diagram illustrating the latency differences between servers under scenarios where dataset changes occur, and whether or not migration operations are performed.

[0017] Figure 5 This is a schematic diagram of the simulation results for system scalability verification. Detailed Implementation

[0018] Besides the high cost of cloud services mentioned in the background, some studies (such as MoE-Infinity) optimize MoE inference on a single device through memory tiering (such as GPU-RAM swapping) or prefetching strategies. However, these single-machine methods cannot leverage the collaborative computing capabilities of multiple devices in edge environments and are limited by single-machine memory capacity, making it difficult to support increasingly large-scale modern MoE models. Some works (such as SmartMoE and EPLB) are designed for data centers, employing uniform expert allocation or load balancing strategies. They do not consider the hardware heterogeneity of edge servers (such as differences in GPU performance and unstable network bandwidth) and dynamic workloads (such as expert activation mode shifts caused by changes in task type), thus failing to achieve ideal results in edge scenarios.

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0021] It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this application are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0022] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "a," and / or "the" are not specifically singular and may include the plural. Generally, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0023] In the description of the embodiments of this application, "a plurality of" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0024] Furthermore, flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Additionally, other operations can be added to these processes, or one or more steps can be removed from them.

[0025] Reference Figure 1 This is a flowchart illustrating an optional example of the multi-edge device collaborative reasoning method for hybrid expert large models according to the present invention. This method can be applied to computer devices, and the multi-edge device collaborative reasoning method proposed in this embodiment may include, but is not limited to, the following steps:

[0026] Step S1: Obtain system data;

[0027] Step S2: Analyze the system data and adjust the expert layout;

[0028] To reduce remote calls while meeting memory and expert coverage constraints, the expert layout adjustment method includes two phases:

[0029] In the first phase, the total number of experts at each level is distributed across all servers by balancing activation diversity and memory capacity.

[0030] In the second stage, based on the number of experts at each level obtained in the first stage, the total expected number of remote expert calls is minimized, and specific experts are selected and assigned to each server.

[0031] In some feasible embodiments, the first stage specifically includes:

[0032] The number of experts N is allocated proportionally based on the local activation diversity of each server. n,l This phase introduces a Shannon entropy-based quantification of activation diversity. It captures the statistical characteristics and uncertainties of expert activation patterns across different servers and model layers, rather than simply relying on the arithmetic mean of historical frequencies. This allows the system to prioritize deploying more experts on servers with more dispersed and unpredictable activation patterns, significantly reducing the expected value of remote calls. Intuitively, a higher entropy value indicates that many experts are called at similar frequencies, requiring the deployment of more expert replicas to ensure lower communication overhead. Conversely, a low-entropy distribution requires fewer local experts to achieve good service.

[0033] Let p = (p1, ..., p) E () represents the activation distribution of E experts in a layer. Let its Shannon entropy be . Then, for any δ∈(0,1), the number of experts required to cover at least (1-δ) activation quality satisfies:

[0034] k δ ≥2 H(p)-δlog E .

[0035] Proof: This conclusion stems from the principle of asymptotic equipartition: the cardinality of a typical set covering (1-δ) probability mass is 2. H(p)-δlog E .

[0036] This result provides a formal explanation for why entropy can serve as an effective proxy for the number of experts: higher entropy implies a more uniform distribution of demand, thus requiring a wider deployment of experts.

[0037] The specific calculation method for the first stage is as follows: First, calculate the entropy of the expert activation of server n in layer l:

[0038]

[0039] in Shannon entropy reflects the diversity of expert usage: lower entropy (low diversity) occurs when a few experts dominate, and higher entropy occurs when expert activation is evenly distributed.

[0040] The number of experts is initialized to:

[0041]

[0042] Where M n =∑ g mem n,g This represents the total amount of server memory. This heuristic reflects the intuitive understanding that the more diverse the activation patterns, the larger the set of experts required for co-location.

[0043] To satisfy the expert coverage constraint in problem modeling (i.e., each layer must satisfy ∑ n N n,l =E l In this embodiment, the initial allocation is adjusted through a rebalancing mechanism. When the total number of experts in a certain layer is less than E... l At this point, the system iteratively borrows expert resources from the currently over-configured layer l'. The server processes these resources in descending order of memory capacity to ensure feasibility, and this process is repeated until the total number of experts at all levels reaches the predetermined requirement.

[0044] In some feasible embodiments, the second stage specifically includes:

[0045] Experimental tests showed that inference latency is positively correlated with the proportion of cross-machine communication. Therefore, we hope to minimize the number of remote expert calls to achieve low-latency inference.

[0046] Build an expert preference list for each server n maintaining layer l And according to historical activation frequency Sort.

[0047] For each layer l, the server selects a preference list. Middle front N n,l One expert is used for local deployment.

[0048] To ensure complete coverage of experts within the system (i.e., for each expert e∈E) l (At least deployed on one server), this embodiment identifies unassigned experts iteratively and reassigns them to servers with redundant experts. For each unassigned expert... In this embodiment, the server with the fewest replica allocations is selected for deployment. Among the selected servers, the replica expert with the lowest usage frequency is replaced by e.

[0049] (Theoretically guaranteed) Minimize the total expected number of remote expert calls. Since no remote communication occurs if the expert is deployed locally on the requesting server, minimizing this objective is equivalent to maximizing the local frequency of experts assigned to each server n.

[0050] in,

[0051]

[0052] This formula represents local allocation. The communication savings efficiency.

[0053] This inspired the following definition of the local utility function and the analysis of the greedy allocation of high-frequency expert processes in algorithms:

[0054] The greedy approximation of local expert allocation, let's assume... This indicates that server n is an expert e∈E at layer l. l The normalized activation frequency. Let... The algorithm provides the total expert budget for server n. Define the local utility function:

[0055]

[0056] in It can be of any size and at most B n A subset of. Then the algorithm returns a greedy allocation. satisfy:

[0057]

[0058] in To maximize U n (·) of B n Optimal allocation of size. Equivalently, expected communication cost:

[0059]

[0060] satisfy:

[0061]

[0062] in To achieve the lowest achievable cost, The difference is the greedy approximation.

[0063] function U n (S) exhibits monotonicity and submodality due to its aggregation of independent non-negative activation weights. The choice of weight with the largest... A greedy algorithm for maximizing U under cardinality constraints. n This results in an approximate ratio of (1-1 / e).

[0064] In some feasible embodiments, it also includes:

[0065] Expert migration mechanism.

[0066] To adapt to dynamic workloads, evolving activation patterns, or hardware fluctuations, the system periodically updates its expert placement strategy. Expert migration is not based on simple threshold triggers, but rather on a precise quantitative trade-off between long-term communication cost-benefit and instantaneous migration overhead. At fixed time intervals, the global scheduler reruns the expert allocation method in step S2 using the latest activation statistics to generate candidate solutions.

[0067] Adopting the new layout will incur migration costs due to the transfer of expert models across servers, the quantification of which is as follows:

[0068]

[0069] in and Indicates whether expert e or expert e' is on GPU g of server n, m e For expert model size, speed n,g This represents the I / O bandwidth of GPU g on server n.

[0070] The system compares the current layout using an objective function. With the new layout The expected total communication cost from the last expert migration to the present time. A new layout will only be adopted if the improvement in communication costs exceeds the migration overhead.

[0071]

[0072] Here, C(·) represents the expected remote expert call cost under a given layout. This condition ensures that the system only triggers migration when the global benefits clearly outweigh the costs, effectively avoiding system oscillations and performance degradation caused by frequent migrations. The migration mechanism of this scheme does not require load changes to follow a specific pattern or distribution. Whether the load drifts slowly or changes abruptly, as long as the change allows the new expert layout to bring sufficient communication cost benefits, the system can trigger an adjustment. This decoupled, benefit-cost-based, general adaptability from specific load patterns demonstrates outstanding robustness and versatility.

[0073] Based on the overall method flow of the above embodiments, the present invention also provides specific data examples:

[0074] Experimental environment: The test platform consisted of a machine equipped with four A100 GPUs (40GB VRAM) and 256GB of RAM. Three edge servers were simulated, with GPU allocation schemes of 1, 1, and 2 GPUs respectively. Inter-server communication was established via a Docker network, and the bandwidth was limited to 500Mbps using the flow control tool (tc) in the Linux kernel.

[0075] Datasets: Two server dataset configuration schemes were evaluated: ① Each server handles different BIG-bench tasks (summarization, arithmetic reasoning, ASCII character recognition); ② MMLU-Pro, WikiText, and Tako tasks are deployed on three separate servers.

[0076] BIG-bench: BIG-bench is a collaborative evaluation framework that includes over 200 diverse tasks. It limits the length of the model output to the length specified for each dataset.

[0077] MMLU-Pro: This dataset covers over 12,000 questions across 14 domains. This example sets the input format to "question-option" pairs and limits the output length to the length of the correct answer.

[0078] WikiText: This language modeling dataset is selected from a featured entry on Wikipedia. This example limits the maximum output length of the model to 20 tokens.

[0079] Tako: When generating benchmark tests based on TACO code, this embodiment uses its test set problem as model input, while limiting the maximum output length to 20 tokens.

[0080] Model: This embodiment uses two MoE architectures in its experiments:

[0081] 1) Mixtral-8x7B (32-layer structure, with 2 out of 8 experts activated in each layer), its total number of parameters exceeds the memory capacity of a single GPU;

[0082] 2) Deepseek-V2-Lite (a 26-layer structure, with 8 experts selected from 64 experts per layer, resulting in 1,664 expert combinations) presents a more complex routing mechanism while maintaining higher memory efficiency. To simulate real-world edge deployment scenarios, this embodiment artificially limits GPU memory allocation: Mixtral-8x7B uses 70% of the total capacity, while Deepseek-V2-Lite uses 30%, reflecting their different memory requirements and operational constraints.

[0083] Comparison Methods: The following methods were selected for comparative experiments in this embodiment:

[0084] Uniform distribution: Expert modules are evenly distributed across all devices. For example, in this experiment using 4 GPUs, the Mixtral model randomly assigns 2 experts per layer to each device, while the Deepseek-V2-Lite model assigns 16 experts. This allocation strategy is consistent with the expert parallel implementation scheme of the Megatron-LM project.

[0085] Redundancy: The heuristic method proposed in this embodiment allows experts to be repeatedly deployed across servers, randomly assigned until the device capacity is fully utilized. When the total GPU memory of the system exceeds the model's requirements, this strategy can improve memory utilization compared to uniform distribution.

[0086] SmartMoE: Although the framework is primarily geared towards training optimization, this embodiment specifically implements its expert placement module, which can dynamically allocate experts among GPUs based on real-time workload, maintaining a uniform distribution of experts among devices by balancing workload.

[0087] Expert Parallel Load Balancer (EPLB): Derived from the redundant expert strategy of DeepSeek-V3, it achieves GPU load balancing by replicating high-load experts and heuristically assigning them. Since the open-source implementation only supports homogeneous systems, this embodiment reimplements the algorithm within the framework to adapt to heterogeneous configurations.

[0088] System parameters: This system performs an expert migration assessment every 5 minutes. This embodiment uses historical communication time and expert computation time as evaluation indicators. Specifically, the average value of all execution records from the most recent deployment plan change to the current moment is selected as the cost calculation benchmark for the new layout. The computation time, as a load-sensitive variable, is updated every 30 seconds.

[0089] As shown in Table 1, this method demonstrates excellent robustness across different datasets and server configurations, showcasing its practical application advantages. Uniform and SmartMoE suffer from limited performance improvements due to low resource utilization; while Redundance, despite introducing an expert replica mechanism, suffers from increased latency due to its suboptimal implementation. Specifically, on the BigBench dataset, compared to the second-best EPLB method, this method reduces the average latency of DeepSeek-V2-Lite by 30.6%; for the Mixtral model, it achieves a small but stable performance improvement. On the MultiData dataset, this method further reduces the latency of DeepSeek-V2-Lite by 10.9%, and the Mixtral model also shows a 2.0% improvement. Since EPLB, as a load balancing algorithm, does not consider cross-machine communication optimization, its performance is inferior to this method. For the Mixtral model, with a small number of experts and limited server capacity, its optimization space is relatively limited, thus the performance gain of this method is relatively small.

[0090] Table 1 compares the service latency (in seconds) of DeepSeek-V2-Lite and the Mixtral 8×7B model on the BigBench (10-second Poisson arrival) and MultiData (20-second Poisson arrival) datasets. The table presents the independent latency of the three servers and their overall average. Bold values ​​indicate the optimal latency, and underlined values ​​indicate the second-best result. This solution achieves the lowest average latency, demonstrating particularly strong performance on models like DeepSeek-V2-Lite, which have a large number of experts.

[0091] Table 1 Comparison Results

[0092]

[0093] To evaluate the impact of cross-machine communication on inference latency, this embodiment uses the local computation ratio as a metric for communication overhead. Except for the Uniform and Redundance baseline methods, all other methods employ the migration strategy of this scheme, but the expert placement strategies differ. For example... Figure 2 As shown, this approach consistently maintains a higher local computation ratio. Notably, this advantage is further amplified after explicit migration events on the BigBench dataset (triggered every 5 minutes) and the MultiData dataset (triggered every 10 minutes).

[0094] This embodiment uses the Deepseek-V2-Lite model ( Figure 3 4) The performance differences between the dynamic migration system and the static system were compared and evaluated. Figure 3 This demonstrates the improved local task computing power resulting from the proposed migration strategy. Figure 4The impact of necessary migration operations between different servers on latency was quantified. The experiment was set up so that each server handled 200 MultiData requests before switching to 200 BIG-bench requests. The initial performance of both systems was the same, but the scheme with the migration mechanism enabled (…) Figure 3 The curve marked "w" shows a significantly higher local computation ratio after the initial migration. (For...) Figure 3 The three migration phases shown are explained as follows: The first migration is triggered by the system detecting a deviation between the accumulated inference data and the dataset used in the initial layout (satisfying the conditions of the expert migration formula); the second migration is triggered by a change in the distribution of inference data (see...). Figure 4 (The "dataset changed" marker is in the middle); since the cost in the formula is calculated based on the cumulative data of the old and new data at this time, as new data continues to accumulate, the system eventually triggers a third migration. Experiments show that the migration scheme achieves a 10% reduction in average latency compared to the static scheme (optimized from 7.48 seconds to 6.73 seconds).

[0095] To verify the scalability and performance of the proposed system, this embodiment developed an event-driven simulator, mainly to achieve two goals: (1) to verify the performance improvement effect of the system by dynamically expanding the number of GPUs from 4 to 256; and (2) to quantify the impact of communication bandwidth (configured in the range of 100Mbps to 1000Mbps) on the system inference time. The simulator coordinates the inference process through the following mechanisms: selecting the network layer to be processed, updating the server timestamp, recording the completion time, and calculating the inference latency. Through incremental time accumulation, the temporal evolution process of the entire system is accurately modeled. The simulator contains five core components:

[0096] (1) Request sequence generator: integrates Poisson arrival time series with operational data collected from this system (including expert selection mode and token processing volume);

[0097] (2) Request route generator: simulates data flow under different expert layout configurations;

[0098] (3) Communication and computation time estimator: Establish a linear model to predict the processing time of each batch of tokens;

[0099] (4) Timestamp calculator: Captures communication and calculation events and timestamps them;

[0100] (5) System timing scheduler: Interleaving communication and computing events according to a preset strategy.

[0101] like Figure 5 As shown, as the number of GPUs increases, the average time per request decreases by 9% (Poisson arrival interval 15 seconds) to 19% (Poisson arrival interval 8 seconds), and the performance improvement is more significant with higher request density (Poisson interval 8 seconds vs 15 seconds). Figure 5 It also shows that higher bandwidth can significantly reduce average processing time, achieving an improvement of over 55% in a 4-GPU configuration, but this advantage diminishes as the GPU size increases, with only a 35% improvement at 256-GPU configurations.

[0102] Figure 5 Simulation results are used to verify the system's scalability. The left subplot shows the potential optimization space for system latency as the number of GPUs increases, through time costs and queuing delays with different numbers of GPUs; the right subplot shows that bandwidth limitations have a significant impact on system performance.

[0103] A multi-edge device collaborative reasoning system for hybrid expert large models, applied to the multi-edge collaborative reasoning method described above, includes a global scheduler and edge servers participating in the system, wherein:

[0104] The global scheduler, as the core intelligent hub of the system, maintains a panoramic view of the distributed environment. It continuously collects and updates system-level data, including cross-server communication latency, computing power, GPU memory, and historical activation patterns of experts on each server. This information drives the "deployment strategy" module, which dynamically determines the optimal allocation scheme of experts to servers based on current resource utilization and historical workload. This adaptive deployment mechanism minimizes cross-server communication while maximizing the local hit rate of frequently accessing experts.

[0105] Each server is equipped with a local gating network that determines the most appropriate expert to activate for each input token, regardless of whether the expert is located on a local or remote server.

[0106] Edge servers are used to execute the inference process. The execution of online inference includes the following steps:

[0107] 1. When input data flows through the system, the hidden state h is first calculated through non-MoE modules. At the MoE layer, the gating network routes h to the selected expert. Data is routed to either the local expert or an expert on a remote server.

[0108] 2. Local experts perform local calculations, while remote experts need to initiate remote calls;

[0109] 3. The expert calculation results are sent back to the original server for aggregation.

[0110] Steps 1, 2, and 3 are repeated with subsequent non-MoE layers and L-1 similar modules to generate the final output.

[0111] Data such as gating decisions and expert call costs during inference are recorded and reported to the global scheduler. This mechanism enables continuous monitoring of workload and resource usage.

[0112] The global scheduler periodically analyzes the collected data and dynamically adjusts the expert layout to adapt to changes in access patterns. When a significant change in the frequency of expert use is detected, the system automatically triggers the expert migration process, thereby maintaining efficient operation in the ever-changing edge environment.

[0113] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0114] A multi-edge device collaborative reasoning device for hybrid expert large models:

[0115] At least one processor;

[0116] At least one memory for storing at least one program;

[0117] When the at least one program is executed by the at least one processor, the at least one processor implements a multi-edge device collaborative reasoning method for a hybrid expert large model as described above.

[0118] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0119] A storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement a multi-edge device collaborative reasoning method for a hybrid expert large model as described above.

[0120] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0121] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-edge device collaborative reasoning method for hybrid expert large models, characterized in that, Includes the following steps: Obtain system data; The system data is analyzed, and the expert layout is adjusted. The adjustment method is as follows: The number of experts at each layer is determined by balancing activation diversity and memory capacity; With the goal of minimizing the total expected number of remote expert calls, experts are assigned to each server based on the number of experts at each layer.

2. The multi-edge device collaborative reasoning method for hybrid expert large models according to claim 1, characterized in that, The step of determining the number of experts at each layer by balancing activation diversity and memory capacity specifically includes: Calculate the entropy of expert activation at a certain layer of the server; Initialize the number of experts based on the entropy; Considering expert coverage constraints, processing is prioritized in descending order of memory capacity, and the number of experts is adjusted through a rebalancing mechanism.

3. The multi-edge device collaborative reasoning method for hybrid expert large models according to claim 2, characterized in that, The formula for calculating the entropy of expert activation at a certain layer of the server is as follows: Among them, v n,l E represents the entropy of the expert activation of the nth server at layer l. l Let p represent all experts in the l-th layer of the model, e represent a specific expert in the l-th layer, and p represent all experts in the l-th layer. e This indicates the frequency at which expert e is activated.

4. The multi-edge device collaborative reasoning method for hybrid expert large models according to claim 2, characterized in that, The initial formula for the number of experts is expressed as follows: Among them, M n Indicates the server's video memory capacity, v n,l Let m represent the entropy of the expert activation of the nth server at layer l. e L represents the amount of video memory occupied by the parameters of expert e, and L represents the total number of layers in the server.

5. The multi-edge device collaborative reasoning method for a hybrid expert large model according to claim 1, characterized in that, The objective of minimizing the total expected number of remote expert calls, and the allocation of experts to each server based on the number of experts at each layer, specifically includes: Build a list of expert preferences and sort it by historical activation frequency; With the goal of minimizing the total expected number of remote expert calls, for each layer, the server selects the top preset number of experts from the expert preference list for local deployment, where the preset number is the number of experts for the corresponding layer. Unassigned experts are identified through an iterative process and reassigned to servers with redundant experts.

6. The multi-edge device collaborative reasoning method for a hybrid expert large model according to claim 2, characterized in that, Also includes: After a preset time interval, the global scheduler regenerates candidate allocation schemes based on activation statistics. Calculate the migration cost of candidate allocation schemes, and when the improvement in communication cost exceeds the migration overhead, adopt the candidate allocation scheme as the new layout.

7. The multi-edge device collaborative reasoning method for hybrid expert large models according to claim 2, characterized in that, The formula for quantifying migration costs is as follows: in, and These represent the expert deployment before and after the migration. and Indicates whether expert e or expert e' is on GPU g of server n, m e For expert model size, speed n,g This represents the I / O bandwidth of GPU g on server n.

8. A multi-edge device collaborative reasoning system for hybrid expert large models, characterized in that, The method for multi-edge device collaborative reasoning for a hybrid expert large model as described in claim 1 includes: The global scheduler continuously collects and updates system data, including cross-server communication latency, computing power, GPU memory, and historical activation patterns of experts on each server, and determines the optimal allocation scheme from experts to servers. Edge servers are used to perform online inference.

9. A multi-edge device collaborative reasoning device for hybrid expert large models, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a multi-edge device collaborative reasoning method for a hybrid expert large model as described in any one of claims 1-7.

Images