Multi-platform large model dynamic fragmentation calculation method and system based on computing power self-sensing

Abstract

The invention provides a multi-platform large model dynamic fragmentation calculation method and system based on computing power self-sensing, and relates to the technical field of computing power resource scheduling. According to the calculation method, due to bandwidth normalization, temperature factors and a complete computing power self-sensing mechanism, stable execution performance can be kept, reasoning delay jitter can be reduced, the computing power prediction capacity of mu ST can actively adjust fragments before the computing power is reduced, passive redeployment is avoided, and re-fragmentation and migration overhead can be reduced; the multi-granularity fragmentation system supports multiple structures, and the deployable range of the model can be effectively improved; a longer reasoning sequence and a higher number of concurrent users can be supported through state-level fragmentation, and the throughput of the long sequence is improved; according to the composite search method combining differentiatable Bandit and GFlowNet, a global sub-optimal even optimal fragmentation strategy can be quickly found in a complex search space, the exploration time can be shortened, and the common local optimum problem of a heuristic method is avoided; the system can be expanded to more than ten cloud-side-end devices for cooperation, and the expansion capability is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing power resource scheduling, in particular to a multi-platform large model dynamic slicing calculation method and system based on computing power self-sensing. BACKGROUND

[0002] With the wide application of large-scale language models and their multi-modal variants in natural language processing, computer vision and other fields, how to achieve efficient inference in a heterogeneous environment with massive edge devices and cloud collaboration has become the focus of academia and industry.

[0003] Traditional centralized cloud computing mode is difficult to meet the demand of real-time requirement of modern intelligent application due to bandwidth limitation, response delay and other problems. Therefore, a multi-platform large model dynamic deployment and resource scheduling system based on computing power self-sensing emerges as the times require. The system dynamically adjusts model deployment and inference strategy by real-time sensing and analyzing heterogeneous computing power resources distributed in end, edge, cloud and other multi-level platforms, so as to realize the optimal balance between service quality and resource efficiency in complex and changeable network environment.

[0004] There are two representative technologies in the prior art: Joint accuracy and latency-aware deep structure decoupling method (JALAD), from the paper "JALAD: Joint Accuracy and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution", co-authored by Hongshan Li, Chenghao Hu, Jingyan Jiang, Zhi Wang, Yonggang Wen, Wenwu Zhu, etc., has been formally published in the well-known conference of Parallel and Distributed Systems in 2018 IEEE 24th International Conference on Parallel and Distributed Systems. The conference was held in Singapore in 2018, and this paper was included in the conference proceedings published by the conference, with specific start and end page numbers from page 671 to page 678. The technology proposes a deep neural network split execution framework that jointly considers accuracy and delay in edge-cloud collaborative environment.

[0005] AdaMEC, a dynamically composable DNN deployment framework for mobile edge computing, is presented in the paper "AdaMEC: Towards a Context-adaptive and Dynamically Combinable DNN Deployment Framework for Mobile Edge Computing," co-authored by Pang Bowen, Liu Sicong, Wang Hongli, and others. It was officially published in the international computer networking journal *ACM Transactions on Sensor Networks* in December 2023 by the Association for Computing Machinery (ACM) in New York, USA. This paper is included in Volume 20, Issue 1, January 2024, article number 21, totaling 28 pages. This technology proposes a context-adaptive and dynamically composable DNN deployment framework for mobile edge computing, aiming to solve the problem of efficiently deploying deep neural networks in dynamic and heterogeneous edge environments.

[0006] However, existing computing resource scheduling suffers from problems such as insufficient computing power awareness, limited granularity of sharding, lack of ability to model the dynamic state of large models, huge search space, and limited support for multi-platform collaboration. Specifically, these problems are as follows: Insufficient computing power awareness and lack of dynamic prediction capabilities. Existing solutions generally rely on the static state of equipment or single-indicator prediction methods based on past measurements, lacking modeling of equipment temperature, bandwidth fluctuations, resource usage changes, and short-term prediction capabilities for future computing power, resulting in delayed sharding decisions and an inability to plan proactively.

[0007] The granularity of the fragmentation is too simple and cannot adapt to large model structures. AdaMEC mainly relies on operator-level atomization, while JALAD only performs simple layer-level segmentation of neural networks, which cannot cover the complex structure of large-scale neural networks. This results in large models not being able to achieve truly executable fine-grained decomposition in an edge-to-edge environment.

[0008] There is a lack of ability to model the dynamic states of large models. Existing solutions often assume that model parameters are static, intermediate states do not expand with the growth of sequences or tasks, and no new shardable structures are generated during inference execution. However, in LLM, medium-to-long sequences, concurrent inference, and multi-task scenarios, the amount of dynamic state in the model increases rapidly, rendering the sharding strategy completely ineffective.

[0009] The search space is enormous, but existing search methods are non-differentiable and non-trainable. Existing solutions typically employ heuristic strategies, ILP, or rule-based search. They cannot self-update during runtime, learn environmental patterns, or support continuous space optimization, making the search highly susceptible to getting trapped in local optima and virtually unable to scale to multi-device scenarios.

[0010] The multi-platform collaboration support is limited and cannot be extended to multi-terminal and multilateral environments. JALAD only supports a two-way collaboration mode of one terminal and one cloud. Although AdaMEC is advertised as having multiple devices, its actual design is mainly optimized around 2-3 devices. It lacks complete multi-level synchronization, communication overhead modeling and scheduling logic, so it is difficult to apply to large-scale heterogeneous device clusters. Summary of the Invention

[0011] To address the shortcomings of existing technologies, this invention provides a multi-platform large model dynamic sharding calculation method and system based on computing power self-awareness, which solves the problems of insufficient computing power awareness, single sharding granularity, lack of modeling ability for the dynamic state of large models, huge search space, and limited multi-platform collaborative support in existing computing power resource scheduling.

[0012] To achieve the above objectives, the present invention provides the following technical solution: A dynamic sharding computation method for large multi-platform models based on self-aware computing power, the computation method comprising: S1. Computing power self-sensing Real-time sensing, quantification, and prediction of the effective available computing power of cloud, edge, and terminal devices; S2. Multi-granularity fragmentation A four-level partitioning space is constructed: operator-level, subgraph-level, tensor-level, and state-level; the partitioning results together constitute a multi-granularity partitioning candidate space. in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. S3. Sharding Scheme Generation The results of self-aware computing power and the multi-granularity partitioning candidate space constructed using preceding modules A global cost model is constructed, and a composite search algorithm combining the jointly differentiable Bandit and GFlowNet is used to search in the multi-granularity partition candidate space to obtain the partition combination with the minimum total cost. ; S4. Multi-platform deployment The model is divided into four granularities. Based on the hardware architecture characteristics of the target device, compilability verification and compiler selection are performed, and a cross-device execution graph is constructed. Then, the shards are distributed and inference is started. During the execution process, small-scale rescheduling is performed based on the device status.

[0013] Preferably, S1 specifically includes: S1.1. Original Normalized Quantity Normalize the different dimensional data sensed by the device to ; Memory availability: in, for time Remaining video memory / RAM on the device; for Total video memory / memory of the device; Idle computing power ratio: in, for time The computing power utilization rate of the equipment ; Bandwidth normalization: in, for Time device With equipment Available bandwidth between; This is the preset upper limit of the bandwidth; Delayed normalization: in, for Time device To the equipment Single-hop delay; This is the upper limit for tolerance of delays; Heat factor: in, for time The temperature of the equipment; for The temperature threshold of the equipment; for The maximum allowable temperature of the equipment; S1.2. Calculation of Effective Computing Power of Equipment Smoothly calculate the effective computing power of multiple platform devices; Effective computing power of the device: in, For equipment Peak computing power of hardware: in, For equipment Number of parallel computing units in the medium; For equipment The clock frequency at which the computing unit executes one hardware cycle; For equipment The number of FLOPs that can be executed per unit per cycle; EMA smoothing: in, Computing power after historical smoothing; This is the smoothing coefficient in the exponential moving average; S1.3. Prediction of future computing power of equipment Using the μST architecture, for the future Use the computing power of the device for short-term prediction within a sampling period; Given input: Given K, construct the history sequence: Data preprocessing: A dual-path embedding approach is adopted, employing both periodic and bursty methods. For periodic paths, perform a Fast Fourier Transform (FFT) on the historical sequence: Periodic features are captured through a linear layer, and the frequency domain features are projected back into the time domain: in, It is a learnable projection matrix; It is a bias; For sudden paths, perform a first-order difference on the historical sequence. For any... The corresponding first-order difference is: The burst pattern shape is captured through local convolution, and the burst features are projected back to the temporal domain: The features extracted from the two paths are concatenated to obtain the initial representation: μST block: A mechanism of parallel global and local attention is adopted; For global attention blocks, the initial representation Transform from the time domain to the frequency domain: Attention in the frequency domain: in, For frequency domain lookup matrix; It is a frequency domain key matrix; It is a frequency domain value matrix; This is the scaling factor; Transform the sequence features in the frequency domain into the time domain: For a local attention block, let the window size be... The sequence features after local attention are: It consists of K+1 vectors generated by performing local attention on each of the K+1 input time steps; Where K represents the length of the historical sequence; L is the size of the local sliding window; No. At each time point, the local attention mechanism is executed to generate vectors. The formula is as follows: in, For the first Query vectors at each time point; For the first Key vectors at neighboring time points; This is a value vector of neighborhood time points; The feature sequences obtained by merging local attention and frequency attention: in, Indicates residual connection; Lightweight decoder: Through a multilayer perceptron, the features provided by the μST block are mapped to the future. One predicted value: Uncertainty estimation: The estimated computing power is conservatively reduced by estimating the model accuracy. Calculate the standard deviation using multiple samples from Dropout. To evaluate the accuracy of the model's predictions: in, This indicates the total number of Dropout samples; Based on the uncertainty, a conservative estimate is then made of the prediction results: in, This is a conservative coefficient.

[0014] Preferably, the specific process of operator-level partitioning includes: The model graph is analyzed by extracting each operator node to form a set of operator nodes; the input is the static representation of the model, and the output is a directed acyclic graph of operators. in, Represents a set of operators; Represents directed edges between operators based on tensor data dependencies; For each operator node, the system records its input tensor dimension, output tensor dimension, parameter size, type of mathematical operation used, and execution attributes; After constructing the operator graph, the system determines whether each operator node can serve as a potential partition boundary. Operator node Whether a node can be used as a sharding point depends on several feasibility conditions, including: topological feasibility, transmission feasibility, memory feasibility, and computing power feasibility; the input is the operator graph and the attribute information of each operator node, and the output is a set of operator-level sharding candidates. ; Define the topology feasibility judgment formula: Operator The feasibility of cross-device access is determined as follows: The size of the operator output tensor is , Time device With equipment The available bandwidth between is Single-hop delay is The maximum acceptable transmission time is ; Operator Can be deployed on devices The memory feasibility is defined as: in, This represents the peak video memory usage required by the operator during execution. for time The remaining video memory of the device; In the equipment The feasibility estimate for the available computing power is as follows: in, This indicates the number of floating-point operations required for the operator. For equipment Conservative computing power estimate, This indicates the maximum acceptable execution time specified by the system. For operator theory execution time; Based on the above four conditions, the overall feasibility judgment condition for operator nodes is defined as follows: Operator nodes that meet this condition and their corresponding deployment devices will be added to the operator-level sharding candidate set. .

[0015] Preferably, the specific process of subgraph-level partitioning includes: The subgraph pattern recognition method is used to find substructures that satisfy the feature template in the operator graph. These structures are obtained by graph pattern matching and strongly connected component analysis. The set of operator nodes contained in each subgraph and its internal data flow are recorded. After obtaining the candidate set of subgraphs, the subgraphs are... Feasibility must be determined; Subgraph The feasibility of cross-device access transmission is determined as follows: in, This is a bandwidth normalization parameter; For subgraph The size of the output tensor; Can be deployed on devices The memory feasibility is denoted as: Among them, the peak memory usage of the sub-image is ; In the equipment The feasibility estimate for the available computing power is as follows: Among them, subgraph The total number of floating-point calculations is: The execution time of operator theory is: Based on the above conditions, the overall feasibility conditions for subgraph-level fragmentation in this invention are defined as follows: Topological feasibility This ensures that the selected subgraph does not disrupt the directed acyclic structure of the computation graph; if the overall feasibility condition is met, the subgraph and its corresponding deployment device are included in the subgraph set fragment candidate set. .

[0016] Preferably, the specific process of tensor-level fragmentation includes: Identify all large tensors that satisfy the condition. This is used as a candidate tensor for tensor-level partitioning; among which A threshold for the size of the tensor given by the user; Perform large tensor row partitioning, column partitioning, and block partitioning; Next, tensor partitioning is performed. Feasibility assessment; Tensor partitioning The feasibility of cross-device access transmission is determined as follows: in, For subgraph The size of the output tensor; Can be deployed on devices The memory feasibility was determined as follows: in, tensor partitioning Peak video memory usage; In the equipment The feasibility of running on the computing power is determined as follows: in, Indicates the execution time of floating-point operations involved in the tensor block; Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: Based on the above determination, the tensor fragments that meet the conditions and their corresponding deployment devices are added to the tensor fragmentation candidate set. .

[0017] Preferably, the specific process of state-level fragmentation includes: The state-level partitioning can be partitioned by header, by sequence, or by batch dimension. Next, proceed with the state fragment. Feasibility assessment; State fragment The formula for determining the feasibility of cross-device access transmission is: in, State fragment Storage capacity; State fragment Can be deployed on devices The formula for determining memory feasibility is: in, State fragment Peak memory usage Then it is in the equipment The feasibility of running on the computing power is determined as follows: Among them, state fragments The computational cost of associated floating-point operations is Execution time is ; Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: When the above constraints are satisfied, the state and its corresponding deployment device will be included in the state-level fragmentation candidate set. .

[0018] Preferably, in step S3, the global cost model construction method specifically includes: The overall deployment plan is represented as a state vector: in, This indicates a segmentation scheme that includes four granularity segmentations; Any partition in the data belongs to the multi-granularity partitioning candidate space. ; It refers to the number of shards included in the deployment plan; vector It is a visual representation of the deployment location of each shard in the heterogeneous set of cloud, edge, and terminal devices; element For the first The device number assigned to each slice. A collection of heterogeneous devices; The total cost is defined as the sum of the cost of each fragment on its corresponding device and the communication cost between them: in, It is the number of fragments; For deployment plan Execution diagram; Indicates the first slice Deployed equipment and the slice Deployed equipment Data exchange exists; The cost of local execution is defined as: in, , , These are the weighting coefficients. Indicates fragmentation The computational complexity of floating-point operations; Indicates fragmentation Peak video memory usage required during execution; This represents a short-term prediction of future secure computing power from the μST model; The communication cost, based on the dependency graph and device bandwidth, is as follows: in, , These are the weighting coefficients; Indicates equipment and equipment In between, transmit information about fragments The amount of data.

[0019] Preferably, in step S3, a composite search algorithm combining differentiable Bandit and GFlowNet is used to find the total cost in this high-dimensional space. The minimum state; First, perform a differentiable bandit decision: Define "deploying shards to devices" as an action, and use the negative value of the local execution cost as the action value: in, Indicates the current search iteration round; Bandit utilizes a differentiable UCB strategy in each iteration to partition the candidate space at multiple granularities. Select action: in, No. The first fragment in the... The equipment selection results for the wheel; As of the date Before the round, the shards will be divided. Place in the device Number of attempts; Indicates the exploration of regulatory factors; Bandit takes the local cost matrix of each shard as input and outputs a set of candidate devices for each shard. Next, the candidate device set for each slice is locally expanded into a series of joint states. ; Next, a GFlowNet search is performed in the space of joint states: The reward for the joint state is defined as a negative exponent of the total cost: And establish flow conservation constraints between all states: in, The flow of state transitions is a learnable parameter of GFlowNet; From state Transition to state The flow rate value; Parameter update process: After completing one GFlowNet state sampling, the system calculates the total cost. This is used to update parameters in two directions: first, as a termination reward to update flow conservation constraints, affecting flow distribution; and second, as a local cost signal for the action value of Bandit. Perform differentiable updates; When the total cost decreases and tends to stabilize, the system outputs the joint sharding-deployment combination with the minimum total cost: The results also confirmed the four-granularity segmentation method. Deployment vectors of all shards .

[0020] Preferably, S4 specifically includes: Deployment plan A compatibility check is performed, taking the list of operators and tensor dimension structure inside the slice as input and outputting a set of intermediate representations that can successfully generate low-level executable code on the corresponding device. During this check, the system determines whether the slice can be directly compiled based on the instruction set, supported tensor shapes, and available mathematical libraries of the target device. If the slice contains operator patterns that are not supported by the target device, the system will rewrite the operators into equivalent operator sequences through an operator substitution table and update the internal execution graph of the slice to meet the architectural constraints of the target device. After compilability verification is completed, the system applies a compiler selection strategy to different fragments based on the device type; for tensor-level fragments, the compiler automatically generates row slices, column slices, or block slices, and for state-level fragments, it automatically generates the corresponding index mapping logic. After completing the fragmented compilation, the system needs to construct a runtime execution graph for cross-device loading and communication; this execution graph uses device allocation vectors. The input is a directed acyclic graph containing all execution nodes and communication nodes. The system uses bandwidth normalization. Delayed normalization quantity Select the optimal communication protocol; among which, establish an independent communication queue for state-level slices; Subsequently, the system sends the model weights, sharding parameters, and runtime status to the corresponding devices according to the deployment plan. For operator-level and subgraph-level sharding, the weights are directly copied to the target device according to the intermediate representation mapping at compile time. For tensor-level sharding, the system performs sharding on the weight matrix according to the sharding method before sending it out. For state-level sharding, the system adopts an incremental sending mechanism, that is, only the part required for each round of inference is sent out. After all weights and statuses are sent out, each device is bound to the cross-device execution graph at runtime using a unified execution entry pointer. Once all preparations are complete, the system distributes the execution graph to the device-side executors to initiate the inference process. During inference, the executors run shards locally according to the execution graph sequence, while simultaneously exchanging intermediate activations, key-value states, or expert weights with other devices through communication nodes. When the device's heat factor... Bandwidth usage Or future secure computing power When significant changes occur, the system runs an online adaptive correction mechanism, which involves micro-recompiling or migrating some fragments.

[0021] A multi-platform large-model dynamic sharding computing system based on computing power self-awareness, the computing system comprising: a computing power self-awareness module, a multi-granularity sharding module, a sharding scheme generation module, and a multi-platform deployment module; The computing power self-sensing module is used to sense, quantify, and predict the effective available computing power of each cloud, edge, and terminal device in real time, providing core basis for model sharding, dynamic scheduling, and deployment decisions. The multi-granularity partitioning module is used to construct a four-level partitioning space: operator-level, subgraph-level, tensor-level, and state-level; the partitioning results collectively constitute the multi-granularity partitioning candidate space. in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. The sharding scheme generation module utilizes the results of computing power self-awareness and the multi-granularity sharding candidate space constructed by the preceding modules. By constructing a global cost model and innovating a state-space algorithm, the combination of partitions with the minimum total cost is obtained. ; The multi-platform deployment module is used to compile and optimize the four types of model granularity according to the hardware architecture characteristics of their target devices, and to build a cross-device execution graph, so that each shard can run collaboratively between heterogeneous terminals, edge nodes and cloud servers. This deployment phase needs to ensure that each shard is compilable, cross-device call relationships can be established, parameters and status can be correctly delivered, and that the hardware status can be continuously monitored during execution for adaptive adjustments.

[0022] This invention provides a method and system for dynamic sharding computation of large multi-platform models based on self-aware computing power. Compared with existing technologies, it has the following advantages: In this invention, the computational method, due to bandwidth normalization, temperature factor, and a complete computing power self-awareness mechanism, can maintain relatively stable execution performance under network fluctuations, temperature increases, and load spikes, reducing inference latency jitter. μST's computing power prediction capability allows the system to proactively adjust sharding before computing power declines, thereby avoiding passive redeployment and reducing the overhead of re-sharding and migration, making it more efficient than schemes relying on simple latency estimation. The multi-granularity sharding system supports various structures such as CNN, Transformer, MoE, and LLM, effectively improving the deployability of models, especially suitable for resource-constrained devices. State-level sharding solves the problem of LLM key-value cache (KV-cache) bloat, enabling devices to support longer inference sequences and higher concurrent users, improving long sequence throughput. The combined search method of differentiable Bandit and GFlowNet can quickly find globally suboptimal or even optimal sharding strategies in complex search spaces, reducing exploration time and avoiding the local optima problem common in heuristic methods. The system can be scaled to more than ten cloud-edge-device collaborative devices, significantly improving scalability. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Fig. 1 This is a flowchart of the calculation method in an embodiment of the present invention.

[0025] Fig. 2 This is a flowchart illustrating the prediction of future computing power of a device based on μST in an embodiment of the present invention.

[0026] Fig. 3 This is a flowchart of the combined search method using differentiable Bandit and GFlowNet in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This application provides a method and system for dynamic sharding calculation of large models across multiple platforms based on self-aware computing power. This solves the problems of insufficient computing power awareness, single sharding granularity, lack of modeling ability for the dynamic state of large models, huge search space, and limited support for multi-platform collaboration in existing computing power resource scheduling.

[0029] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0030] Example: like Figs. 1-3 As shown, this invention provides a multi-platform large model dynamic sharding calculation method based on self-aware computing power, the calculation method including: S1. Computing power self-sensing Real-time sensing, quantification, and prediction of the effective available computing power of cloud, edge, and terminal devices provide core basis for model sharding, dynamic scheduling, and deployment decisions; S2. Multi-granularity fragmentation A four-level partitioning space is constructed: operator level, subgraph level, tensor level, and state level; the partitioning results together constitute the multi-granularity partitioning candidate space (MGPS); in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. Subsequent sharding deployment schemes will be searched in the multi-granularity sharding candidate space; the multi-granularity sharding candidate space not only describes feasible sharding nodes, but also implicitly contains information about the node deployment devices; S3. Sharding Scheme Generation The results of self-aware computing power and the multi-granularity partitioning candidate space constructed using preceding modules By constructing a global cost model and using a composite search algorithm combining differentiable Bandit and GFlowNet, the combination of fragments with the minimum total cost is obtained. ; S4. Multi-platform deployment The model fragments of four granularities are compiled and optimized according to the hardware architecture characteristics of their target devices, and a cross-device execution graph is constructed so that each fragment can run collaboratively across heterogeneous terminals, edge nodes and cloud servers. This deployment phase needs to ensure that each fragment is compilable, cross-device call relationships can be established, parameters and status can be correctly delivered, communication latency is controllable, and hardware status can be continuously monitored during execution for adaptive adjustments.

[0031] S1 specifically includes: S1.1. Original Normalized Quantity Normalize the different dimensional data sensed by the device to This is for subsequent calculations; Memory availability: in, for time Remaining video memory / RAM on the device; for Total video memory / memory of the device; Idle computing power ratio: in, for time The computing power utilization rate of the equipment ; Bandwidth normalization: in, for Time device With equipment Available bandwidth between; This is the preset upper limit of the bandwidth; Delayed normalization: in, for Time device To the equipment Single-hop delay; This is the upper limit for tolerance of delays; Thermal factor (used for temperature-induced frequency / performance degradation): in, for time The temperature of the equipment; for The device has a temperature threshold; when the temperature exceeds this threshold, the device automatically reduces its frequency. for The maximum allowable temperature of the device. When the device temperature approaches this value, the hardware will significantly reduce power or even trigger a protective shutdown.

[0032] S1.2. Calculation of Effective Computing Power of Equipment The effective computing power of multi-platform devices is calculated smoothly, providing a basis for subsequent device computing power prediction. Effective computing power of the device: in, For equipment Peak TFLOPS is determined by the hardware components of the device. in, For equipment The number of parallel computing units can be represented by the SIMD lanes of a CPU, the CUDA cores of a GPU, and the matrix multiplication units of a TPU / NPU, etc. For equipment The clock frequency at which the computing unit executes one hardware cycle; For equipment The number of FLOPs that can be executed per unit per cycle; This calculation method is applicable to heterogeneous platforms.

[0033] EMA smoothing: in, Computing power after historical smoothing; The smoothing coefficient in the exponential moving average (EMA) is used to control the degree of influence of the current observation on the smoothing result; it... The values ​​are between, where the larger one is the largest. This makes the system more focused on the latest equipment status and responds more quickly, but it is more susceptible to noise interference, while smaller... This emphasizes historical information, making the curve more stable but reacting more slowly to changes.

[0034] S1.3. Prediction of future computing power of equipment Using the microstructure Transformer (μST) architecture, for the future The short-term prediction of the device's computing power for each sampling period aims to provide support for the equivalent computing power of subsequent computing model shards running on the device; Given input: Given K, construct the history sequence: Data preprocessing: Dual-path embedding is used to amplify the periodicity and burstiness characteristics of the input sequence; For periodic paths, perform a Fast Fourier Transform (FFT) on the historical sequence to highlight the periodic characteristics: Periodic features are captured through a linear layer, and the frequency domain features are projected back into the time domain: in, It is a learnable projection matrix; It is a bias; For sudden paths, first-order differencing of historical sequences is performed to highlight short-term changes. For any... The corresponding first-order difference is: The burst pattern shape is captured through local convolution, and the burst features are projected back to the temporal domain: The features extracted from the two paths are concatenated to obtain the initial representation: Microstructure Transformer (μST) block: It adopts a dual-attention parallel mechanism, that is, it pays attention to the nearby history / local dynamics, and also utilizes frequency information / global patterns; For global attention blocks, the focus is on capturing long-term fluctuation patterns; Initial representation Transform from the time domain to the frequency domain: Attention in the frequency domain: in, For frequency domain lookup matrix; It is a frequency domain key matrix; It is a frequency domain value matrix; This is the scaling factor; Transform the sequence features in the frequency domain into the time domain: For local attention blocks, the focus is on capturing short-term, sudden changes; Let the window size be The sequence features after local attention are: It consists of K+1 vectors generated by performing local attention on each of the K+1 input time steps; Where K represents the length of the historical sequence; L is the size of the local sliding window; No. At each time point, the local attention mechanism is executed to generate vectors. The formula is as follows: in, For the first Query vectors at each time point; For the first Key vectors at neighboring time points; This is a value vector of neighborhood time points; This is the scaling factor; The feature sequences obtained by merging local attention and frequency attention: in, Indicates residual connection; Lightweight decoder: The features provided by the microstructure Transformer (μST) block are mapped to the future using a multilayer perceptron (MLP). One predicted value: Uncertainty estimation: The estimated computing power is conservatively reduced by estimating the model accuracy. Calculate the standard deviation using multiple samples from Dropout. To evaluate the accuracy of the model's predictions: in, This indicates the total number of Dropout samples; Based on the uncertainty, a conservative estimate is then made of the prediction results: in, This is a conservative coefficient.

[0035] In S2, operator-level sharding, as the lowest granularity of the model structure, is used to perform the finest-grained sharding of operators in the deep learning model computation graph to support heterogeneous scheduling between the end, edge nodes, and cloud platforms.

[0036] An operator is an atomic operation node in a deep learning framework. After being compiled by the framework, it forms a directed acyclic computation graph (DAG). Each node in the graph represents an indivisible operator, such as matrix multiplication, convolution, normalization, feedforward network unit, attention matrix computation (Attention Softmax), etc. Its input comes from the output of the preceding operator or the model input, and its output is consumed by subsequent operators. Therefore, operators constitute the execution path of the model.

[0037] The core objective of operator-level sharding is to identify operator nodes that can serve as sharding boundaries from the operator graph, and to determine which operators are combined to form an operator subgraph that can be deployed to a specific hardware platform through feasibility screening, thereby providing the basic structure for subsequent subgraph-level sharding.

[0038] The specific process of operator-level partitioning includes: Model graph parsing: During the model loading phase, the system parses the computation graph structure from the trained model format (such as TorchScript, ONNX, TensorRT IR, etc.) and extracts each operator node to form an operator node set; the input is the static representation of the model, such as ONNX GraphProto or PyTorch FX Graph, and the output is an operator directed acyclic graph. in, Represents a set of operators; Represents directed edges between operators based on tensor data dependencies; For each operator node, the system records its input tensor dimension, output tensor dimension, parameter size, type of mathematical operation used, and execution attributes.

[0039] After constructing the operator graph, the system determines whether each operator node can serve as a potential partition boundary.

[0040] Whether an operator node can serve as a sharding point depends on several feasibility conditions, including: topological feasibility, transmission feasibility, memory feasibility, and computational feasibility; the input is the operator graph and the attribute information of each operator node, and the output is a set of operator-level sharding candidates. .

[0041] Topological feasibility requires that the operator node not be in a loop structure in the computation graph to ensure that the model graph maintains a directed acyclic structure and avoids execution dependency cycles. Therefore, the topological feasibility judgment formula is defined as follows: Transmission feasibility requires that the output tensor of the operator can be transmitted between platforms and meet bandwidth constraints; let the size of the operator's output tensor be... , Time device With equipment The available bandwidth between is Single-hop delay is The maximum acceptable transmission time is (Manually set), the judgment is as follows: Memory feasibility requires ensuring that the target platform's available video memory can accommodate the memory requirements for executing the operator, including: operator parameter usage and intermediate activation usage; assuming This represents the peak video memory usage required by the operator during execution. for time The remaining video memory of the device is then defined as follows: Computational feasibility requires the platform to complete operator computations within a limited delay budget under its conservatively predicted computing power; assuming This indicates the number of floating-point operations required for the operator. For equipment Conservative computing power estimate, The maximum acceptable execution time specified by the system is indicated by the following computational feasibility estimate: in, This represents the execution time of the operator theory.

[0042] Based on the above four conditions, the overall feasibility judgment condition for operator nodes is defined as follows: Operator nodes that meet this condition and their corresponding deployment devices will be added to the operator-level sharding candidate set. This will serve as the basis for subsequent sharding combinations. A global execution cost model will then be used to select the optimal cost combination, forming a deployable solution.

[0043] In S2, subgraph-level partitioning is based on operator-level partitioning, further partitioning the set of operators with strong semantic relevance and high computational coupling in the model to obtain a more stable cross-device execution granularity.

[0044] A subgraph is a set of operators with a strongly connected structure in the computation graph (e.g., Transformer blocks, convolutional blocks, residual blocks, feedforward network blocks). These operators often form fixed computational patterns during inference. The input is the directed acyclic graph of the model's operators and the attributes of the operator nodes, and the output is a set of formally defined candidate subgraphs.

[0045] The specific process of subgraph-level partitioning includes: The subgraph pattern recognition method is used to find substructures that satisfy the feature template in the operator graph, such as the Transformer block composed of multi-head attention and feedforward network. These structures are obtained by graph pattern matching and strongly connected component (SCC) analysis, and the set of operator nodes contained in each subgraph and its internal data flow are recorded.

[0046] After obtaining the candidate set of subgraphs, the feasibility of the subgraphs is determined by using the device's real-time computing power normalization parameters and future computing power prediction values.

[0047] Unlike operator-level piecewise partitioning, subgraphs typically have larger input and output tensors and more frequent data transfers between internal operators. Therefore, a bandwidth normalization parameter is used in the transmission feasibility assessment. For cross-platform transmission, subgraph The size of the output tensor is denoted as , If the upper limit of the preset bandwidth is used, then the feasibility of transmission is determined as follows: On the other hand, since the subgraph contains multiple operators, the peak memory required for its execution is equal to the sum of the memory occupied by all operator parameters and the memory occupied by activation. Memory feasibility is denoted as: Regarding computing power feasibility, the conservative predicted computing power obtained from the forecast will still be used. The evaluation was conducted based on the total floating-point computation of the subgraph. Based on this, its theoretical execution time is calculated. The computational feasibility estimate is as follows: in This represents the execution time of the operator theory.

[0048] Based on the above conditions, the overall feasibility conditions for subgraph-level fragmentation in this invention are defined as follows: Topological feasibility This ensures that the selected subgraph does not disrupt the directed acyclic structure of the computation graph. If the overall feasibility condition is met, the subgraph and its corresponding deployment device are included in the subgraph set sharding candidate set. Furthermore, a global execution cost model will be used to select the optimal cost combination, forming a deployable subgraph sharding scheme. This method enables complex structures (such as Transformer Blocks) to be migrated to different devices for execution as a whole, significantly reducing the communication overhead between operators and improving the efficiency and stability of cloud-edge-device collaborative inference.

[0049] In model architectures, large-scale weight tensors (such as linear layer weight matrices and attention projection matrices) often cannot be deployed entirely on edge devices with limited GPU memory. Therefore, a tensor-level partitioning mechanism is introduced to ensure that these large tensors can be distributed and computed in a distributed manner using column-split, row-split, or block-split methods. The input consists of the large-scale weight tensors in the model. The output is one or more sub-tensor blocks. .

[0050] In S2, the specific process of tensor-level fragmentation includes: Identify all large tensors that satisfy the condition. ,in A threshold for tensor size is set by the user. Large tensors that meet this threshold are considered candidate tensors for tensor-level partitioning.

[0051] For the row splitting pattern, the weight tensor is divided into: Calculate separately for each device: The final output is concatenated as follows: For the column partitioning pattern, the tensor is divided into: Calculate separately for each device: Output merged into For the block partitioning mode, the tensor is divided into A distributed computing approach is adopted.

[0052] When determining the feasibility of tensor fragmentation, the bandwidth normalization parameter is used to determine the tensor fragmentation method. Can the transmission be completed within the time limit? The feasibility of the transmission is determined as follows: For memory feasibility, dynamic constraints are applied using the device's memory availability parameter. For a given number of fragments... Its peak memory usage is denoted as This tensor fragment can then be deployed on the device. The memory feasibility was determined as follows: Regarding computing power feasibility, the conservative predicted computing power obtained from the forecast will still be used. If the execution time of matrix multiplications involved in tensor partitioning meets the system latency budget, then the computational feasibility is determined as follows: in, This indicates the execution time of floating-point operations involved in the tensor block.

[0053] Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: Based on the above determination, tensors that meet the conditions and their corresponding deployment devices are added to the tensor-level fragmentation candidate set. Subsequently, a global cost model is used to select the optimal combination, so that the high-dimensional weight structure of the large model can be deployed on edge devices with limited computing power, thereby improving the model coverage and availability.

[0054] Deep learning models dynamically generate a large amount of state data during inference, including key-value caches in the attention mechanism, expert parameters and activation states in the Model of Experts (MoE), and activation checkpoints. The state data expands exponentially with the length of the input sequence or the number of batches. Therefore, a state-level sharding mechanism is introduced to distribute and manage state data across different devices, thereby avoiding memory overflow and ensuring long-sequence inference capabilities. The input is a set of state tensors from the model's inference process, and the output is its sharded distributed storage structure.

[0055] The specific process of state-level sharding includes: Taking KV-cache as an example, this invention supports sharding by attention head-wise, by sequence range, or by batch dimension. When sharding by head, the following applies: When partitioning by sequence, we have: When splitting by batch dimension, we have: Generally speaking, if you need to access a state segment across devices... This invention uses bandwidth normalization parameters to determine its transmission feasibility: Similarly, determine memory feasibility. If a certain state segment If the following formula is satisfied, then the state fragment can be deployed on the device. The formula for determining memory feasibility is: For state segments that require computation, such as MoE expert activation, this invention still uses conservative predictive computing power. Determine its computational feasibility. If the state segment... The computational cost of associated floating-point operations is Execution time is Then it is in the equipment The feasibility of running on the computing power is determined as follows: Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: When the above constraints are satisfied, the present invention includes the state and its corresponding deployment device into the state-level fragmentation candidate set. The model is optimized through a global cost model to form an executable state-level sharding scheme, enabling the model to support long sequence inference, multi-user concurrency, and cross-platform expert scheduling.

[0056] In S3, the global cost model is constructed by the device state representation output by the adaptive computing power perception module, which is a unified, differentiable, and optimizable multi-objective cost function. This function is related to physical constraints and performance costs, making different devices and different granularities comparable in the same cost space, thereby supporting the subsequent differentiable state space search strategy.

[0057] The core inputs provided by the adaptive perception module include: memory availability. Idle computing power ratio Bandwidth normalization Delayed normalization thermal factors Effective computing power of equipment and the future secure computing power predicted by the μST model These parameters provide information about the device's computational availability, communication availability, and thermal-power status, enabling cost modeling to accurately reflect the execution costs of operator / subgraph / tensor / state-level partitioning schemes across different devices and time intervals.

[0058] The global cost model construction method specifically includes: The overall deployment plan is represented as a state vector: in, This indicates a partitioning scheme that includes four granularity partitioning levels; among them, Any partition in the data belongs to the multi-granularity partitioning candidate space. ; It refers to the number of shards included in the deployment plan; vector It is a visual representation of the deployment location of each shard in the heterogeneous set of cloud, edge, and terminal devices; element For the first The device number assigned to each slice. A collection of heterogeneous devices; This joint state vector can express the fragmentation method and deployment method.

[0059] Since the actual operation of the model requires multiple shards to execute in parallel on different devices, the total cost is defined as the sum of the cost of each shard on its corresponding device and the communication cost between them: in, It is the number of fragments; For deployment plan An execution graph represents the interaction dependencies between devices; Indicates the first slice Deployed equipment and the slice Deployed equipment Data exchange exists, including but not limited to the distribution of shards and the exchange of computation results.

[0060] The first item represents the execution cost of all slices on their deployment device; The second item represents the cost of cross-device communication between slices.

[0061] The cost of local execution depends on the ratio of fragmented computation to the device's secure computing power, and is defined by combining the heat factor and memory pressure as follows: in, , , These are the weighting coefficients. Indicates fragmentation The computational complexity of floating-point operations; Indicates fragmentation The peak video memory usage required during execution is calculated from the fragmented tensor structure; This represents a short-term prediction of future secure computing power from the μST model; The first item describes the latency of piecewise computation; The second item is based on heat factors. Estimate the risk of frequency reduction due to temperature in the future; The third item is based on memory availability. Describe memory pressure.

[0062] All parameters come from the adaptive perception module, ensuring that cost modeling does not depend on external hardware information.

[0063] The communication cost is modeled based on the amount of data exchanged between the two shards in the dependency graph, device bandwidth, and end-to-end latency as follows: in, , These are the weighting coefficients; Indicates equipment and equipment In between, transmit information about fragments The amount of data includes the fragmented model tensors and the fragmented output data.

[0064] The global cost model enables unified comparison of different sharding and deployment methods, thereby obtaining a truly globally optimal combination, rather than relying on local heuristics for individual devices. In complex large-scale model structures, the joint state space expands exponentially. Therefore, this invention proposes a composite search algorithm combining the differentiable Bandit and GFlowNet to find the total cost in this high-dimensional space. The minimum state.

[0065] During the search process, the differentiable bandit plays the role of single-step decision-making, and its goal is to process the fragments. The problem involves the local selection of which device to place the fragments on. The basic principle is to "distribute the fragments..." Distributed to equipment "Consider it as an action in a multi-armed bandit, and use the negative of the local execution cost as the action value:" in, This indicates the current search iteration round.

[0066] The value of an action is set so that the lower the cost, the higher the benefit.

[0067] Bandit utilizes a differentiable Upper Confidence Bound (UCB) strategy in each iteration to partition the candidate space into multiple granularities. Select actions (assignable devices) to find the most valuable actions (assignable devices): in, No. The first fragment in the... The equipment selection results for the wheel; As of the date Before the round, the shards will be divided. Place in the device Number of attempts; This indicates an exploration of regulatory factors.

[0068] The second term in parentheses represents the confidence interval, ensuring that the search explores devices that have not been fully explored, thus avoiding getting trapped in local optima.

[0069] Bandit takes the piecewise local cost matrix as input and outputs the set of candidate devices for each piece, thus forming a set of state vectors. The local expansion direction of the joint state space.

[0070] However, Bandit alone cannot handle the entire joint state space because it is a local action selector, essentially employing a greedy exploration strategy rather than a global search. Therefore, this invention introduces GFlowNet (Generative Flow Network) to generate the probability distribution of the entire joint state space, ensuring that the final sampled complete deployment state... The probability of reaching a joint state is proportional to its termination reward. Therefore, this invention defines the reward of a joint state as a negative exponent of the total cost: And establish flow conservation constraints between all states: in, The flow of state transitions is a learnable parameter of GFlowNet; From state Transition to state The flow rate value.

[0071] The principle of flow conservation means that the "inflow" to a certain state must equal its "outflow plus reward." Therefore, the distribution generated by the network in a steady state satisfies: That is, the lower the total cost, the higher the sampling probability. The goal of the search is to minimize the cost, which is represented by maximizing the reward in the probability space. GFlowNet achieves this precisely through probabilistic modeling.

[0072] The differentiability of the algorithms is reflected in the fact that both Bandit and GFlowNet directly use... Either its local shape or its shape can be used as an optimization signal, and the parameters of both can be updated through gradient descent.

[0073] In practice, the search algorithm uses the Bandit output as the entry point for expanding the GFlowNet state to narrow its search range, and then constructs the complete state layer by layer through the flow network. After completing one GFlowNet state sampling, the system calculates the total cost. This is used to update parameters in two directions: first, as a termination reward to update flow conservation constraints, affecting flow distribution; and second, as a local cost signal for the action value of Bandit. Differentiable updates are performed. With multiple iterations, Bandit becomes increasingly accurate in selecting low-cost devices, and GFlowNet gradually samples optimal deployment schemes across the entire state space. Finally, when the total cost decreases and stabilizes, the system outputs the joint sharding-deployment combination with the minimum total cost: The results also confirmed the four-granularity segmentation method. Deployment vectors of all shards This enables globally optimal deployment across devices and platforms.

[0074] S4 specifically includes: For the sharding scheme A compatibility check is performed, taking the list of operators and tensor dimension structure within the fragment as input and outputting a set of intermediate representations (IRs) that can successfully generate low-level executable code on the corresponding device. During this check, the system determines whether the fragment can be directly compiled based on the target device's instruction set, supported tensor shapes, and available mathematical libraries (such as cuBLAS, TensorRT, ACL, etc.). If an operator pattern not supported by the target device appears in the fragment, the system will rewrite the operator into an equivalent operator sequence through an operator substitution table and update the fragment's internal execution graph to satisfy the target device's architectural constraints.

[0075] After compilability verification, the system applies a compiler selection strategy to different slices based on the device type. For GPU devices, a custom kernel is generated using CUDA Graph, TVM, TensorRT, or PyTorch Inductor; for CPU devices, vectorized operator kernel functions are generated using the OneDNN or XLA compiler; for NPU / AI acceleration chips, the system converts the slices into an execution format adapted to NNEngine, Ascend Graph, or other hardware backends. Specifically, the compiler automatically generates row slices, column slices, or block slices based on tensor-level slices, and automatically generates corresponding index mapping logic for state-level slices to ensure efficient reading of local states at runtime. After completing the fragmented compilation, the system needs to construct a runtime execution graph for cross-device loading and communication. This execution graph uses device allocation vectors. The input is a directed acyclic graph containing all execution nodes (computation nodes) and communication nodes (transmission nodes). Each edge in the diagram depends on the communication path between devices in the sharding scheme. The system uses bandwidth normalization. Delayed normalization quantity The optimal communication protocol is selected, including RPC, gRPC, RDMA, NCCL, OneCCL, or local shared memory. Specifically, for tensor-level slices, the system automatically configures AllGather, ReduceScatter, or SendRecv mechanisms at runtime; for state-level slices, the system establishes independent communication queues for different sequence slices to avoid blocking caused by buffered transmissions.

[0076] Subsequently, the system sends the model weights, sharding parameters, and runtime status to the corresponding devices according to the deployment plan. For operator-level and subgraph-level sharding, the weights are directly copied to the target device according to the compile-time IR mapping; for tensor-level sharding, the system slices the weight matrix according to the sharding method before sending it; for state-level sharding, the system adopts an incremental sending mechanism, that is, only sending the part required for each round of inference, in order to reduce bandwidth consumption. After all weights and states have been sent, each device is bound to the runtime cross-device execution graph using a unified execution entry pointer.

[0077] Once all preparations are complete, the system distributes the execution graph to the device-side executor to initiate the inference process. During inference, the executor runs fragments locally according to the execution graph sequence, while simultaneously exchanging intermediate activations, key-value states, or expert weights with other devices via communication nodes. (The last sentence appears to be incomplete and possibly refers to a different topic.) Bandwidth usage Or future secure computing power When significant changes occur, the system runs an online adaptive correction mechanism, which avoids prolonged performance degradation or execution failures by micro-recompiling or migrating some shards. The correction process achieves fine-grained drift at runtime, adjusting the position and scheduling of shards without stopping the entire inference process.

[0078] Ultimately, once the entire deployment process is complete, each shard runs on its most suitable hardware environment, cross-device communication channels are stably established, execution graphs are smoothly scheduled, and collaborative inference across cloud, edge, and device is achieved. The compilation and deployment method described above in this invention is fully implementable and can be directly implemented using existing deep learning compilers, communication libraries, and instruction sets of common hardware platforms.

[0079] This invention provides a multi-platform large-model dynamic sharding computing system based on computing power self-awareness. The computing system includes: a computing power self-awareness module, a multi-granularity sharding module, a sharding scheme generation module, and a multi-platform deployment module. The computing power self-sensing module is used to sense, quantify, and predict the effective available computing power of each cloud, edge, and terminal device in real time, providing core basis for model sharding, dynamic scheduling, and deployment decisions. The multi-granularity partitioning module is used to construct a four-level partitioning space: operator level, subgraph level, tensor level, and state level; the partitioning results together constitute the multi-granularity partitioning candidate space (MGPS). in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. Subsequent sharding schemes will be searched in the multi-granularity sharding candidate space; the sharding state space describes not only feasible sharding nodes, but also the devices on which the nodes are deployed; The sharding scheme generation module utilizes the results of computing power self-awareness and the multi-granularity sharding candidate space constructed by the preceding modules. By constructing a global cost model and innovating a state-space algorithm, the combination of partitions with the minimum total cost is obtained. ; The multi-platform deployment module is used to compile and optimize the four types of model granularity according to the hardware architecture characteristics of their target devices, and to build a cross-device execution graph, so that each shard can run collaboratively between heterogeneous terminals, edge nodes and cloud servers. This deployment phase needs to ensure that each shard is compilable, cross-device call relationships can be established, parameters and status can be correctly delivered, communication latency is controllable, and hardware status can be continuously monitored during execution for adaptive adjustments.

[0080] In summary, compared with the prior art, the present invention has the following beneficial effects: In this embodiment of the invention, the computation method, due to bandwidth normalization, temperature factor, and a complete computing power self-awareness mechanism, can maintain relatively stable execution performance under network fluctuations, temperature increases, and load spikes, reducing inference latency jitter. μST's computing power prediction capability allows the system to proactively adjust sharding before computing power declines, thereby avoiding passive redeployment and reducing the overhead of re-sharding and migration, making it more efficient than schemes relying on simple latency estimation. The multi-granularity sharding system supports various structures such as CNN, Transformer, MoE, and LLM, effectively improving the deployability of models, especially suitable for resource-constrained devices. State-level sharding solves the problem of LLM key-value cache (KV-cache) bloat, enabling devices to support longer inference sequences and higher concurrent users, improving long sequence throughput. The combined search method of differentiable Bandit and GFlowNet can quickly find globally suboptimal or even optimal sharding strategies in complex search spaces, reducing exploration time and avoiding the local optimum problem common in heuristic methods. The system can be scaled to more than ten cloud-edge-device collaborative devices, significantly improving scalability.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic sharding computation method for large multi-platform models based on self-aware computing power, characterized in that, The calculation method includes: S1. Computing power self-sensing Real-time sensing, quantification, and prediction of the effective available computing power of cloud, edge, and terminal devices; S2. Multi-granularity fragmentation A four-level partitioning space is constructed: operator-level, subgraph-level, tensor-level, and state-level; the partitioning results together constitute a multi-granularity partitioning candidate space. in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. S3. Sharding Scheme Generation The results of self-aware computing power and the multi-granularity partitioning candidate space constructed using preceding modules A global cost model is constructed, and a composite search algorithm combining the jointly differentiable Bandit and GFlowNet is used to search in the multi-granularity partition candidate space to obtain the partition combination with the minimum total cost. ; S4. Multi-platform deployment The model is divided into four granularities. Based on the hardware architecture characteristics of the target device, compilability verification and compiler selection are performed, and a cross-device execution graph is constructed. Then, the shards are distributed and inference is started. During the execution process, small-scale rescheduling is performed based on the device status.

2. The multi-platform large-scale model dynamic sharding calculation method based on self-aware computing power as described in claim 1, characterized in that, S1 specifically includes: S1.

1. Original Normalized Quantity Normalize the different dimensional data sensed by the device to ; Memory availability: in, for time Remaining video memory / RAM on the device; for Total video memory / memory of the device; Idle computing power ratio: in, for time The computing power utilization rate of the equipment ; Bandwidth normalization: in, for Time device With equipment Available bandwidth between; This is the preset upper limit of the bandwidth; Delayed normalization: in, for Time device To the equipment Single-hop delay; This is the upper limit for tolerance of delays; Heat factor: in, for time The temperature of the equipment; for The temperature threshold of the equipment; for The maximum allowable temperature of the equipment; S1.

2. Calculation of Effective Computing Power of Equipment Smoothly calculate the effective computing power of multiple platform devices; Effective computing power of the device: in, For equipment Peak computing power of hardware: in, For equipment Number of parallel computing units in the medium; For equipment The clock frequency at which the computing unit executes one hardware cycle; For equipment The number of FLOPs that can be executed per unit per cycle; EMA smoothing: in, Computing power after historical smoothing; This is the smoothing coefficient in the exponential moving average; S1.

3. Prediction of future computing power of equipment Using the μST architecture, for the future Use the computing power of the device for short-term prediction within a sampling period; Given input: Given K, construct the history sequence: Data preprocessing: A dual-path embedding approach is adopted, employing both periodic and bursty methods. For periodic paths, perform a Fast Fourier Transform (FFT) on the historical sequence: Periodic features are captured through a linear layer, and the frequency domain features are projected back into the time domain: in, It is a learnable projection matrix; It is a bias; For sudden paths, perform a first-order difference on the historical sequence. For any... The corresponding first-order difference is: The burst pattern shape is captured through local convolution, and the burst features are projected back to the temporal domain: The features extracted from the two paths are concatenated to obtain the initial representation: μST block: A mechanism of parallel global and local attention is adopted; For global attention blocks, the initial representation Transform from the time domain to the frequency domain: Attention in the frequency domain: in, For frequency domain lookup matrix; It is a frequency domain key matrix; It is a frequency domain value matrix; This is the scaling factor; Transform the sequence features in the frequency domain into the time domain: For a local attention block, let the window size be... The sequence features after local attention are: It consists of K+1 vectors generated by performing local attention on each of the K+1 input time steps; Where K represents the length of the historical sequence; L is the size of the local sliding window; No. At each time point, the local attention mechanism is executed to generate vectors. The formula is as follows: in, For the first Query vectors at each time point; For the first Key vectors at neighboring time points; This is a value vector of neighborhood time points; The feature sequences obtained by merging local attention and frequency attention: in, Indicates residual connection; Lightweight decoder: Through a multilayer perceptron, the features provided by the μST block are mapped to the future. One predicted value: Uncertainty estimation: The estimated computing power is conservatively reduced by estimating the model accuracy. Calculate the standard deviation using multiple samples from Dropout. To evaluate the accuracy of the model's predictions: in, This indicates the total number of Dropout samples; Based on the uncertainty, a conservative estimate is then made of the prediction results: in, This is a conservative coefficient.

3. The multi-platform large-scale model dynamic sharding calculation method based on self-aware computing power as described in claim 1, characterized in that, The specific process of operator-level partitioning includes: The model graph is analyzed by extracting each operator node to form a set of operator nodes; the input is the static representation of the model, and the output is a directed acyclic graph of operators. in, Represents a set of operators; Represents directed edges between operators based on tensor data dependencies; For each operator node, the system records its input tensor dimension, output tensor dimension, parameter size, type of mathematical operation used, and execution attributes; After constructing the operator graph, the system determines whether each operator node can serve as a potential partition boundary. Operator node Whether a node can be used as a sharding point depends on several feasibility conditions, including: topological feasibility, transmission feasibility, memory feasibility, and computing power feasibility; the input is the operator graph and the attribute information of each operator node, and the output is a set of operator-level sharding candidates. ; Define the topology feasibility judgment formula: Operator The feasibility of cross-device access is determined as follows: The size of the operator output tensor is , Time device With equipment The available bandwidth between is Single-hop delay is The maximum acceptable transmission time is ; Operator Can be deployed on devices The memory feasibility is defined as: in, This represents the peak video memory usage required by the operator during execution. for time The remaining video memory of the device; In the equipment The feasibility estimate for the available computing power is as follows: in, This indicates the number of floating-point operations required for the operator. For equipment Conservative computing power estimate, This indicates the maximum acceptable execution time specified by the system. For operator theory execution time; Based on the above four conditions, the overall feasibility judgment condition for operator nodes is defined as follows: Operator nodes that meet this condition and their corresponding deployment devices will be added to the operator-level sharding candidate set. .

4. The multi-platform large-model dynamic sharding calculation method based on self-aware computing power as described in claim 3, characterized in that, The specific process of subgraph-level partitioning includes: The subgraph pattern recognition method is used to find substructures that satisfy the feature template in the operator graph. These structures are obtained by graph pattern matching and strongly connected component analysis. The set of operator nodes contained in each subgraph and its internal data flow are recorded. After obtaining the candidate set of subgraphs, the subgraphs are... Feasibility must be determined; Subgraph The feasibility of cross-device access transmission is determined as follows: in, This is a bandwidth normalization parameter; For subgraph The size of the output tensor; Can be deployed on devices The memory feasibility is denoted as: Among them, the peak memory usage of the sub-image is ; In the equipment The feasibility estimate for the available computing power is as follows: Among them, subgraph The total number of floating-point calculations is: The execution time of operator theory is: Based on the above conditions, the overall feasibility conditions for subgraph-level fragmentation in this invention are defined as follows: Topological feasibility This ensures that the selected subgraph does not disrupt the directed acyclic structure of the computation graph; if the overall feasibility condition is met, the subgraph and its corresponding deployment device are included in the subgraph set fragment candidate set. .

5. The multi-platform large-scale model dynamic sharding calculation method based on self-aware computing power as described in claim 4, characterized in that, The specific process of tensor-level fragmentation includes: Identify all large tensors that satisfy the condition. This is used as a candidate tensor for tensor-level partitioning; among which A threshold for the size of the tensor given by the user; Perform large tensor row partitioning, column partitioning, and block partitioning; Next, tensor partitioning is performed. Feasibility assessment; Tensor partitioning The feasibility of cross-device access transmission is determined as follows: in, For subgraph The size of the output tensor; Can be deployed on devices The memory feasibility was determined as follows: in, tensor partitioning Peak video memory usage; In the equipment The feasibility of running on the computing power is determined as follows: in, Indicates the execution time of floating-point operations involved in the tensor block; Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: Based on the above determination, the tensor fragments that meet the conditions and their corresponding deployment devices are added to the tensor fragmentation candidate set. .

6. The multi-platform large-model dynamic sharding calculation method based on self-aware computing power as described in claim 5, characterized in that, The specific process of state-level sharding includes: The state-level partitioning can be partitioned by header, by sequence, or by batch dimension. Next, proceed with the state fragment. Feasibility assessment; State fragment The formula for determining the feasibility of cross-device access transmission is: in, State fragment Storage capacity; State fragment Can be deployed on devices The formula for determining memory feasibility is: in, State fragment Peak memory usage Then it is in the equipment The feasibility of running on the computing power is determined as follows: Among them, state fragments The computational cost of associated floating-point operations is Execution time is ; Based on the above conditions, the overall feasibility conditions for tensor-level piecewise partitioning are defined as follows: When the above constraints are satisfied, the state and its corresponding deployment device will be included in the state-level fragmentation candidate set. .

7. The multi-platform large-model dynamic sharding computation method based on self-aware computing power as described in claim 1, characterized in that, In S3, the global cost model construction method specifically includes: The overall deployment plan is represented as a state vector: in, This indicates a segmentation scheme that includes four granularity segmentations; Any partition in the data belongs to the multi-granularity partitioning candidate space. ; It refers to the number of shards included in the deployment plan; vector It is a visual representation of the deployment location of each shard in the heterogeneous set of cloud, edge, and terminal devices; element For the first The device number assigned to each slice. A collection of heterogeneous devices; The total cost is defined as the sum of the cost of each fragment on its corresponding device and the communication cost between them: in, It is the number of fragments; For deployment plan Execution diagram; Indicates the first slice Deployed equipment and the slice Deployed equipment Data exchange exists; The cost of local execution is defined as: in, , , These are the weighting coefficients. Indicates fragmentation The computational complexity of floating-point operations; Indicates fragmentation Peak video memory usage required during execution; This represents a short-term prediction of future secure computing power from the μST model; The communication cost, based on the dependency graph and device bandwidth, is as follows: in, , These are the weighting coefficients; Indicates equipment and equipment In between, transmit information about fragments The amount of data.

8. The multi-platform large-scale model dynamic sharding calculation method based on self-aware computing power as described in claim 7, characterized in that, In S3, a composite search algorithm combining differentiable Bandit and GFlowNet is used to find the total cost in this high-dimensional space. The minimum state; First, perform a differentiable bandit decision: Define "deploy shards to devices" as an action, and use the negative value of the local execution cost as the action value: in, Indicates the current search iteration round; Bandit utilizes a differentiable UCB strategy in each iteration to partition the candidate space at multiple granularities. Select action: in, No. The first fragment in the... The equipment selection results for the wheel; As of the date Before the round, the shards will be divided. Place in the device Number of attempts; Indicates the exploration of regulatory factors; Bandit takes the local cost matrix of each shard as input and outputs a set of candidate devices for each shard. Next, the candidate device set for each slice is locally expanded into a series of joint states. ; Next, a GFlowNet search is performed in the space of joint states: The reward for the joint state is defined as a negative exponent of the total cost: And establish flow conservation constraints between all states: in, The flow of state transitions is a learnable parameter of GFlowNet; From state Transition to state The flow rate value; Parameter update process: After completing one GFlowNet state sampling, the system calculates the total cost. This is used to update parameters in two directions: first, as a termination reward to update flow conservation constraints, affecting flow distribution; and second, as a local cost signal for the action value of Bandit. Perform differentiable updates; When the total cost decreases and tends to stabilize, the system outputs the joint sharding-deployment combination with the minimum total cost: The results also confirmed the four-granularity segmentation method. Deployment vectors of all shards .

9. The multi-platform large-scale model dynamic sharding calculation method based on self-aware computing power as described in claim 1, characterized in that, S4 specifically includes: Deployment plan A compatibility check is performed, taking the list of operators and tensor dimension structure inside the slice as input and outputting a set of intermediate representations that can successfully generate low-level executable code on the corresponding device. During this check, the system determines whether the slice can be directly compiled based on the instruction set, supported tensor shapes, and available mathematical libraries of the target device. If the slice contains operator patterns that are not supported by the target device, the system will rewrite the operators into equivalent operator sequences through an operator substitution table and update the internal execution graph of the slice to meet the architectural constraints of the target device. After compilability verification is completed, the system applies a compiler selection strategy to different fragments based on the device type; for tensor-level fragments, the compiler automatically generates row slices, column slices, or block slices, and for state-level fragments, it automatically generates the corresponding index mapping logic. After completing the fragmented compilation, the system needs to construct a runtime execution graph for cross-device loading and communication; this execution graph uses device allocation vectors. The input is a directed acyclic graph containing all execution nodes and communication nodes. The system uses bandwidth normalization. Delayed normalization quantity Select the optimal communication protocol; among which, establish an independent communication queue for state-level slices; Subsequently, the system sends the model weights, sharding parameters, and runtime status to the corresponding devices according to the deployment plan. For operator-level and subgraph-level sharding, the weights are directly copied to the target device according to the intermediate representation mapping at compile time. For tensor-level sharding, the system performs sharding on the weight matrix according to the sharding method before sending it out. For state-level sharding, the system adopts an incremental sending mechanism, that is, only the part required for each round of inference is sent out. After all weights and statuses are sent out, each device is bound to the cross-device execution graph at runtime using a unified execution entry pointer. Once all preparations are complete, the system distributes the execution graph to the device-side executors to initiate the inference process. During inference, the executors run shards locally according to the execution graph sequence, while simultaneously exchanging intermediate activations, key-value states, or expert weights with other devices through communication nodes. When the device's heat factor... Bandwidth usage Or future secure computing power When significant changes occur, the system runs an online adaptive correction mechanism, which involves micro-recompiling or migrating some fragments.

10. A multi-platform large-scale model dynamic sharding computing system based on self-aware computing power, characterized in that, The computing system includes: a computing power self-sensing module, a multi-granularity sharding module, a sharding scheme generation module, and a multi-platform deployment module; The computing power self-sensing module is used to sense, quantify, and predict the effective available computing power of each cloud, edge, and terminal device in real time, providing core basis for model sharding, dynamic scheduling, and deployment decisions. The multi-granularity partitioning module is used to construct a four-level partitioning space: operator-level, subgraph-level, tensor-level, and state-level; the partitioning results collectively constitute the multi-granularity partitioning candidate space. in, This is the set of candidate slices for operator-level partitioning. This is the candidate set for subgraph-level partitioning. For tensor-level slice candidate set; This is a candidate set for state-level partitioning. The sharding scheme generation module utilizes the results of computing power self-awareness and the multi-granularity sharding candidate space constructed by the preceding modules. By constructing a global cost model and innovating a state-space algorithm, the combination of partitions with the minimum total cost is obtained. ; The multi-platform deployment module is used to compile and optimize the four types of model granularity according to the hardware architecture characteristics of their target devices, and to build a cross-device execution graph, so that each shard can run collaboratively between heterogeneous terminals, edge nodes and cloud servers. This deployment phase needs to ensure that each shard is compilable, cross-device call relationships can be established, parameters and status can be correctly delivered, and that the hardware status can be continuously monitored during execution for adaptive adjustments.

Images