A large model lightweight deployment method supporting dynamic hardware adaptation

Abstract

The application provides a large model lightweight deployment method supporting dynamic hardware adaptation, comprising constructing a runtime adaptive system with closed-loop perception and decision-making capability, taking a hardware-independent intermediate representation layer UDL as a technical cornerstone, and realizing dynamic lightweight deployment of a large model on an edge device through a complete closed-loop process of monitoring-decision-reconstruction-compilation-execution-feedback, specifically comprising model unified conversion, runtime state perception, reconstruction decision generation, model lightweight reconstruction, hardware adaptation code generation, optimized code execution, and performance feedback optimization.The application has the beneficial effects that an adaptive system based on a unified intermediate representation and having runtime perception and reconstruction capability can be constructed, dynamic lightweight deployment of a large model on an edge device can be realized, real-time deep collaborative adaptation of a model and hardware resources of an edge device can be realized, and resource utilization, system robustness and service reliability can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence computing technology, and in particular to a method for lightweight deployment of large models that supports dynamic hardware adaptation. Background Technology

[0002] As artificial intelligence technology extends from the cloud to the edge, the demand for low-latency, high-energy-efficiency intelligent processing in edge scenarios such as industrial IoT, autonomous driving, and mobile devices is becoming increasingly urgent. Deploying large-scale pre-trained models (LLMs) with massive parameters to edge devices has become a key to industry development. However, edge devices face strict constraints in computing, memory, and energy consumption, diverse hardware architectures (CPU, GPU, NPU, etc.), and dynamic fluctuations in available resources (remaining memory, CPU load, network bandwidth, etc.) during runtime. Therefore, both academia and industry generally believe that achieving a balance between accuracy and efficiency, overcoming hardware fragmentation, and adapting to dynamic environments are the core challenges that must be overcome in edge AI deployment.

[0003] Existing technologies primarily attempt to address the aforementioned problems through model compression and resource scheduling, but both have significant limitations. Model compression and static lightweighting techniques mainly include: pruning, removing redundant weights or neurons from the model; quantization, converting model weights and / or activation values ​​from high precision (e.g., FP32) to low precision (e.g., INT8, INT4) to significantly reduce storage and computational overhead; and knowledge distillation, training a lightweight "student model" to mimic a large "teacher model." However, these model compression and static lightweighting techniques (pruning, quantization, knowledge distillation) are offline static optimization processes. After model deployment, the structure and computational precision are fixed, making it unable to respond to dynamic changes in runtime resources. This can easily lead to service failures due to insufficient resources or underutilization of computing power when resources are plentiful.

[0004] To address the dynamic nature of edge environments, some research attempts to optimize task scheduling. Computation offloading and collaborative inference offload some computational tasks from terminal devices to edge servers or the cloud to alleviate terminal pressure. More refined model splitting techniques, such as the DynaSplit framework, attempt to dynamically find the optimal splitting points for neural networks, computing the first few layers of the model at the edge and the later layers in the cloud. Dynamic resource allocation algorithms utilize reinforcement learning (such as deep deterministic policy gradient algorithms) to make online decisions on task offloading strategies and resource allocation to optimize latency and energy consumption. However, these dynamic resource scheduling and collaborative computing techniques (computation offloading, model splitting, and dynamic resource allocation) only focus on scheduling tasks or data streams between devices, failing to achieve adaptive model structure and unable to reconstruct the model computation graph online at a fine-grained level based on real-time resources. They only solve the problem of "where to compute," not the core issue of "what form of model to use for computation based on current resources."

[0005] The closest existing technology is a combination of the adaptive model compression framework of federated learning and the dynamic partitioning framework of hardware-software co-design, but it still has shortcomings: the former adjusts the model during the training phase and is a static model after deployment, which cannot respond to resource fluctuations during inference; the latter operates on fixed model blocks and external hardware configurations and does not touch the runtime real-time reconstruction of the model's internal computation graph.

[0006] In summary, existing technologies suffer from drawbacks such as mismatch between static and dynamic environments, single optimization dimensions, and lack of coordination. There is an urgent need for an integrated method that can dynamically trigger and complete the lightweight reconstruction of large models and hardware adaptation compilation based on real-time resource status when running on edge devices, so as to achieve the optimal balance of performance, efficiency, and stability in resource-constrained environments. Summary of the Invention

[0007] To address the problems in existing technologies, this invention provides a lightweight deployment method for large models that supports dynamic hardware adaptation. Based on a hardware-independent intermediate representation layer (UDL), it constructs an adaptive system with runtime awareness and reconfiguration capabilities through a complete closed-loop process of monitoring-decision-reconstruction-compilation-execution-feedback. This enables the dynamic lightweight deployment of large models on edge devices, achieving real-time deep collaborative adaptation between the model and edge device hardware resources. This improves resource utilization, system robustness, and service reliability, while reducing operational costs. It solves the problems of static deployment of large models, cumbersome hardware adaptation, and inability to respond to runtime environment changes in existing edge computing scenarios.

[0008] This invention provides a method for lightweight deployment of large models that supports dynamic hardware adaptation. It constructs a runtime adaptive system with closed-loop perception and decision-making capabilities, using a hardware-independent intermediate representation layer (UDL) as its technological foundation. Through a complete closed-loop process of monitoring-decision-reconstruction-compilation-execution-feedback, it achieves dynamic lightweight deployment of large models on edge devices. Specifically, it includes the following steps:

[0009] Step 1: Unified model conversion. The standard large model received from the cloud is converted into a hardware-independent intermediate representation layer UDL computation graph composed of atomic operators through unified compilation and presentation layer.

[0010] Step 2, runtime state awareness: The dynamic resource status of edge devices is collected through the runtime dynamic monitoring and decision engine, and the static characteristics of the device hardware are analyzed to form a hardware feature description file.

[0011] Step 3, Reconstruction Decision Generation: The lightweight decision model generates reconstruction decision instructions that include target accuracy, operator fusion strategy, and model segmentation points based on dynamic resource status and hardware static characteristics.

[0012] Step 4, Model Lightweight Reconstruction: The Model Lightweight Reconstruction module performs online fine-grained structural transformation on the intermediate representation layer UDL computation graph according to the reconstruction decision instructions.

[0013] Step 5: Hardware adaptation code generation. The multi-backend adaptive code generator combines the reconstructed intermediate representation layer UDL computation graph with the hardware feature description file to generate optimized target code adapted to the current hardware.

[0014] Step 6: Optimize code execution. The execution engine loads and runs the optimized target code to complete the large model inference task.

[0015] Step 7, Performance Feedback Optimization: The performance and accuracy monitor collects performance data of the inference task and feeds it back to the lightweight decision model to optimize the subsequent reconstruction decision strategy and form an adaptive closed loop.

[0016] The present invention is further improved in that, in step 2, the dynamic resource status includes CPU / GPU / NPU utilization, cache utilization at each level, remaining available memory, network round-trip latency, and available bandwidth, which are collected in real time by the resource monitor at a millisecond frequency; the hardware static features include instruction set architecture, memory hierarchy, and parallel thread scale, which are analyzed and obtained by the hardware feature extractor when the device is initialized or when new hardware is detected.

[0017] In a further improvement, in step 3, the lightweight decision model determines whether to trigger model reconstruction based on preset heuristic rules or online training reinforcement learning strategies; the reconstruction decision instruction also includes the desired computational granularity.

[0018] In a further improvement to this invention, in step 3, the decision strategy of the lightweight decision model can be replaced by a multi-layer conditional judgment rule tree, or a reconstructed strategy library can be generated in advance through simulation and then online query and matching can be performed based on the device status during runtime.

[0019] The present invention is further improved in that, in step 4, the online fine-grained structure transformation includes one or more of dynamic precision quantization, network-aware operator segmentation, and adaptive subgraph pruning; dynamic precision quantization is to convert the weights and activation values ​​of specified operators or subgraphs in the UDL computation graph online between FP16, INT8, and INT4, and to minimize precision loss by combining dynamic calibration; network-aware operator segmentation is to dynamically select the optimal segmentation point in the UDL computation graph based on real-time network latency data, and to intelligently compress or block the output feature maps of large operators; adaptive subgraph pruning is to dynamically skip redundant branches or operators in the UDL computation graph that have low contribution to the current inference task.

[0020] In a further improvement, in step 4, the granularity of the lightweight model reconstruction operation can be replaced by using a predefined computational subgraph as the unit to uniformly switch or prune the overall units of the Transformer attention module.

[0021] In a further improvement, in step 5, the multi-backend adaptive code generator calls a pre-generated hardware adaptation rule library. The hardware adaptation rule library maps hardware static features to specific optimization rules. The multi-backend adaptive code generator generates target code through just-in-time compilation, generating PTX and CUDA kernels for NVIDIA GPUs and CANN operators for Huawei Ascend NPUs.

[0022] In a further improvement, in step 5, the implementation of the hardware adaptation rule base can be replaced by embedding a micro-cost model in the compiler, generating near-optimal hardware adaptation code through rapid search, thereby reducing the dependence on the rule base.

[0023] The present invention is further improved in that, in step 6, the execution engine uses a millisecond-level seamless hot-switching method to load new optimized target code, replace the original running kernel, and ensure that the business is unaware of it.

[0024] In a further improvement, in step 7, the performance data includes the execution time, power consumption, and output accuracy of the inference task. The performance and accuracy monitor compares the performance data with the corresponding data of the cloud benchmark model to form a feedback signal.

[0025] The beneficial effects of this invention are as follows: This invention provides a lightweight deployment method for large models that supports dynamic hardware adaptation. By deeply integrating the UDL layer with a runtime dynamic monitoring and reconstruction engine, it endows large models with the characteristics of a "dynamic life form," constructing a perception-decision-reconstruction closed loop. This fundamentally solves the core contradiction of mismatch between static models and dynamic edge environments, achieving a paradigm shift from "static deployment" to dynamic adaptation. Fine-grained and precise optimization is performed at the UDL atomic operator level, dynamically adjusting the model form according to real-time resource status to achieve "on-demand supply," improving resource utilization by an average of over 30%. Simultaneously, cross-stack collaborative optimization of computation and communication is achieved through network-aware operator segmentation, reducing end-to-end latency by up to 40% in edge-cloud collaborative inference scenarios. When edge device resources fluctuate drastically, the system can trigger lightweight reconstruction in milliseconds, achieving uninterrupted service degradation and seamless switching to a high-precision model after resource recovery. This approach provides stable support for critical tasks with stringent continuity requirements, such as autonomous driving and industrial quality inspection. It transforms the hardware adaptation, model compression, and deployment optimization processes, which rely on expert experience, into a data-driven automated workflow. Through a built-in decision model, it autonomously learns optimal reconstruction strategies, significantly reducing reliance on senior AI system engineers and lowering the total cost of ownership for large-scale edge AI deployments. Based on the UDL layer, it decouples computational logic from hardware, supporting various heterogeneous hardware such as CPUs, GPUs, and NPUs. The hardware adaptation rule base has an extensible architecture, allowing for rapid adaptation to new hardware types. It enables dynamic, lightweight deployment of large models on edge devices, achieving real-time, deep collaborative adaptation between models and edge device hardware resources. This improves resource utilization, system robustness, and service reliability, solving the problems of static large model deployment, cumbersome hardware adaptation, and inability to respond to runtime environment changes in existing edge computing scenarios. Attached Figure Description

[0026] Figure 1 This is a flowchart of a method for lightweight deployment of large models that supports dynamic hardware adaptation, according to the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0028] Please see Figure 1 This invention provides a method for lightweight deployment of large models that supports dynamic hardware adaptation. It constructs a runtime adaptive system with closed-loop perception and decision-making capabilities, using a hardware-independent intermediate representation layer (UDL) as its technological foundation. Through a complete closed-loop process of monitoring-decision-reconstruction-compilation-execution-feedback, it achieves dynamic lightweight deployment of large models on edge devices. Specifically, it includes the following steps:

[0029] Step 1: Model Unified Conversion. Through unified compilation and presentation layer receiving the standard large model from the cloud, it is converted into a hardware-independent intermediate presentation layer UDL computation graph composed of atomic operators.

[0030] Step 2, Runtime State Awareness: The dynamic resource status of edge devices is collected through the runtime dynamic monitoring and decision engine, and the static characteristics of the device hardware are analyzed to form a hardware feature description file. The dynamic resource status includes CPU / GPU / NPU utilization, cache utilization at each level, remaining available memory, network round-trip latency, and available bandwidth, which are collected in real time by the resource monitor at a millisecond frequency. The hardware static characteristics include instruction set architecture, memory hierarchy, and parallel thread scale, which are analyzed and obtained by the hardware feature extractor when the device is initialized or when new hardware is detected.

[0031] Step 3, Reconstruction Decision Generation: The lightweight decision model generates reconstruction decision instructions based on dynamic resource status and hardware static characteristics. These instructions include target accuracy, operator fusion strategy, and model segmentation points. The lightweight decision model determines whether to trigger model reconstruction based on preset heuristic rules or online-trained reinforcement learning strategies. The reconstruction decision instructions also include the desired computational granularity. The decision strategy of the lightweight decision model can be replaced with a multi-layer conditional judgment rule tree, or a reconstruction strategy library can be pre-generated through simulation and matched online based on the device status at runtime.

[0032] Step 4, Model Lightweight Reconstruction: The model lightweight reconstruction module performs online fine-grained structural transformation on the intermediate representation layer UDL computation graph according to the reconstruction decision instructions. This online fine-grained structural transformation includes one or more of the following: dynamic precision quantization, network-aware operator segmentation, and adaptive subgraph pruning. Dynamic precision quantization involves online conversion of the weights and activation values ​​of specified operators or subgraphs in the UDL computation graph between FP16, INT8, and INT4, combined with dynamic calibration to minimize precision loss. Network-aware operator segmentation involves dynamically selecting the optimal segmentation point in the UDL computation graph based on real-time network latency data, and intelligently compressing or dividing the output feature maps of large operators. Adaptive subgraph pruning involves dynamically skipping redundant branches or operators in the UDL computation graph that contribute little to the current inference task. The granularity of the model lightweight reconstruction operation can be replaced by uniform precision switching or pruning of the entire Transformer attention module unit based on predefined computation subgraphs.

[0033] Step 5, Hardware Adaptation Code Generation: The multi-backend adaptive code generator combines the reconstructed intermediate representation layer UDL computation graph with the hardware feature description file to generate optimized target code adapted to the current hardware. Specifically, the multi-backend adaptive code generator calls a pre-generated hardware adaptation rule library, which maps static hardware features to specific optimization rules. The multi-backend adaptive code generator generates target code through just-in-time compilation, generating PTX and CUDA kernels for NVIDIA GPUs and CANN operators for Huawei Ascend NPUs. The implementation of the hardware adaptation rule library can be replaced by embedding a micro-cost model in the compiler, generating near-optimal hardware adaptation code through rapid search, reducing reliance on the rule library.

[0034] Step 6: Optimize code execution. The execution engine loads and runs the optimized target code to complete the large model inference task. The execution engine uses a millisecond-level seamless hot-switching method to load the new optimized target code, replacing the original running kernel, ensuring that the business is unaware of it.

[0035] Step 7, Performance Feedback Optimization: The performance and accuracy monitor collects performance data of the inference task and feeds it back to the lightweight decision model to optimize subsequent reconstruction decision strategies and form an adaptive closed loop. The performance data includes the execution time, power consumption, and output accuracy of the inference task. The performance and accuracy monitor compares the performance data with the corresponding data of the cloud benchmark model to generate feedback signals.

[0036] Please see Figure 1 As an embodiment of the present invention, the system's overall architecture and workflow are as follows: the system is deployed on an edge computing node and works in collaboration with a possible cloud-based training / management platform.

[0037] 1. The Unified Compilation and Representation Layer is the technological cornerstone of the system, inheriting and extending the achievements of previous patents. Its main function is to receive standard large models (such as PyTorch and TensorFlow formats) from the cloud and convert them into a hardware-independent intermediate representation—the Unified Operator Description Language (UDL) computation graph—through a front-end parser. The intermediate representation layer's UDL computation graph decomposes complex network layers (such as the attention layer and feedforward layer of the Transformer) into a series of well-defined atomic operators (such as matrix multiplication, convolution, and activation functions) and models the data flow dependencies of the operators. This process achieves the first decoupling between computational logic and hardware implementation.

[0038] 2. Runtime Dynamic Monitoring and Decision Engine: This is the core driving module of this invention. It consists of the following sub-modules: a resource monitor, which collects local hardware status in real time at millisecond frequency, including but not limited to CPU / GPU / NPU utilization, cache occupancy rates at all levels, remaining available memory, power limit status, and round-trip latency and available bandwidth of the current network connection; a hardware feature extractor, which actively analyzes static features such as the instruction set architecture (ISA), memory hierarchy, and parallel thread scale of the computing unit when the device is initialized or new hardware is detected, forming a hardware feature description file to provide a basis for code generation; and a lightweight decision model, which receives real-time data streams from the monitor. This decision model can be based on pre-set heuristic rules (e.g., "if available memory is below threshold A, then trigger quantization reconstruction to INT8") or an online-trained micro-reinforcement learning model. Its output is a reconstruction decision instruction, which includes whether to trigger reconstruction immediately, the target model accuracy (FP16 / INT8 / INT4), the operator fusion strategy, the model sharding point (for end-edge-cloud collaboration), and the desired computational granularity.

[0039] 3. The lightweight model reconstruction module acts as the "operating room" for dynamic adjustments. It receives instructions from the decision engine and the original intermediate representation layer UDL computation graph, performing rapid graph transformations in memory. Specific techniques include: dynamic precision quantization, which converts the weights and activation values ​​of specified operators or subgraphs online according to instructions. For example, it uses FP16 computation when resources are plentiful and seamlessly switches to INT8 or even INT4 when resources are scarce. This process can be combined with dynamic calibration to minimize precision loss; network-aware operator segmentation, which dynamically selects the optimal segmentation point in the intermediate representation layer UDL computation graph based on real-time network latency data when decision-making involves collaborative inference. It intelligently compresses or blocks the output feature graphs of large operators, optimizing the overlap between computation and communication and reducing end-to-end latency; and adaptive subgraph pruning, which dynamically skips redundant branches or operators in the intermediate representation layer UDL computation graph that contribute little to the current task, achieving instantaneous load adjustment based on real-time computing resources.

[0040] 4. Multi-backend adaptive code generator: This module is the ultimate guarantee of performance. It contains a scalable hardware adaptation rule base generated by previous patents. This rule base maps the static characteristics of various hardware types (such as memory hierarchy and thread bundle size) to specific optimization rules (such as loop block size and memory access merging strategy). The code generator receives the reconstructed intermediate representation layer UDL computation graph, combines it with the feature description of the currently active hardware, matches the optimal rule combination from the rule base, and calls the corresponding LLVM or hardware-specific compiler (such as NVCC) to compile and generate highly optimized machine code in just-in-time. For example, for NVIDIA GPUs, it generates PTX and CUDA kernels; for Huawei Ascend NPUs, it generates CANN operators. The generated code is loaded into the execution engine for execution immediately.

[0041] 5. Execution Engine and Feedback Loop: The execution engine is responsible for running the generated optimized code to complete the inference task. Simultaneously, a performance and accuracy monitor continuously collects the actual execution time, power consumption, and output accuracy (e.g., compared with a cloud-based benchmark model) for each inference task, and sends this data back to the decision engine as feedback signals to optimize subsequent decision-making strategies, forming a complete adaptive closed loop of "monitoring-decision-refactoring-execution-feedback".

[0042] In this embodiment, the overall system architecture and basic deployment process are essentially described. The runtime adaptive system of this invention is deployed on edge computing nodes and works in collaboration with the cloud training / management platform. The system includes six core modules: a unified compilation and representation layer, a runtime dynamic monitoring and decision engine, a lightweight model reconstruction module, a multi-backend adaptive code generator, an execution engine, and a performance and accuracy monitor. The specific implementation steps are as follows: Initial deployment: The cloud training / management platform distributes standard large models in PyTorch / TensorFlow format to the edge computing nodes. The unified compilation and representation layer converts the standard large models into atomic operators such as matrix multiplication, convolution, and activation functions through a front-end parser. The intermediate representation layer UDL computation graph is pre-compiled to generate target code adapted to the default hardware configuration of edge devices, completing the initial deployment of the system. Hardware feature extraction: During the device initialization phase, the hardware feature extractor of the runtime dynamic monitoring and decision engine analyzes the static features of the edge device's CPU / GPU / NPU instruction set architecture, memory hierarchy, parallel thread scale, etc., to form a hardware feature description file and store it in the hardware adaptation rule base. The monitoring module starts up, and the resource monitor collects data such as CPU / GPU / NPU utilization, cache occupancy rate at each level, remaining available memory, network round-trip latency, and available bandwidth of the edge device in real time at a frequency of 50 milliseconds, and continuously transmits it to the lightweight decision model.

[0043] Please see Figure 1As a second embodiment of the present invention, the dynamic adaptation process under network bandwidth fluctuations is described. In this embodiment, when the network bandwidth of the edge device suddenly drops (from 100Mbps to 10Mbps), the dynamic adaptation process of the present invention is as follows:

[0044] Decision triggering: The lightweight decision model receives network bandwidth data from the resource monitor and generates a reconstruction decision instruction based on the preset rule "when the network bandwidth is less than 20Mbps, trigger operator output compression and model segmentation". This instruction specifies feature compression of the operator nodes to be transmitted in the intermediate representation layer UDL computation graph and determines the end-edge collaborative model segmentation point.

[0045] Model reconstruction: The lightweight model reconstruction module receives reconstruction decision instructions, locates the affected operator nodes in the intermediate representation layer UDL computation graph, inserts a lightweight feature compression subgraph online, adjusts the input dimension of subsequent operators, and splits the intermediate representation layer UDL computation graph into edge-side computation subgraphs and cloud-side computation subgraphs according to the split point to complete the lightweight model reconstruction.

[0046] Hardware-adaptive compilation: The multi-backend adaptive code generator combines the reconstructed intermediate representation layer UDL computation graph with the hardware feature description file, matches optimization rules for network-constrained scenarios from the hardware adaptation rule base, calls the LLVM compiler for just-in-time compilation, and generates CUDA kernel code containing feature compression and model splitting logic.

[0047] Seamless hot switching: In the next batch of inference tasks, the execution engine loads the newly compiled CUDA kernel code at millisecond speeds and replaces the original kernel. The entire switching process is imperceptible to the business.

[0048] Inference execution: The execution engine runs the new kernel code, completes the inference of the edge-side computation subgraph, and transmits the compressed feature data to the cloud, where the cloud completes the inference of the remaining subgraph, realizing end-edge collaborative inference;

[0049] Feedback optimization involves a performance and accuracy monitor that collects data such as end-to-end latency, feature transmission time, and output accuracy for this collaborative inference. This data is then fed back to a lightweight decision model, which fine-tunes the feature compression intensity based on the feedback to optimize reconstruction decisions under subsequent network bandwidth fluctuation scenarios.

[0050] Please see Figure 1As a third embodiment of the present invention, the dynamic adaptation process under memory resource constraints is as follows: When the remaining available memory of the edge device falls below a preset threshold due to multiple concurrent applications, the dynamic adaptation process of the present invention is as follows: Decision triggering: The lightweight decision model generates a reconstruction decision instruction based on the memory data of the resource monitor, requiring the precision of all operators in the intermediate representation layer UDL computation graph to be dynamically converted from FP16 to INT8, and redundant operator branches with a contribution of less than 5% to the current inference task to be pruned; Model reconstruction: The lightweight model reconstruction module performs dynamic precision quantization on the intermediate representation layer UDL computation graph, converts the operator weights and activation values ​​from FP16 to INT8 and performs dynamic calibration, while skipping redundant operator branches and completing subgraph pruning; Hardware adaptation compilation: The multi-backend adaptive code generator matches the memory-optimized compilation rules for the INT8 precision UDL computation graph and generates lightweight target code; Execution and feedback: The execution engine loads the lightweight target code to complete the inference, and the performance and precision monitor feeds back data such as inference time, memory usage, and precision loss to the lightweight decision model, providing a basis for the precision rollback strategy after memory resources are restored.

[0051] As can be seen from the above, the beneficial effects of this invention are as follows: This invention provides a lightweight deployment method for large models that supports dynamic hardware adaptation. By deeply integrating the UDL layer and the runtime dynamic monitoring and reconstruction engine, it endows large models with the characteristics of a "dynamic life form," constructs a perception-decision-reconstruction closed loop, fundamentally solves the core contradiction of mismatch between static models and dynamic edge environments, and realizes a paradigm shift from "static deployment" to dynamic adaptation. Fine-grained and precise optimization is performed at the UDL atomic operator level, and the model form is dynamically adjusted according to the real-time resource status to achieve "on-demand supply," improving resource utilization by an average of more than 30%. At the same time, cross-stack collaborative optimization of computing and communication is achieved through network-aware operator segmentation, reducing end-to-end latency by up to 40% in edge-cloud collaborative inference scenarios. When edge device resources fluctuate drastically, the system can trigger lightweight reconstruction in milliseconds, achieving uninterrupted service degradation operation, and seamlessly switching to high-performance operation after resource recovery. The precision mode provides stable support for critical tasks with stringent continuity requirements, such as autonomous driving and industrial quality inspection. It transforms the hardware adaptation, model compression, and deployment optimization processes that rely on expert experience into data-driven automated processes. Through its built-in decision model, it autonomously learns the optimal reconstruction strategy, significantly reducing reliance on senior AI system engineers and lowering the total cost of ownership for large-scale edge AI deployments. Based on the UDL layer, it decouples computational logic from hardware, supporting various heterogeneous hardware such as CPUs, GPUs, and NPUs. The hardware adaptation rule base has an extensible architecture, which can quickly adapt to new hardware types. It enables dynamic lightweight deployment of large models on edge devices, achieving real-time deep collaborative adaptation between models and edge device hardware resources. This improves resource utilization, system robustness, and service reliability, solving the problems of static deployment of large models, cumbersome hardware adaptation, and inability to respond to changes in the runtime environment in existing edge computing scenarios.

[0052] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A method for lightweight deployment of large models supporting dynamic hardware adaptation, characterized in that, A runtime adaptive system with closed-loop perception and decision-making capabilities is constructed, based on a hardware-independent intermediate representation layer (UDL). Through a complete closed-loop process of monitoring, decision-making, refactoring, compilation, execution, and feedback, the dynamic and lightweight deployment of large models on edge devices is achieved. Specifically, the following steps are included: Step 1: Unified model conversion. The standard large model received from the cloud is converted into a hardware-independent intermediate representation layer UDL computation graph composed of atomic operators through unified compilation and presentation layer. Step 2, runtime state awareness: The dynamic resource status of edge devices is collected through the runtime dynamic monitoring and decision engine, and the static characteristics of the device hardware are analyzed to form a hardware feature description file. Step 3, Reconstruction Decision Generation: The lightweight decision model generates reconstruction decision instructions that include target accuracy, operator fusion strategy, and model segmentation points based on dynamic resource status and hardware static characteristics. Step 4, Model Lightweight Reconstruction: The Model Lightweight Reconstruction module performs online fine-grained structural transformation on the intermediate representation layer UDL computation graph according to the reconstruction decision instructions. Step 5: Hardware adaptation code generation. The multi-backend adaptive code generator combines the reconstructed intermediate representation layer UDL computation graph with the hardware feature description file to generate optimized target code adapted to the current hardware. Step 6: Optimize code execution. The execution engine loads and runs the optimized target code to complete the large model inference task. Step 7, Performance Feedback Optimization: The performance and accuracy monitor collects performance data of the inference task and feeds it back to the lightweight decision model to optimize the subsequent reconstruction decision strategy and form an adaptive closed loop.

2. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 1, characterized in that: In step 2, the dynamic resource status includes CPU / GPU / NPU utilization, cache utilization at each level, remaining available memory, network round-trip latency, and available bandwidth, which are collected in real time by the resource monitor at a millisecond frequency; the hardware static characteristics include instruction set architecture, memory hierarchy, and parallel thread scale, which are analyzed and obtained by the hardware feature extractor when the device is initialized or new hardware is detected.

3. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 2, characterized in that: In step 3, the lightweight decision model determines whether to trigger model reconstruction based on preset heuristic rules or online-trained reinforcement learning strategies; the reconstruction decision instruction also includes the desired computational granularity.

4. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 3, characterized in that: In step 3, the decision strategy of the lightweight decision model can be replaced by a multi-layer conditional judgment rule tree, or a reconstructed strategy library can be generated in advance through simulation and then online query and matching can be performed based on the device status at runtime.

5. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 4, characterized in that: In step 4, the online fine-grained structure transformation includes one or more of dynamic precision quantization, network-aware operator segmentation, and adaptive subgraph pruning. Dynamic precision quantization involves online conversion of the weights and activation values ​​of specified operators or subgraphs in the UDL computation graph between FP16, INT8, and INT4, combined with dynamic calibration to minimize precision loss. Network-aware operator segmentation involves dynamically selecting the optimal segmentation point in the UDL computation graph based on real-time network latency data, and intelligently compressing or dividing the output feature maps of large operators. Adaptive subgraph pruning involves dynamically skipping redundant branches or operators in the UDL computation graph that contribute little to the current inference task.

6. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 5, characterized in that: In step 4, the granularity of the lightweight model reconstruction operation can be replaced by using a predefined computational subgraph as the unit to uniformly switch or prune the precision of the entire Transformer attention module unit.

7. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 6, characterized in that: In step 5, the multi-backend adaptive code generator calls a pre-generated hardware adaptation rule library. The hardware adaptation rule library maps hardware static features to specific optimization rules. The multi-backend adaptive code generator generates target code through just-in-time compilation, generating PTX and CUDA kernels for NVIDIA GPUs and CANN operators for Huawei Ascend NPUs.

8. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 7, characterized in that: In step 5, the implementation of the hardware adaptation rule base can be replaced by embedding a micro-cost model in the compiler, generating near-optimal hardware adaptation code through fast search, thereby reducing the dependence on the rule base.

9. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 8, characterized in that: In step 6, the execution engine uses a millisecond-level seamless hot-switching method to load the new optimized target code, replacing the original running kernel, ensuring that the business is unaware of it.

10. The method for lightweight deployment of large models supporting dynamic hardware adaptation as described in claim 9, characterized in that: In step 7, the performance data includes the execution time, power consumption, and output accuracy of the inference task. The performance and accuracy monitor compares the performance data with the corresponding data of the cloud benchmark model to form a feedback signal.

Images