Multi-dimensional adaptive large model inference engine method and system

Abstract

The invention provides a multi-dimensional self-adaptive large model inference engine method and system, and relates to the technical field of artificial intelligence, and the method comprises the steps: collecting hardware identification information of calculation hardware, and calling a hardware fingerprint database for matching to generate a hardware type result and a hardware characteristic parameter; constructing a hardware abstraction layer, loading a hardware abstraction driving plug-in according to a hardware type result, and virtualizing computing hardware into a unified logic computing power unit through a hardware abstraction interface; when a large model reasoning task is triggered, dynamically routing to a back-end executor according to a hardware type result by querying a back-end registry; on the basis of hardware characteristic parameters and a hardware abstract interface, a hardware compiling tool chain is called through a rear-end executor, graph optimization and operator fusion are executed on a large model calculation graph, and therefore self-adaptive deployment and efficient reasoning of a large model on heterogeneous hardware can be achieved; the problems that in the prior art, heterogeneous hardware is insufficient in compatibility, configuration is tedious and rigid, and compilation optimization barriers and performance losses are caused during model cross-platform migration are solved.

Description

A method and system for a multidimensional adaptive large model inference engine Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for a multidimensional adaptive large model inference engine. Background Technology

[0002] With the rapid iteration of artificial intelligence technology and the accelerated pace of domestic computing power substitution, large-scale model inference deployment has become a crucial link supporting the implementation of various intelligent applications. As the process of domestic substitution deepens, multi-source heterogeneous hardware is widely used in various industries.

[0003] However, the limitations of existing technologies are becoming increasingly apparent, making it difficult to meet the needs of practical applications. First, there is a heavy reliance on the hardware ecosystem and a lack of heterogeneous compatibility. Mainstream AI inference platforms are deeply tied to specific foreign hardware and supporting software stacks. Due to differences in instruction sets and driver architectures, existing inference engines cannot recognize domestically produced chips, leading to code refactoring and posing risks to domestic substitution and supply chain security. Second, operational complexity is high, and fault location is difficult. In existing distributed systems, AI applications, computing resources, and hardware status lack a unified management mechanism. The engine lacks real-time awareness of the underlying hardware status, making it difficult for maintenance personnel to quickly locate the root cause of faults. Third, system silos are evident, hindering automatic adaptation. The software stacks and toolchains of various hardware manufacturers are independent, forming system silos. Existing engines cannot automatically detect the type of accelerator card connected to the server, requiring manual recompilation of the model and modification of configuration files for each hardware change. This cumbersome model deployment process is difficult to adapt to the flexible access requirements of multi-source heterogeneous devices in domestic computing power application scenarios. Finally, the development and deployment cycle is long. Due to the lack of a unified hardware abstraction layer, the introduction of new hardware requires a long process of model conversion, optimization and adaptation, which restricts the agile iteration and rapid launch of AI applications.

[0004] Therefore, it is necessary to provide a multidimensional adaptive large model inference engine method and system to solve the above-mentioned technical problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multidimensional adaptive large model inference engine method and system, which solves the problems of insufficient heterogeneous hardware compatibility, cumbersome and rigid configuration, and compilation optimization barriers and performance losses when migrating models across platforms in existing technologies.

[0006] This invention provides a multi-dimensional adaptive large model inference engine method, comprising: collecting hardware identification information of computing hardware by scanning the PCIe bus and system device tree, calling a preset hardware fingerprint library for matching, and generating hardware type results and hardware characteristic parameters; constructing a hardware abstraction layer, dynamically loading matching hardware abstraction driver plugins according to the hardware type results, and virtualizing the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; when a user triggers a large model inference task, dynamically routing to the corresponding backend executor according to the hardware type results by querying a preset backend registry; and based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit, calling the hardware compilation toolchain through the backend executor to perform execution graph optimization and operator fusion on the large model computation graph in the large model inference task.

[0007] Preferably, the construction of the hardware abstraction layer, based on the hardware type result, dynamically loads matching hardware abstraction driver plugins, and virtualizes the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface. Specifically, the hardware abstraction layer includes a unified device context abstraction base class, which defines standardized virtual functions for initialization, memory allocation, data copying, and computation issuance. The standardized hardware abstraction interface is constructed based on this base class. Based on the hardware type result, hardware abstraction driver plugins matching the computing hardware are selected and dynamically loaded. The standardized hardware abstraction interface masks the underlying implementation differences of different computing hardware, virtualizing the computing hardware into the unified logical computing unit. A heartbeat mechanism with the underlying driver is established to collect hardware physical status information in real time, including at least computing hardware occupancy, memory usage, computing hardware temperature, and fan speed.

[0008] Preferably, based on the hardware characteristic parameters and the hardware physical state information, a computing hardware capability mapping model is constructed to quantify the equivalent computing power value of the computing hardware. The corresponding calculation formula is as follows: In the formula, This represents the hardware adaptation efficiency coefficient. ; This indicates the peak computing power of the computing hardware. This indicates the available video memory capacity of the computing hardware; This represents the total video memory capacity of the computing hardware. This indicates the available memory bandwidth of the computing hardware; This indicates the maximum memory bandwidth of the computing hardware; Indicates the temperature effect coefficient. ; This represents the standardized temperature of the computing hardware. , This indicates the real-time temperature of the computing hardware. This indicates the minimum temperature threshold for computing hardware under normal operating conditions. This represents the highest temperature threshold for the computing hardware to operate safely; based on the equivalent computing power value. The computing resources of the computing hardware are normalized and aggregated, and the computing resources are encapsulated into a unified virtual computing power set through the standardized hardware abstraction interface to form the unified logical computing power unit.

[0009] Preferably, when the user triggers a large model inference task, dynamically routing to the corresponding backend executor based on the hardware type result by querying a preset backend registry specifically includes: when the large model inference task is triggered, querying the backend registry through a routing engine; if a matching record between the hardware type result and the backend executor exists in the cache of the routing engine, then directly calling the corresponding backend executor based on the matching record; otherwise, traversing the backend registry, comparing the hardware type result with the hardware type identifier in the backend registry, matching the corresponding backend executor, and dynamically distributing the large model computation graph in the large model inference task to the corresponding backend executor; wherein, the backend executor includes at least Ascend ACL backend, Cambricon CNRT backend, AMD ROCm backend, and GPU backend.

[0010] Preferably, the standard hardware abstraction interface corresponding to the hardware characteristic parameters and the unified logic computing unit, through the backend executor calling the hardware compilation toolchain, optimizes the execution graph and merges operators in the large model computation graph of the large model inference task, specifically including: the backend executor obtaining the data flow dependencies and node computational volume data of the large model computation graph through the standard hardware abstraction interface, and calling the hardware compilation toolchain matched with the computing hardware; the hardware compilation toolchain calculates the validity weight of the i-th node in the large model computation graph. The corresponding calculation formula is as follows: In the formula, Represents the balance coefficient. ; This represents the computational cost of the i-th node in the computational graph of the large model. This represents the cumulative computational cost of all nodes in the large model computation graph. This represents the data flow dependency depth of the i-th node in the computational graph of the large model; This represents the maximum dependency depth of data flow across all nodes in the large model computation graph; within the large model computation graph, validity weights are filtered and retained. Not less than the preset effective node threshold The effective nodes are reorganized, their data flow dependencies are re-examined, and their execution order is adjusted in conjunction with the parallel computing architecture of the computing hardware.

[0011] Preferably, the step of performing operator fusion on the large model computation graph in the large model inference task specifically includes: dividing the basic operators into several consecutive basic operator groups based on the data flow dependencies of the large model computation graph; and the hardware compilation toolchain calculating the fusion gain of the consecutive basic operator groups based on the hardware characteristic parameters. The corresponding calculation formula is as follows: In the formula, This represents the expected memory saving after fusing consecutive basic operator groups; This represents the minimum video memory required for inference of a continuous set of basic operators. This represents the cost balance factor. ; This represents the additional computational overhead expected during the fusion of consecutive basic operator groups; This represents the total computational cost before fusion of the continuous basic operator group; if the fusion gain of the continuous basic operator group... If the value is greater than 0, it is determined that there is a benefit after the fusion of the continuous basic operators group, and the continuous basic operators group is fused into a composite operator.

[0012] Preferably, it also includes adaptive quantization compression and intelligent computing power scheduling: The adaptive quantization compression means that the backend executor obtains the data precision supported by the computing hardware from the hardware characteristic parameters through the standardized hardware abstraction interface. If the computing hardware supports low-precision hardware acceleration, the quantization configuration is automatically loaded to convert the model precision format from a high-precision format to the corresponding low-precision format; if the computing hardware does not support low-precision hardware acceleration, the original model precision format is maintained for inference. The intelligent computing power scheduling means that based on the unified logical computing power unit, a multi-level task queue is established, and the large model inference task is divided into a high-priority task queue and a low-priority task queue. Computing power channels are reserved in the high-priority task queue, and a priority-based preemptive scheduling mechanism is implemented.

[0013] A multidimensional adaptive large model inference engine system, comprising: a data acquisition and matching module, used to acquire hardware identification information of computing hardware by scanning the PCIe bus and system device tree, call a preset hardware fingerprint database for matching, and generate hardware type results and hardware characteristic parameters; a hardware abstraction layer module, used to construct a hardware abstraction layer, dynamically load matching hardware abstraction driver plugins according to the hardware type results, and virtualize the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; a dynamic routing module, used to dynamically route to the corresponding backend executor by querying a preset backend registry according to the hardware type results when a large model inference task is triggered by a user terminal; and a compilation and optimization module, used to perform large model computation graph execution graph optimization and operator fusion in the large model inference task by calling a hardware compilation toolchain through the backend executor based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit.

[0014] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor performs the steps of a multidimensional adaptive large model inference engine method as described in any of the above.

[0015] A readable storage medium storing a computer program, which, when executed by a processor, is used to implement the steps of a multidimensional adaptive large model inference engine method as described in any of the above.

[0016] Compared with related technologies, the multi-dimensional adaptive large model inference engine method and system provided by this invention has the following beneficial effects: This invention collects hardware identification information of computing hardware by scanning the PCIe bus and system device tree, calls a preset hardware fingerprint library for matching, and generates hardware type results and hardware characteristic parameters; it constructs a hardware abstraction layer, dynamically loads matching hardware abstraction driver plugins according to the hardware type results, and virtualizes computing hardware into unified logical computing units through standardized hardware abstraction interfaces; when the user triggers a large model inference task, it dynamically routes to the corresponding backend executor according to the hardware type results by querying a preset backend registry; based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit, the backend executor calls the hardware compilation toolchain to optimize the execution graph and operator fusion of the large model computation graph in the large model inference task, thereby enabling adaptive deployment and efficient inference of large models on heterogeneous hardware, solving the problems of insufficient heterogeneous hardware compatibility, cumbersome and rigid configuration, compilation optimization barriers and performance loss when migrating models across platforms in existing technologies.

[0017] This invention achieves automatic identification and characteristic awareness of domestically produced heterogeneous hardware by scanning the PCIe bus and system device tree to collect hardware identification information and matching it with a hardware fingerprint database. Combined with a standardized hardware abstraction layer and dynamically loaded hardware abstraction driver plugins, it completely breaks down the underlying differences between hardware from different manufacturers, virtualizing heterogeneous chips into unified logical computing units. This removes the static binding between models and specific hardware, eliminating the need for manual code reconstruction or configuration modification, significantly improving the compatibility and adaptation efficiency of domestically produced hardware, and ensuring supply chain security. This invention, relying on a backend registry and dynamic routing mechanism, accurately distributes large model inference tasks to suitable backend executors. It combines hardware characteristics with the invocation of a dedicated hardware compilation toolchain, significantly reducing memory access and computational overhead through node validity filtering, data flow optimization, and operator fusion, thus solving the performance loss problem of cross-platform migration. In this invention, adaptive quantization compression dynamically adjusts the model format according to the hardware's precision support capabilities, improving inference speed and reducing GPU memory usage while ensuring accuracy. Intelligent computing power scheduling, through multi-level task queues and a preemptive mechanism, reserves dedicated channels for high-priority tasks, ensuring the service stability of important businesses. This invention breaks the black box of operation and maintenance by collecting hardware physical status information in real time and managing the entire chain, simplifies the fault location process, and, combined with a unified interface and automated deployment process, significantly reduces the complexity of operation and maintenance and the development and implementation cycle, thus meeting the needs of agile business iteration. Attached Figure Description

[0018] Figure 1 is a flowchart of a multidimensional adaptive large model inference engine method provided by an embodiment of the present invention; Figure 2 is a system block diagram of a multidimensional adaptive large model inference engine system provided by an embodiment of the present invention; Figure 3 is a hardware structure diagram of an electronic device provided by an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Figure 1 shows a flowchart of a multi-dimensional adaptive large model inference engine method provided by an embodiment of the present invention. The execution subject of the method shown in Figure 1 can be software and / or hardware devices. The execution subject of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. Among them, user equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud based on cloud computing composed of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. The method includes steps S1 to S4, specifically as follows: S1, by scanning the PCIe bus and system device tree, the hardware identification information of the computing hardware is collected, and a preset hardware fingerprint library is called for matching to generate hardware type results and hardware characteristic parameters; wherein, the PCIe bus is a high-speed hardware transmission interface inside the computer. The system device tree is a hardware configuration description file. Computing hardware refers to physical devices that carry data computing and processing tasks, covering hardware carriers with computing capabilities such as GPUs and dedicated chips. Hardware identification information includes the manufacturer's identifier and device identifier of the computing hardware, used to uniquely identify the manufacturer and specific model of the hardware. The hardware fingerprint database pre-stores the manufacturer's identifier, device identifier, and corresponding hardware characteristic parameters for various domestically produced hardware. Hardware type results include Huawei Ascend, Cambricon MLU, Hygon DCU, and general-purpose GPUs, etc. Hardware characteristic parameters include the peak computing power, memory capacity, and supported data precision of the computing hardware.

[0021] Understandably, the matching process involves accurately comparing the real-time collected hardware identification information with the pre-stored data in the hardware fingerprint database, automatically outputting a uniquely corresponding hardware type result and hardware characteristic parameters, without requiring manual intervention to modify configuration files or recompile code.

[0022] S2, Constructing a hardware abstraction layer: Based on the hardware type result, dynamically load matching hardware abstraction driver plugins, and virtualize the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; Specifically, constructing the hardware abstraction layer, based on the hardware type result, dynamically loading matching hardware abstraction driver plugins, and virtualizing the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface includes: the hardware abstraction layer includes a unified device context abstraction base class, which defines standardized virtual functions for initialization, memory allocation, data copying, and computation issuance; the standardized hardware abstraction interface is constructed based on the device context abstraction base class; based on the hardware type result, filter and dynamically load the hardware abstraction driver plugins that match the computing hardware, and shield the underlying implementation differences of different computing hardware through the standardized hardware abstraction interface, virtualizing the computing hardware into the unified logical computing unit; establishing a heartbeat mechanism with the underlying driver to collect hardware physical status information in real time, the hardware physical status information including at least computing hardware occupancy rate, memory usage rate, computing hardware temperature, and fan speed.

[0023] The Device Context Abstract Base Class serves as the logical carrier of the hardware abstraction layer, providing a unified functional abstraction template for various computing hardware. Standardized virtual functions, defined in the Device Context Abstract Base Class, are generic operation interfaces covering essential initialization, memory allocation, data copying, and computational dispatch operations required for computing hardware operation, ensuring consistency in basic operation interfaces across different computing hardware. Standardized hardware abstract interfaces, built upon the Device Context Abstract Base Class, shield the differences in underlying driver architectures and instruction sets between different computing hardware, decoupling upper-layer logic from hardware implementation. Hardware abstraction driver plugins are adaptation modules developed for specific computing hardware, containing dedicated driver logic and interface implementations. The unified logical computing power unit, a virtual computing power carrier formed through interface shielding and resource aggregation, enables heterogeneous computing hardware to present a unified computing power call format. The heartbeat mechanism is a continuous communication link between the engine and the underlying driver, ensuring real-time hardware status awareness. Hardware physical status information reflects the operating status of the computing hardware, including GPU utilization, memory usage, chip temperature, fan speed, etc., and is reported to the upper-level operations and maintenance center, breaking the black box of operations and maintenance.

[0024] In practical applications, if a domestic computing cluster simultaneously deploys Huawei Ascend and Cambricon MLU heterogeneous computing hardware, after system startup, the hardware abstraction layer first loads standardized virtual functions such as initialization, memory allocation, data copying, and computation issuance through the device context abstraction base class, forming a standardized hardware abstraction interface. Based on the hardware type results obtained by scanning the PCIe bus and system device tree, the system selects and dynamically loads hardware abstraction driver plugins that match the two types of chips. Through the standardized hardware abstraction interface, the differences in instruction sets and driver architectures between the two are masked, and the physical chips are virtualized into unified logical computing units for unified scheduling of large-scale model inference tasks at the upper layer. At the same time, the heartbeat mechanism continuously interacts with the underlying driver, collecting real-time hardware hardware utilization of Cambricon MLU, memory usage of Huawei Ascend, and hardware physical status information such as temperature and fan speed of the two types of chips, and reporting it to the operation and maintenance center in real time.

[0025] By adopting the above methods, unified adaptation and virtualization management of heterogeneous computing hardware can be achieved, shielding the differences in underlying drivers and architectures, forming a unified logical computing unit for upper-layer scheduling, and obtaining hardware physical status information in real time through a heartbeat mechanism, thus solving the problems of insufficient heterogeneous compatibility and complex operation and maintenance.

[0026] Based on the hardware characteristic parameters and the hardware physical state information, a computing hardware capability mapping model is constructed to quantify the equivalent computing power value of the computing hardware. The corresponding calculation formula is as follows: In the formula, This represents the hardware adaptation efficiency coefficient. ; This indicates the peak computing power of the computing hardware. This indicates the available video memory capacity of the computing hardware; This represents the total video memory capacity of the computing hardware. This indicates the available memory bandwidth of the computing hardware; This indicates the maximum memory bandwidth of the computing hardware; Indicates the temperature effect coefficient. ; This represents the standardized temperature of the computing hardware. , This indicates the real-time temperature of the computing hardware. This indicates the minimum temperature threshold for computing hardware under normal operating conditions. This represents the highest temperature threshold for the computing hardware to operate safely; based on the equivalent computing power value. The computing resources of the computing hardware are normalized and aggregated, and the computing resources are encapsulated into a unified virtual computing power set through the standardized hardware abstraction interface to form the unified logical computing power unit.

[0027] The equivalent computing power value is the actual usable computing power of the hardware, quantified through a hardware capability mapping model. The hardware adaptation efficiency coefficient is used to correct the compatibility between the computing hardware and the engine. Peak computing power is the maximum computing capability achievable by the computing hardware. Available video memory capacity is the currently allocable video memory resources. Total video memory capacity is the total video memory scale equipped by the computing hardware. Available memory bandwidth is the currently available memory data transfer rate. Maximum memory bandwidth is the highest data transfer rate supported by the computing hardware.

[0028] The temperature impact coefficient is used to adjust the degree of temperature's influence on computing power. Real-time temperature is the current operating temperature of the computing hardware. The minimum temperature threshold is the lowest temperature limit for the computing hardware to operate normally. The maximum temperature threshold is the highest temperature limit for the computing hardware to operate safely. Standardized temperature is the result of normalizing the real-time temperature, eliminating differences in temperature ranges between different hardware. Normalization unifies the computing power of different computing hardware to the same scale. Aggregation integrates multi-path hardware computing power resources. Virtual computing power aggregation is a unified logical computing power carrier encapsulated, ultimately forming a unified logical computing power unit, enabling unified scheduling of heterogeneous computing hardware.

[0029] Understandably, the computing hardware capability mapping model accurately quantifies the actual available computing power of computing hardware by integrating hardware characteristic parameters and real-time hardware physical state information. After normalization processing to eliminate the specification differences of different computing hardware, it integrates multi-path computing resources through aggregation processing. Finally, with the help of standardized hardware abstraction interfaces, it encapsulates multi-path computing resources into a virtual computing power set with unified attributes, forming a unified logical computing power unit. This enables integrated management and scheduling of heterogeneous computing power, eliminating the impact of underlying hardware differences on upper-layer applications.

[0030] S3, when the user triggers a large model inference task, a preset backend registry is queried, and the system dynamically routes the system to the corresponding backend executor based on the hardware type result. Specifically, this involves: when the large model inference task is triggered, the routing engine queries the backend registry; if a matching record exists between the hardware type result and the backend executor in the routing engine's cache, the corresponding backend executor is directly invoked based on the matching record; otherwise, the backend registry is traversed, the hardware type result is compared with the hardware type identifier in the backend registry, a matching backend executor is found, and the large model computation graph in the large model inference task is dynamically distributed to the corresponding backend executor; wherein, the backend executor includes at least the Ascend ACL backend, Cambricon CNRT backend, AMD ROCm backend, and GPU backend.

[0031] The large-model inference task is a task type where a pre-trained large model is used to generate a target output through calculations based on predetermined parameters after inputting data. The backend registry stores the association between hardware type results and corresponding backend executors, pre-collecting backend executor information for various computing hardware. The routing engine is responsible for initiating backend registry queries and executing matching logic. The cache temporarily stores completed hardware type result and backend executor matching records, enabling rapid reuse of matching results and reducing the overhead of repeated queries. The hardware type identifier is a unique identifier in the backend registry used to distinguish different computing hardware, forming a precise comparison basis with the hardware type results.

[0032] The backend executor is an inference execution module adapted to specific computing hardware. It includes Ascend ACL backend, Cambricon CNRT backend, AMD ROCm backend, and general-purpose GPU backend, each corresponding to different brands or types of computing hardware, and possesses dedicated execution logic adapted to the corresponding computing hardware. The large model computation graph is a structured representation of the computational logic of large model inference tasks, using nodes to represent operators and edges to represent data flow dependencies, supporting efficient compilation optimization and computing power scheduling.

[0033] Understandably, dynamic routing is based on hardware type results, relying on a pre-defined backend registry. The routing engine prioritizes retrieving matching records from the cache; if no cache is available, it traverses the backend registry to perform accurate comparisons and quickly locate the corresponding backend executor. The core functionality lies in dynamically distributing large model computation graphs. It adapts to different types of backend executors, such as Ascend ACL backends, Cambricon CNRT backends, AMD ROCm backends, and general-purpose GPU backends, without manual intervention. This ensures that large model inference tasks are automatically associated with the underlying computing hardware after adaptation, guaranteeing efficient and smooth execution of inference tasks in heterogeneous computing hardware environments and demonstrating the engine's flexible adaptability to multi-source computing hardware.

[0034] S4. Based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logic computing unit, the hardware compilation toolchain is called through the backend executor to perform graph optimization and operator fusion on the large model computation graph in the large model inference task.

[0035] The standardized hardware abstraction interface corresponding to the hardware characteristic parameters and the unified logical computing unit, through the backend executor calling the hardware compilation toolchain, optimizes the execution graph and merges operators in the large model computation graph of the large model inference task. Specifically, the backend executor obtains the data flow dependencies and node computation data of the large model computation graph through the standardized hardware abstraction interface, and calls the hardware compilation toolchain matched with the computing hardware; the hardware compilation toolchain calculates the validity weight of the i-th node in the large model computation graph. The corresponding calculation formula is as follows: In the formula, Represents the balance coefficient. ; This represents the computational cost of the i-th node in the computational graph of the large model. This represents the cumulative computational cost of all nodes in the large model computation graph. This represents the data flow dependency depth of the i-th node in the computational graph of the large model; This represents the maximum dependency depth of data flow across all nodes in the large model computation graph; within the large model computation graph, validity weights are filtered and retained. Not less than the preset effective node threshold The effective nodes are reorganized, their data flow dependencies are re-examined, and their execution order is adjusted in conjunction with the parallel computing architecture of the computing hardware.

[0036] Among these, data flow dependencies are the sequential relationships between data transmission and computation execution between nodes in a large-scale computational graph, determining the logical order of task execution. Node computational load data represents the scale of computational tasks undertaken by a single node, reflecting the computational complexity of the node. The hardware compilation toolchain is a collection of compilation tools designed for specific computing hardware, possessing adaptation and optimization capabilities such as graph optimization and operator fusion.

[0037] The balancing coefficient is used to adjust the relative impact of computational load and data flow dependency depth on node effectiveness evaluation. Node computational load refers to the scale of the specific computational tasks undertaken by a single node. Cumulative computational load is the sum of the computational tasks of all nodes in the large model computation graph. Data flow dependency depth refers to the number of levels at which a single node depends on its predecessor nodes in the computation process. The maximum data flow dependency depth is the maximum value of the dependency levels of all nodes in the large model computation graph.

[0038] The validity weight is a quantitative result of node importance obtained by combining the proportion of computational load and the proportion of data flow dependency depth. The preset valid node threshold is the critical criterion for determining whether a node should be retained. A valid node is a node whose validity weight reaches the preset valid node threshold and makes a substantial contribution to the inference result. The parallel computing architecture is a hardware structure design that supports simultaneous processing of multiple tasks. By adjusting the execution order of valid nodes, the parallel processing potential of the computing hardware is fully utilized, computational waiting time is reduced, and the overall execution efficiency of the inference task is improved.

[0039] In practical applications, if a domestic intelligent computing platform deploys a Huawei Ascend chip, when a user triggers a large-scale text generation and inference task, the Ascend ACL backend obtains the data flow dependencies and computational load data of each node in the large-scale model computation graph through a standardized hardware abstraction interface, and calls a hardware compilation toolchain adapted for Huawei Ascend. The hardware compilation toolchain calculates the validity weight of each node according to a formula, selects valid nodes with validity weights not less than a preset valid node threshold, and eliminates redundant intermediate data forwarding nodes. Subsequently, it re-examines the data flow dependencies of valid nodes, and, combined with Huawei Ascend's parallel computing architecture, adjusts feature extraction nodes and semantic encoding nodes without dependencies to execute in parallel, and sets dependent result integration nodes for subsequent execution. This fully leverages the hardware's parallel processing capabilities, reduces the overall execution time of the inference task, and ensures efficient operation of the large model on domestic hardware.

[0040] The operation of operator fusion on the large model computation graph in the large model inference task specifically includes: dividing the basic operators into several consecutive basic operator groups based on the data flow dependencies of the large model computation graph; and the hardware compilation toolchain calculating the fusion gain of the consecutive basic operator groups based on the hardware characteristic parameters. The corresponding calculation formula is as follows: In the formula, This represents the expected memory saving after fusing consecutive basic operator groups; This represents the minimum video memory required for inference of a continuous set of basic operators. This represents the cost balance factor. ; This represents the additional computational overhead expected during the fusion of consecutive basic operator groups; This represents the total computational cost before fusion of the continuous basic operator group; if the fusion gain of the continuous basic operator group... If the value is greater than 0, it is determined that there is a benefit after the fusion of the continuous basic operators group, and the continuous basic operators group is fused into a composite operator.

[0041] In this context, a basic operator is the smallest independent computational unit indivisible within the computational graph of a large model, carrying a single basic computational function. A continuous set of basic operators is a collection of basic operators arranged consecutively based on data flow dependencies and exhibiting data transfer relationships. The fusion gain is a quantitative indicator that measures the benefit of fusing continuous sets of basic operators, directly reflecting the performance improvement effect of fusion on the system.

[0042] Memory usage refers to the amount of video memory resources used by a group of consecutive basic operators during inference. Minimum memory capacity is the minimum threshold of video memory resources required to ensure normal inference for the group of consecutive basic operators. The overhead balancing coefficient is used to adjust the weight of additional computational overhead in the fusion gain evaluation, balancing the relationship between memory savings and computational overhead. Additional computational overhead is the additional computational cost incurred during the fusion of consecutive basic operators. Total computational overhead is the total cost of performing all computational tasks before the fusion of consecutive basic operators. A composite operator is an integrated computational unit formed by combining consecutive basic operators that have positive fusion benefits, reducing the number of data interactions between operators and improving inference efficiency.

[0043] Understandably, operator fusion divides continuous basic operator groups based on the data flow dependencies of the large model computation graph. It then uses a hardware compilation toolchain combined with hardware characteristic parameters to quantify the fusion gain, integrating only operator groups that yield positive benefits after fusion. By using composite operators, it reduces the number of data transfers and interactions between operators, fully adapting to the computational characteristics of the underlying computing hardware, reducing memory usage and computational overhead, thereby improving the execution efficiency of large model inference tasks and demonstrating the engine's targeted adaptation capabilities to hardware characteristics.

[0044] It also includes adaptive quantization compression and intelligent computing power scheduling: The adaptive quantization compression means that the backend executor obtains the data precision supported by the computing hardware from the hardware characteristic parameters through the standardized hardware abstraction interface. If the computing hardware supports low-precision hardware acceleration, the quantization configuration is automatically loaded to convert the model precision format from the high-precision format to the corresponding low-precision format; if the computing hardware does not support low-precision hardware acceleration, the original model precision format is maintained for inference. The intelligent computing power scheduling means that based on the unified logical computing power unit, a multi-level task queue is established to divide the large model inference task into a high-priority task queue and a low-priority task queue. Computing power channels are reserved in the high-priority task queue, and a priority-based preemptive scheduling mechanism is implemented.

[0045] Data precision refers to the numerical computation precision specifications supported by the computing hardware, determining the numerical storage and computation format for large model inference. Low-precision hardware acceleration refers to the dedicated acceleration capabilities of computing hardware for low-precision data formats, improving inference speed and reducing resource consumption. Quantization configuration is a set of parameters used to convert the model precision format, including precision conversion rules and adaptation logic. Model precision format refers to the storage format of model weights and data during computation. High-precision formats are numerical storage formats with higher precision, such as FP32 and FP16. Low-precision formats are numerical storage formats with relatively lower precision but that can guarantee inference performance, such as INT8. The original model precision format is the initial precision format of the model before quantization compression.

[0046] A multi-level task queue is a resource scheduling queue system divided according to task priority, used to distinguish inference tasks of different importance. High-priority task queues are used to carry important business-related inference tasks that need to be executed first, such as real-time analysis tasks. Low-priority task queues are used to carry non-critical inference tasks that can be delayed, such as offline model compilation tasks. Computing power channels are dedicated computing power resource links reserved for high-priority task queues, ensuring that important tasks are not interfered with by other tasks. The preemptive scheduling mechanism is a scheduling logic where the scheduler, based on task priority, pauses low-priority tasks and releases resources for high-priority tasks when resources are insufficient. Low-priority tasks resume execution after the high-priority tasks are completed, ensuring service level agreement (SLA) guarantees for important business operations.

[0047] The above approach achieves precise matching between model precision format and computing hardware data precision support capabilities, accelerates inference speed and reduces memory usage by leveraging low-precision hardware, optimizes resource allocation efficiency of unified logic computing units through multi-level task queues and preemptive scheduling mechanisms, provides dedicated computing power guarantees for high-priority tasks, avoids interference from low-priority tasks for important business operations, and ultimately achieves a dual improvement in large model inference efficiency and service stability in heterogeneous hardware environments, adapting to the needs of flexible access to multi-source hardware and agile business iteration in domestic scenarios.

[0048] It also includes: providing a standardized encapsulation process to package the compiled and optimized model into microservices. After the user triggers the deployment operation through the interface, the system automatically requests resources from the computing power pool and starts the service without the need for manual environment configuration or intervention in the deployment process.

[0049] A unified API interface is exposed to the outside world. The API interface includes at least inference call, result acquisition and status query interfaces. Upper-layer business systems interact with the large model inference engine through the unified API interface, which is not affected by the type of underlying computing hardware and breaks down information silos.

[0050] The above methods automate the model deployment process and standardize the interaction interface, completely shielding the complex underlying details of large model inference and the differences in computing hardware. After the model is packaged as a microservice, resource requests and service startup can be completed without manual intervention, significantly simplifying the deployment process and reducing operational costs. The unified API interface breaks down the interaction barriers between different hardware and systems, enabling upper-layer business systems to seamlessly connect with the large model inference engine without needing to be concerned with the underlying computing hardware type. This effectively eliminates information silos, improves multi-system integration efficiency, and adapts to the core needs of agile business iteration and multi-system collaboration in a domestically produced heterogeneous hardware environment.

[0051] Figure 2 shows a system block diagram of a multi-dimensional adaptive large model inference engine system provided in an embodiment of the present invention. The system includes: a data acquisition and matching module, used to acquire hardware identification information of computing hardware by scanning the PCIe bus and system device tree, call a preset hardware fingerprint library for matching, and generate hardware type results and hardware characteristic parameters; a hardware abstraction layer module, used to construct a hardware abstraction layer, dynamically load matching hardware abstraction driver plugins according to the hardware type results, and virtualize the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; a dynamic routing module, used to dynamically route to the corresponding backend executor by querying a preset backend registry according to the hardware type results when the user triggers a large model inference task; and a compilation and optimization module, used to perform large model computation graph execution graph optimization and operator fusion in the large model inference task by calling the hardware compilation toolchain through the backend executor based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit.

[0052] The apparatus shown in Figure 2 can be used to perform the steps in the method embodiment shown in Figure 1. Its implementation principle and technical effect are similar, and will not be described again here.

[0053] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor performs the steps of a multidimensional adaptive large model inference engine method as described in any of the above.

[0054] Figure 3 shows a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. The electronic device 30 includes a processor 31, a memory 32, and a computer program; wherein the memory 32 is used to store the computer program, and the memory may also be flash memory. The computer program is, for example, an application program or functional module that implements the above-described method.

[0055] Processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0056] Alternatively, the memory 32 can be either standalone or integrated with the processor 31.

[0057] When the memory 32 is a device independent of the processor 31, the device may further include a bus 33 for connecting the memory 32 and the processor 31.

[0058] A readable storage medium storing a computer program, which, when executed by a processor, is used to implement the steps of a multidimensional adaptive large model inference engine method as described in any of the above.

[0059] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0060] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the methods provided in the various embodiments described above.

[0061] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0062] Through the above embodiments, this invention scans the PCIe bus and system device tree to collect hardware identification information of computing hardware, calls a preset hardware fingerprint database for matching, and generates hardware type results and hardware characteristic parameters. It constructs a hardware abstraction layer, dynamically loads matching hardware abstraction driver plugins based on the hardware type results, and virtualizes the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface. When a user triggers a large model inference task, it queries a preset backend registry and dynamically routes the system to the corresponding backend executor based on the hardware type results. Based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit, the backend executor calls the hardware compilation toolchain to optimize the execution graph and fuse operators in the large model computation graph of the large model inference task. This enables adaptive deployment and efficient inference of large models on heterogeneous hardware, solving the problems of insufficient heterogeneous hardware compatibility, cumbersome and rigid configuration, and compilation optimization barriers and performance losses during model cross-platform migration in existing technologies.

[0063] This invention achieves automatic identification and characteristic awareness of domestically produced heterogeneous hardware by scanning the PCIe bus and system device tree to collect hardware identification information and matching it with a hardware fingerprint database. Combined with a standardized hardware abstraction layer and dynamically loaded hardware abstraction driver plugins, it completely breaks down the underlying differences between hardware from different manufacturers, virtualizing heterogeneous chips into unified logical computing units. This removes the static binding between models and specific hardware, eliminating the need for manual code reconstruction or configuration modification, significantly improving the compatibility and adaptation efficiency of domestically produced hardware, and ensuring supply chain security. This invention, relying on a backend registry and dynamic routing mechanism, accurately distributes large model inference tasks to suitable backend executors. It combines hardware characteristics with the invocation of a dedicated hardware compilation toolchain, significantly reducing memory access and computational overhead through node validity filtering, data flow optimization, and operator fusion, thus solving the performance loss problem of cross-platform migration. In this invention, adaptive quantization compression dynamically adjusts the model format according to the hardware's precision support capabilities, improving inference speed and reducing GPU memory usage while ensuring accuracy. Intelligent computing power scheduling, through multi-level task queues and a preemptive mechanism, reserves dedicated channels for high-priority tasks, ensuring the service stability of important businesses. This invention breaks the black box of operation and maintenance by collecting hardware physical status information in real time and managing the entire chain, simplifies the fault location process, and, combined with a unified interface and automated deployment process, significantly reduces the complexity of operation and maintenance and the development and implementation cycle, thus meeting the needs of agile business iteration.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for a multidimensional adaptive large model inference engine, characterized in that, The method includes: collecting hardware identification information of computing hardware by scanning the PCIe bus and system device tree; matching the hardware with a preset hardware fingerprint database to generate hardware type results and hardware characteristic parameters; constructing a hardware abstraction layer; dynamically loading matching hardware abstraction driver plugins based on the hardware type results; virtualizing the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; when a user triggers a large model inference task, dynamically routing to the corresponding backend executor based on the hardware type results by querying a preset backend registry; and, based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit, calling the hardware compilation toolchain through the backend executor to perform execution graph optimization and operator fusion on the large model computation graph in the large model inference task.

2. The method for a multidimensional adaptive large model inference engine according to claim 1, characterized in that, The construction of the hardware abstraction layer, based on the hardware type result, dynamically loads matching hardware abstraction driver plugins, and virtualizes the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface. Specifically, the hardware abstraction layer includes a unified device context abstraction base class, which defines standardized virtual functions for initialization, memory allocation, data copying, and computation issuance. The standardized hardware abstraction interface is constructed based on this base class. Based on the hardware type result, hardware abstraction driver plugins matching the computing hardware are selected and dynamically loaded. The standardized hardware abstraction interface masks the underlying implementation differences of different computing hardware, virtualizing the computing hardware into the unified logical computing unit. A heartbeat mechanism with the underlying driver is established to collect hardware physical status information in real time, including at least computing hardware occupancy, memory usage, computing hardware temperature, and fan speed.

3. The method for a multidimensional adaptive large model inference engine according to claim 2, characterized in that, Based on the hardware characteristic parameters and the hardware physical state information, a computing hardware capability mapping model is constructed to quantify the equivalent computing power value of the computing hardware. The corresponding calculation formula is as follows: In the formula, This represents the hardware adaptation efficiency coefficient. ； This indicates the peak computing power of the computing hardware. This indicates the available video memory capacity of the computing hardware; This represents the total video memory capacity of the computing hardware. This indicates the available memory bandwidth of the computing hardware; This indicates the maximum memory bandwidth of the computing hardware; Indicates the temperature effect coefficient. ； This represents the standardized temperature of the computing hardware. ， This indicates the real-time temperature of the computing hardware. This indicates the minimum temperature threshold for computing hardware under normal operating conditions. This represents the highest temperature threshold for the computing hardware to operate safely; based on the equivalent computing power value. The computing resources of the computing hardware are normalized and aggregated, and the computing resources are encapsulated into a unified virtual computing power set through the standardized hardware abstraction interface to form the unified logical computing power unit.

4. The method for a multidimensional adaptive large model inference engine according to claim 1, characterized in that, When a user triggers a large model inference task, the system dynamically routes the hardware type result to the corresponding backend executor by querying a preset backend registry. Specifically, this includes: when the large model inference task is triggered, querying the backend registry through a routing engine; if a matching record exists between the hardware type result and the backend executor in the routing engine's cache, the corresponding backend executor is directly invoked based on the matching record; otherwise, the backend registry is traversed, the hardware type result is compared with the hardware type identifier in the backend registry, a matching backend executor is found, and the large model computation graph in the large model inference task is dynamically distributed to the corresponding backend executor; wherein, the backend executor includes at least the Ascend ACL backend, Cambricon CNRT backend, AMD ROCm backend, and GPU backend.

5. The method for a multidimensional adaptive large model inference engine according to claim 1, characterized in that, The standardized hardware abstraction interface corresponding to the hardware characteristic parameters and the unified logical computing unit, through the backend executor calling the hardware compilation toolchain, optimizes the execution graph and merges operators in the large model computation graph of the large model inference task. Specifically, the backend executor obtains the data flow dependencies and node computation data of the large model computation graph through the standardized hardware abstraction interface, and calls the hardware compilation toolchain matched with the computing hardware; the hardware compilation toolchain calculates the validity weight of the i-th node in the large model computation graph. The corresponding calculation formula is as follows: In the formula, Represents the balance coefficient. ； This represents the computational cost of the i-th node in the computational graph of the large model. This represents the cumulative computational cost of all nodes in the large model computation graph. This represents the data flow dependency depth of the i-th node in the computational graph of the large model; This represents the maximum dependency depth of data flow across all nodes in the large model computation graph; within the large model computation graph, validity weights are filtered and retained. Not less than the preset effective node threshold The effective nodes are reorganized, their data flow dependencies are re-examined, and their execution order is adjusted in conjunction with the parallel computing architecture of the computing hardware.

6. The multidimensional adaptive large model inference engine method according to claim 5, characterized in that, The operation of operator fusion on the large model computation graph in the large model inference task specifically includes: dividing the basic operators into several consecutive basic operator groups based on the data flow dependencies of the large model computation graph; and the hardware compilation toolchain calculating the fusion gain of the consecutive basic operator groups based on the hardware characteristic parameters. The corresponding calculation formula is as follows: In the formula, This represents the expected memory saving after fusing consecutive basic operator groups; This represents the minimum video memory required for inference of a continuous set of basic operators. This represents the cost balance factor. ； This represents the additional computational overhead expected during the fusion of consecutive basic operator groups; This represents the total computational cost before fusion of the continuous basic operator group; if the fusion gain of the continuous basic operator group... If the value is greater than 0, it is determined that there is a benefit after the fusion of the continuous basic operators group, and the continuous basic operators group is fused into a composite operator.

7. The method for a multidimensional adaptive large model inference engine according to claim 1, characterized in that, It also includes adaptive quantization compression and intelligent computing power scheduling: The adaptive quantization compression means that the backend executor obtains the data precision supported by the computing hardware from the hardware characteristic parameters through the standardized hardware abstraction interface. If the computing hardware supports low-precision hardware acceleration, the quantization configuration is automatically loaded to convert the model precision format from the high-precision format to the corresponding low-precision format; if the computing hardware does not support low-precision hardware acceleration, the original model precision format is maintained for inference. The intelligent computing power scheduling means that based on the unified logical computing power unit, a multi-level task queue is established to divide the large model inference task into a high-priority task queue and a low-priority task queue. Computing power channels are reserved in the high-priority task queue, and a priority-based preemptive scheduling mechanism is implemented.

8. A multidimensional adaptive large model inference engine system, applied to the multidimensional adaptive large model inference engine method as described in any one of claims 1-7, characterized in that, The system includes: a data acquisition and matching module, used to acquire hardware identification information of computing hardware by scanning the PCIe bus and system device tree, call a preset hardware fingerprint database for matching, and generate hardware type results and hardware characteristic parameters; a hardware abstraction layer module, used to construct a hardware abstraction layer, dynamically load matching hardware abstraction driver plugins according to the hardware type results, and virtualize the computing hardware into a unified logical computing unit through a standardized hardware abstraction interface; a dynamic routing module, used to dynamically route to the corresponding backend executor by querying a preset backend registry according to the hardware type results when the user triggers a large model inference task; and a compilation and optimization module, used to perform large model computation graph execution graph optimization and operator fusion in the large model inference task by calling the hardware compilation toolchain through the backend executor based on the hardware characteristic parameters and the standardized hardware abstraction interface corresponding to the unified logical computing unit.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor runs the computer program stored in the memory, the processor performs the steps of a multidimensional adaptive large model inference engine method as described in any one of claims 1-7.

10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it is used to implement the steps of a multidimensional adaptive large model inference engine method as described in any one of claims 1-7.

Images