AI model parallel loading and serial reasoning method and system under Serverless architecture

Abstract

The invention provides a layered artificial intelligence reasoning method and system based on Serverless, and is used for solving the problem of high cold start delay of a large-scale artificial intelligence model in a Serverless scene. The method comprises the following steps: firstly, dividing a to-be-reasoned large artificial intelligence model into a plurality of logic layers and physical layers, and constructing a workflow to coordinate an execution sequence and data transmission of each physical layer; after an inference request is received, a workflow engine creates Serverless function instances matched with resource requirements of all physical layers in parallel according to workflows, and parallel loading of models is achieved; and then coordinating serial execution of each physical layer function instance through a workflow to realize streamlined processing of model reasoning. Optimization mechanisms such as self-adaptive batch processing and cross-layer caching are further introduced, and the throughput and performance of the system are further improved. Compared with the prior art, the method has the advantages that the cold start delay of the large artificial intelligence model in the Serverless scene can be remarkably reduced, the resource utilization rate and the reasoning performance are improved, and the practical value is high.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-scale artificial intelligence model hierarchical loading and reasoning method and system based on a Serverless computing architecture. Background Art

[0002] Artificial intelligence technology has made great progress in recent years and has shown broad application prospects in many fields. In particular, with the continuous maturity of deep learning algorithms and the rapid development of computing resources, complex artificial intelligence models can solve more and more practical problems. Combining artificial intelligence technology with cloud computing to provide intelligent services in an on-demand and elastic manner has become an important trend in the industrialization of artificial intelligence. In the field of cloud computing, Serverless architecture is receiving more and more attention because of its feature that "users do not need to manage servers and only need to focus on core code." Serverless computing has important applications in artificial intelligence reasoning services.

[0003] At present, there are two main solutions for deploying artificial intelligence services in a Serverless environment. The first is to load the entire model into a single Serverless function instance, which completes the end-to-end reasoning process. The advantage of this solution is that it is simple to implement, but the disadvantage is that it is difficult to adapt to large and complex models. Especially for scenarios such as large-scale language models that require loading a large number of parameters, the resource limitations of a single function instance lead to extremely long loading times, which may even reach minutes, seriously affecting the service response speed and making it difficult to meet real-time requirements. The second solution is to deploy the model in an independent reasoning service, and the Serverless function obtains the reasoning results through API calls. However, this solution requires additional maintenance of the reasoning service, which increases the infrastructure cost. In addition, as an independent system, the reasoning service is difficult to enjoy the advantages of the Serverless platform such as automatic elastic scaling and pay-as-you-go, and it is difficult to optimize resource utilization and cost-effectiveness.

[0004] To solve the above problems, some studies have tried to optimize the Serverless reasoning process. For example, preloading the model into memory to avoid cold start; or using containerization technology to achieve fast startup of function instances. However, these methods sacrifice the elasticity and cost-effectiveness of Serverless to a certain extent, and are still insufficient when dealing with high-concurrency requests. In addition, most of the existing improvement solutions are targeted at specific machine learning frameworks and lack universality. With the continuous development of artificial intelligence models, a single optimization strategy is difficult to meet diverse needs.

[0005] An important development direction of AI reasoning services is to improve resource utilization in Serverless environments and reduce cold start latency of large models. In practical applications, model reasoning usually manifests as bursty and unbalanced loads, which places higher demands on the elasticity of the Serverless platform. In addition, with the penetration of AI in more fields, model reasoning services need to support larger-scale user concurrency and meet real-time constraints. Therefore, there is an urgent need for a general Serverless reasoning architecture that can automatically optimize resource allocation according to model characteristics, achieve efficient loading and low-latency reasoning of large models, retain the elasticity and cost advantages of Serverless, and meet the real-time and concurrency requirements of AI services, in order to meet the key challenges of the industrialization of AI services. Summary of the invention

[0006] The purpose of the present invention is to solve the problems faced in the prior art when deploying large-scale artificial intelligence models in a Serverless environment, such as extended model loading time and low resource utilization.

[0007] By introducing model layering and parallel loading mechanisms, the present invention can significantly reduce the cold start delay of large-scale model inference services and improve the resource utilization efficiency of the Serverless platform.

[0008] In addition, the present invention also aims to provide a general Serverless reasoning architecture that supports multiple artificial intelligence frameworks and model types to meet the scalability requirements of the industrialization of artificial intelligence services.

[0009] To achieve the above objectives, the present invention provides a Serverless-based hierarchical artificial intelligence reasoning method, which is characterized by: dividing the large artificial intelligence model to be inferred into multiple logical layers and physical layers, and building a workflow to coordinate the execution of each layer; after receiving the reasoning request, the workflow engine creates function instances that match the resource requirements of each physical layer in parallel according to the workflow to realize parallel loading of the model; then the serial execution of the function instances of each layer is coordinated through the workflow to complete the end-to-end reasoning process.

[0010] In some embodiments, the present invention also introduces optimization mechanisms such as adaptive batch processing and cross-layer caching to further improve system throughput and performance.

[0011] Specifically, the model partitioning process of the present invention comprehensively considers factors such as the topological structure of the model, data dependencies between operators, and execution overhead, so as to achieve a balance between parallelism and function granularity.

[0012] Furthermore, the physical layer partitioning adopts a heuristic algorithm based on graph partitioning, which reduces the inter-layer data transmission overhead by merging logical layers with related data and similar overhead.

[0013] Preferably, the workflow constructed by the present invention is in the form of a directed acyclic graph (DAG), and the execution order between layers is determined by topological sorting to maximize parallelism.

[0014] Optionally, the workflow engine adopts an event-driven architecture to trigger and coordinate the execution of hierarchical function instances through mechanisms such as message queues to improve system flexibility.

[0015] In one embodiment, the present invention adopts an independent automatic expansion and contraction strategy for each physical layer, and dynamically adjusts resource configuration according to request characteristics and load levels.

[0016] In certain embodiments, the batch processing optimization module of the present invention adopts an adaptive batch processing algorithm based on reinforcement learning to balance latency and throughput by dynamically adjusting the batch size.

[0017] In addition, the present invention can also simultaneously consider the priority and deadline of the reasoning request in batch optimization to support multiple service quality requirements.

[0018] In other embodiments, the cross-layer cache module adaptively caches and reuses intermediate calculation results based on cache replacement strategies such as LRU, thereby reducing repeated calculation overhead.

[0019] In a preferred embodiment, the serverless reasoning system of the present invention adopts a two-layer elastic architecture based on Kubernetes. At the bottom layer, the Kubernetes cluster manages the life cycle and resource allocation of function instances; at the top layer, the workflow engine orchestrates the hierarchical reasoning process based on Serverless frameworks such as Knative, and implements intelligent routing and load balancing through service grids such as Istio.

[0020] In another preferred embodiment, the present invention introduces a model compression mechanism based on knowledge distillation, which further reduces the resource overhead in the online reasoning stage by compressing a large model into multiple lightweight sub-models offline.

[0021] By adopting the above scheme, the present invention has the following beneficial effects: (1) Significantly reduce the cold start latency of large model inference services, shortening the response time from minutes to seconds; (2) Improve the resource utilization of the Serverless platform and save computing costs; (3) Supports a variety of artificial intelligence frameworks and models, with good versatility and scalability; (4) Automatically optimize resource allocation to achieve high concurrency and elastic scaling; (5) Integrate multiple performance optimization mechanisms to comprehensively improve system throughput and cost-effectiveness.

[0022] In summary, the present invention provides an efficient, economical, and flexible Serverless artificial intelligence reasoning solution, which is expected to accelerate the application and popularization of artificial intelligence technology in practical scenarios and has important industrial value and social significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is the overall architecture diagram of the present invention.

[0025] Figure 2 It is a flowchart of model partitioning.

[0026] Figure 3 Shows how an inference request is processed in the workflow.

[0027] Figure 4 The principles and strategies of adaptive batch optimization are explained.

[0028] Figure 5 The structure and working principle of the cross-layer cache module are demonstrated. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0031] Embodiment 1: This embodiment provides a hierarchical artificial intelligence reasoning method based on Serverless.

[0032] like Figure 1As shown in the figure, the method includes the following steps: First, the user uploads the large-scale artificial intelligence model to be deployed to the Serverless platform through the front-end interface or API interface. The platform automatically parses the model's topological structure, operator dependencies, computing overhead and other meta-information based on the model's file format and structure. Based on the obtained model meta-information, the model partitioning module uses a heuristic algorithm to stratify the model. Figure 2 As shown in Figure 1, the algorithm takes minimizing the data transmission overhead between layers as the optimization goal, while taking into account load balancing, that is, the computational overhead of each layer is as close as possible. Specifically, the algorithm first constructs a directed acyclic graph (DAG) based on data dependencies. The nodes in the graph represent the operators or submodules of the model, and the edges represent the data dependencies between nodes.

[0033] Then, the algorithm uses a greedy strategy to iteratively partition the graph into connected subgraphs, each of which is a logical layer. In order to minimize data transmission between layers, the algorithm selects the node with the largest degree in the graph as the seed node in each iteration, and uses breadth-first search (BFS) to find all nodes that are directly or indirectly connected to the node and have not been partitioned, and puts them into the same subgraph until all nodes are partitioned.

[0034] set up is a DAG constructed from model data dependencies, where is a node set, is an edge set. Define the degree of node v is the number of edges connected to v, subgraph The computational overhead is the sum of the computational costs of the nodes it contains. Then the partitioning algorithm can be expressed as: in represents a subgraph, and The amount of data transferred between The number of logical layers to be divided.

[0035] This optimization goal can be approximately solved by a greedy strategy, and its time complexity is .

[0036] Compared with the traditional graph partitioning method, the present invention introduces the concept of seed nodes, which preferentially partitions highly ordered nodes into the same subgraph, thereby significantly reducing inter-layer data transmission.

[0037] Based on the logical layer division, the physical layer division module further organizes the logical layer into physical layers. Each physical layer contains one or more logical layers, and the physical layers are connected through data dependency relationships.

[0038] like Figure 2 As shown in Figure 1, the physical layer partitioning adopts the classic Kernighan-Lin algorithm, which takes minimizing inter-layer communication as the optimization goal and considers the intra-layer computing parallelism. Similar to the logical layer partitioning, the Kernighan-Lin algorithm is also an iterative optimization algorithm. Each iteration continuously reduces the inter-layer communication overhead by exchanging node pairs in the two physical layers until a local optimum is reached.

[0039] In order to take into account the degree of parallelism, the present invention introduces an additional balancing factor on the basis of the original algorithm to avoid excessive differences in physical layer scales.

[0040] Specifically, let the DAG of the logical layer be ,Will The nodes are divided into Physical Layer . Define the communication overhead between physical layers for: The balance factor balance is: The optimization objective of the partition can be expressed as: in, is the preset upper limit of the balance factor, usually set to 1.5-2.0. In each iteration, the algorithm enumerates all possible node pair exchange operations and selects the operation that can minimize the objective function value and does not violate the balance constraint until it cannot be optimized further. After introducing the balance factor, the time complexity of the Kernighan-Lin algorithm is still .

[0041] Compared with traditional methods, this algorithm minimizes inter-layer communication while avoiding the decrease in parallel efficiency caused by the disparity in physical layer scale, achieving a balance between communication overhead and parallelism.

[0042] After the model layering is completed, the workflow construction module automatically generates a workflow in the form of a DAG based on the physical layer information. In the workflow, each node corresponds to a physical layer, and the edges between nodes represent the data dependencies and execution order between layers. The workflow engine coordinates the execution of each physical layer function instance based on the workflow DAG.

[0043] When the system receives an inference request, the workflow engine first parses the service quality parameters of the request, mainly including the expected response time and computing cost budget. Then, the engine combines the current system load status and the computing characteristics of the physical layer, and uses an optimization algorithm to determine the best resource configuration for each physical layer, including the number of instances.

[0044] The optimization algorithm is constrained by meeting the service quality requirements and aims to minimize resource costs. It comprehensively considers factors such as instance performance, concurrency, and cold start time to solve the optimal resource allocation solution. Compared with fixed resource configuration, this optimization mechanism can be dynamically adjusted for different requests, maximizing resource utilization while ensuring service quality.

[0045] After determining the resource configuration, the workflow engine asynchronously creates a function instance that meets the resource requirements through the API of the Serverless platform and loads the physical layer model into the instance. Thanks to the layered method of the present invention, a large model is divided into multiple lightweight sub-models, and the cold start time of a single instance is greatly shortened.

[0046] In addition, the creation and loading of multiple instances can be performed concurrently on the Serverless platform, further improving the overall loading speed of the model. After the instance is loaded, a ready signal is sent to the workflow engine. After receiving the ready signals of all instances, the workflow engine begins to coordinate multiple instances to perform online reasoning. The reasoning request is first sent to the entry node of the workflow DAG. After receiving the input data, each node calls the corresponding instance to perform calculations and sends the calculation results to the downstream nodes. Data is transmitted between nodes through high-performance distributed memory or message queues to reduce communication overhead. The entire reasoning process is carried out in a pipeline manner to maximize parallelism. When the termination node completes the calculation, the reasoning result is returned to the user.

[0047] Compared with the existing Serverless reasoning methods, this invention adopts model layering and parallel loading, which can reduce the cold start time of large models from minutes to seconds, and reduce the end-to-end delay by 1-2 orders of magnitude. Compared with the traditional model partitioning method, this invention introduces new optimization goals and constraints in the logical layer and physical layer partitioning, while minimizing communication overhead, taking into account load balancing and parallelism, and can better adapt to the elastic resource supply in the Serverless environment.

[0048] Example 2: Based on Example 1, this example further proposes an adaptive batch processing method for Serverless reasoning.

[0049] This method is implemented through a batch optimization module, which can significantly improve the throughput and resource utilization of inference services.

[0050] Specifically, if Figure 4 As shown in the figure, after the batch optimization module receives the inference request monitored by the receiving port, it will not immediately trigger the workflow execution, but cache the request in the corresponding queue according to its service quality requirements. Each queue corresponds to a service quality level, which is defined by two dimensions: the upper limit of the response time and the computing power cost budget. The length of the queue is dynamically adjusted and is calculated by the batch optimization module based on indicators such as the arrival rate of the request and the average execution time of the batch. When the length of a queue reaches the current batch threshold, the optimization module packages the requests in the queue into a batch task and submits it to the workflow engine for execution. The execution process of a batch task is similar to that of an ordinary inference request. The difference is that the function instance triggered by the batch task can perform batch inference on multiple requests, thereby diluting the instance's call overhead and resource occupancy cost.

[0051] In order to adaptively adjust the batch size under different load levels to achieve the best balance between latency and throughput, the batch optimization module adopts a dynamic batch decision algorithm based on reinforcement learning. For the state space, in batch adjustment mode is the action space, with weighted response time and throughput as the reward function R, and the batch decision strategy is continuously optimized through policy iteration , so as to maximize the long-term cumulative reward. Formally, this reinforcement learning problem can be modeled as a Markov decision process (MDP): State Space: Action Space: State transition probability: Reward function: Strategy: The goal is to maximize the long-term cumulative reward (or discounted cumulative reward): or in, and The weight coefficient to balance response time and throughput, is the discount factor, which ranges from 0 to 1. The algorithm continuously updates the state value function through methods such as value iteration or policy gradient. Or action-value function , and then optimize the strategy .

[0052] Compared with traditional fixed batch or rule-based adjustment methods, reinforcement learning can autonomously learn and optimize according to the real-time feedback of the system, without the need to manually set thresholds and rules, and can adapt to the dynamic changes of request distribution and system status. Batch optimization combined with the hierarchical reasoning of Example 1 can achieve more fine-grained resource optimization. Requests with different service quality levels are separated in the batch queue. High-priority requests can use smaller batches and be allocated to the physical layer with more computing resources to ensure their response time; low-priority requests can use larger batches and be executed in the physical layer with fewer resources to improve resource utilization. The physical layer can dynamically adjust the number and type of instances according to the batch granularity to minimize resource usage costs while meeting service quality requirements.

[0053] This embodiment makes full use of an important feature of the artificial intelligence reasoning service, that is, the response time requirements of most reasoning requests are far lower than the maximum throughput capacity of the model, which provides space for request merging.

[0054] Traditional batch processing uses fixed-size batches, which often results in longer response times when the load is low and less than maximum throughput when the load is high. However, the adaptive batch processing based on reinforcement learning in this embodiment can intelligently balance latency and throughput according to the characteristics of the request and the state of the system to find the optimal batch processing granularity.

[0055] At the same time, the function instances triggered by batch requests can be automatically scaled down when idle, further improving resource utilization. The present invention provides a new optimization idea for the performance, cost and scalability of artificial intelligence reasoning services.

[0056] Example 3: Based on Example 1 and Example 2, this example further provides a Serverless-based hierarchical artificial intelligence reasoning system to support the implementation of the aforementioned hierarchical reasoning method and adaptive batch processing method. Figure 1 As shown, the system mainly includes the following functional modules: Model partitioning module: This module is responsible for receiving large artificial intelligence models uploaded by users, and dividing them into multiple logical layers and physical layers according to the model's topological structure, operator dependencies, computing overhead and other characteristics. The partitioning process is implemented through heuristic algorithms and graph partitioning algorithms, and the specific partitioning method is described in Example 1. After the model is partitioned, the module passes the partitioning results to the workflow construction module.

[0057] Workflow construction module: This module automatically generates a workflow in the form of a directed acyclic graph (DAG) based on the model partitioning results. In the workflow, each node corresponds to a physical layer, and the edges between nodes represent the data dependencies between layers. This module is also responsible for the persistent storage and version management of the workflow. The constructed workflow description file is submitted to the global management module for scheduling the execution of the workflow.

[0058] Global management module: This module is the central control center of the entire system. It is responsible for receiving and parsing the inference requests submitted by users, and determining the corresponding batch processing strategy and resource allocation plan according to the service quality requirements of the requests. This module monitors the system's load status and resource usage in real time, and dynamically adjusts the batch processing granularity and physical layer instance specifications to meet the service quality constraints. At the same time, this module is also responsible for collecting the execution indicators of the inference tasks, such as latency, throughput, resource utilization, etc., to optimize the batch processing strategy and resource supply strategy.

[0059] Resource management module: This module is responsible for applying for and managing the computing resources of the Serverless platform based on the resource allocation plan formulated by the global management module. Specifically, this module creates function instances that meet specific specifications in the specified resource pool through the API provided by the Serverless platform, and loads the divided physical layer model into the instance. During the execution of the inference task, the module will dynamically adjust the number and specifications of instances according to load changes and the instructions of the global management module, and recycle or release instances that are no longer needed to achieve elastic supply and on-demand use of resources. When the system is idle for a long time, the module will automatically reduce the number of instances to a minimum to reduce resource usage costs.

[0060] Inference execution module: This module coordinates multiple physical layer function instances to complete end-to-end inference tasks. The inference request is first submitted to the instance of the entry physical layer. After completing the calculation of this layer, the instance writes the intermediate results to the distributed storage and triggers the execution of the downstream physical layer instance through the message queue. This process is performed recursively until the termination layer instance completes the calculation. The inference results are written back to the distributed storage and returned to the user by the global management module. The physical layer instances are interconnected through a high-speed network, and the data between layers is transmitted in a pipeline manner through shared storage or message queues to minimize communication overhead.

[0061] The hierarchical reasoning system also provides a set of auxiliary tools to support the development and deployment of reasoning tasks, including but not limited to: Web portal, used to visually manage models, build workflows, submit inference requests, monitor task execution status, etc. SDK, which is used to convert models trained by various machine learning frameworks into intermediate formats supported by the system; CLI, used by users of different roles to operate the system through the command line; Monitoring and logging platform for real-time collection, storage, analysis, and visualization of system operation indicators and event information.

[0062] This embodiment provides a complete systematic solution based on the hierarchical reasoning method of embodiment 1 and the adaptive batch processing method of embodiment 2.

[0063] Through layering and parallelism, the system can reduce the inference time of large models from minutes to seconds; through batch processing and resource scaling, the system can achieve optimal latency and throughput in high-concurrency scenarios.

[0064] More importantly, this embodiment adopts a full-stack Serverless architecture, from model loading to resource management to task scheduling, all links can be automated and used on demand, greatly reducing the user's usage threshold and operation and maintenance costs. Users do not need to worry about the configuration and scheduling of underlying resources. They only need to submit inference requests through the portal or API, and the system can automatically complete the end-to-end inference process according to the service quality requirements.

[0065] In addition to artificial intelligence reasoning, the present invention can also be applied to layered Serverless processing of other computing-intensive tasks, such as scientific computing, big data analysis, image rendering, etc.

[0066] The key to the present invention is to divide the original computing process into layers and load it in parallel, finely orchestrate the data transmission and execution process between layers, and design optimization mechanisms such as batch processing and caching based on the characteristics of the Serverless environment.

[0067] Therefore, the core idea of ​​the present invention is universal and can introduce the elasticity and efficiency of Serverless to traditional computing tasks.

[0068] In summary, the Serverless hierarchical reasoning method and system provided by the present invention has the advantages of low cold start latency, high resource utilization, strong scalability, good versatility, etc., which can significantly improve the performance and efficiency of artificial intelligence reasoning services and reduce user usage thresholds and costs. With the continuous popularization of artificial intelligence in all walks of life, the present invention is expected to enable more application scenarios, promote the engineering and industrialization of artificial intelligence technology, and inject new momentum into the economy and society.

[0069] For those skilled in the art, many improvements and modifications can be made without violating the principles of the present invention, all of which fall within the scope of protection of the present invention. For example, the present invention can be combined with virtualization, edge computing and other technologies to further expand the application scenarios and effects of intelligent scheduling.

Claims

1. A Serverless-based hierarchical artificial intelligence reasoning method is used to solve the problem of high cold start latency of large artificial intelligence models in Serverless scenarios, characterized by: The following steps are involved: (1) receiving a large artificial intelligence model to be inferred and dividing the model into multiple logical layers; (2) dividing the physical layer based on the logical layer, each physical layer including at least one logical layer; (3) Build a workflow to coordinate the execution order and data transfer of each physical layer; (4) After receiving the inference request, the workflow engine creates Serverless function instances that match the resource requirements of each physical layer in parallel according to the workflow to achieve parallel loading of the model; (5) The serial execution of each physical layer function instance is coordinated through the workflow to achieve pipelined processing of model reasoning.

2. The method according to claim 1, characterized in that When dividing the logical layers in step (1), factors such as the model's topological structure, data dependencies between operators, and execution overhead are comprehensively considered.

3. The method according to claim 1, characterized in that: When dividing the physical layer in step (2), multiple logical layers with data dependencies and matching overheads are merged.

4. The method according to claim 1, characterized in that: The resource requirements in step (4) include CPU, GPU, memory, and the maximum number of concurrent requests.

5. The method according to claim 1, characterized in that The number of function instances created in step (4) and their resource specifications can be dynamically adjusted according to the load.

6. The method according to claim 1, characterized in that Further including: Adaptively batch input data to merge multiple inference requests to increase throughput.

7. The method according to claim 1, characterized in that Further including: A cache mechanism is introduced between adjacent physical layers to store intermediate calculation results for reuse by subsequent layers.

8. The method according to any one of claims 1 to 7, characterized in that: The artificial intelligence models include deep neural networks, decision trees, support vector machines, etc.

9. A Serverless-based hierarchical artificial intelligence reasoning system is used to solve the problem of high cold start latency of large artificial intelligence models in Serverless scenarios, characterized by: include: A model partitioning module is used to divide the artificial intelligence model to be inferred into multiple logical layers and physical layers; A workflow building module, used to build workflows based on dependencies between physical layers; The function generation module is used to create Serverless function instances that match the resource requirements of each physical layer in parallel according to the workflow after receiving the inference request, so as to realize parallel loading of the model; The reasoning process orchestration module is used to coordinate the serial execution of each physical layer function instance through workflow to achieve pipelined processing of model reasoning.

10. The system according to claim 9, characterized in that Further including: Batch optimization module, used to perform adaptive batch processing on input data; The cross-layer cache module is used to cache intermediate calculation results between adjacent physical layers.

Images