A method for optimizing AI model deployment to improve hardware throughput

Abstract

The present invention provides a method for optimizing the deployment of AI models to improve hardware throughput, wherein the method includes: segmenting the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain a plurality of corresponding model subgraphs; compiling the target model and the model subgraphs at different optimization levels according to the environment configuration to obtain a plurality of corresponding executable instructions; real-time monitoring of the resource usage of the current scene; determining the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation of the target model corresponding to the current scene; executing the executable instructions corresponding to the target task, and recording the execution parameters of the executable instructions. This method realizes the efficient use of computing resources through model segmentation and dynamic task allocation, while ensuring the flexibility and performance of model execution, and is suitable for the optimized deployment of various deep learning models in resource-constrained or multi-task environments.

Description

Technical Field

[0001] The present invention relates to the field of model deployment technology, and in particular to a method for optimizing the deployment of an AI model for improving hardware throughput. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning models have been widely used in many fields such as image recognition, natural language processing, and speech synthesis. The complexity and scale of these models continue to increase, and the demand for computing resources is also growing. In the actual deployment process, deep learning models often face many challenges, such as limited computing resources, heterogeneous hardware, and high real-time requirements.

[0003] In the prior art, deep learning models are usually deployed in a single model deployment mode. This method treats the entire deep learning model as an indivisible whole and compiles and executes it on the target hardware platform. This deployment mode has the advantages of simple implementation and convenient management, so it is widely used in early deep learning applications.

[0004] However, as application scenarios become increasingly complex and diverse, the limitations of a single model deployment method are gradually becoming apparent. The main drawback is that a single large-scale model often monopolizes a large amount of computing resources, resulting in low hardware resource utilization. In particular, on different computing platforms, different types of computing units (such as CPU, GPU, TPU, etc.) have different characteristics, and a single model cannot fully utilize all types of computing resources. In this case, some hardware resources may be idle, while other resources may be overloaded, ultimately resulting in overall performance failing to meet expectations. Summary of the invention

[0005] The main purpose of the present invention is to solve the technical problem that a single existing model monopolizes a large amount of computing resources, resulting in low utilization of hardware resources.

[0006] A first aspect of the present invention provides a method for optimizing deployment of an AI model to improve hardware throughput, the method comprising:

[0007] According to the preset environment configuration and segmentation requirements, the target model corresponding to the current scene is segmented to obtain corresponding multiple model sub-graphs;

[0008] Compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions;

[0009] Real-time monitoring of resource usage of the current scenario;

[0010] Determine the target task of the current scenario and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scenario;

[0011] Execute the executable instruction corresponding to the target task, and record the execution parameters of the executable instruction corresponding to the target task.

[0012] Optionally, in a first implementation of the first aspect of the present invention, the target model corresponding to the current scene is segmented according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model subgraphs, including:

[0013] Determining the priority level of each target model corresponding to the current scene according to the environment configuration and the segmentation requirement;

[0014] Determine the division granularity of each target model corresponding to the current scene according to the priority level;

[0015] The corresponding model is segmented according to the division granularity to obtain corresponding multiple model subgraphs.

[0016] Optionally, in a second implementation of the first aspect of the present invention, the performing model segmentation on the corresponding model according to the division granularity to obtain the corresponding multiple model subgraphs includes:

[0017] Performing operator analysis on the target model to obtain an operator sequence in the target model;

[0018] Performing hardware adaptability analysis on operators in the operator sequence to obtain hardware analysis results;

[0019] Dividing the operators in the operator sequence into groups according to the hardware analysis result and the division granularity to obtain a plurality of operator combinations;

[0020] Generate corresponding multiple model subgraphs according to the multiple operator combinations.

[0021] Optionally, in a third implementation of the first aspect of the present invention, the compiling of the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions includes:

[0022] Performing intermediate code generation processing on the target model and the model subgraph to obtain corresponding intermediate representations;

[0023] Optimizing the intermediate representation according to the environment configuration to obtain an optimized intermediate code;

[0024] Performing target code generation processing on the optimized intermediate code to obtain assembly code;

[0025] The assembly code is processed into a machine code to obtain corresponding multiple executable instructions.

[0026] Optionally, in a fourth implementation manner of the first aspect of the present invention, determining the target task of the current scene and the executable instruction corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene includes:

[0027] Analyzing and processing the resource usage to obtain currently available hardware resource information;

[0028] Performing task matching on multiple model subgraphs of the target model according to the requirements of the current scene to obtain the target task of the current scene;

[0029] According to the target task and the available hardware resource information, selecting from the multiple executable instructions to obtain an executable instruction corresponding to the target task;

[0030] The target task and the executable instruction are associated with each other to obtain the target task of the current scene and the executable instruction corresponding to the target task.

[0031] Optionally, in a fifth implementation manner of the first aspect of the present invention, executing the executable instruction corresponding to the target task and recording the execution parameters of the executable instruction corresponding to the target task includes:

[0032] Parsing the executable instructions corresponding to the target task to obtain a corresponding instruction execution sequence;

[0033] Executing the executable instructions corresponding to the target task one by one according to the instruction execution sequence to obtain the execution result of each executable instruction corresponding to the target task;

[0034] The execution results are monitored and recorded in real time to obtain the corresponding execution parameters.

[0035] Optionally, in a sixth implementation of the first aspect of the present invention, the parsing and processing the executable instructions corresponding to the target task to obtain the corresponding instruction execution sequence includes:

[0036] Performing a demand analysis on the target task to obtain the timeliness requirements of different target models corresponding to the target task in the current scenario;

[0037] Prioritize the executable instructions corresponding to the model subgraphs of each target model according to the timeliness requirement to obtain a preliminary execution order;

[0038] Estimating the running time of the executable instructions in the preliminary execution sequence to obtain an estimated execution time of each executable instruction;

[0039] The preliminary execution order is adjusted according to the estimated execution time and the timeliness requirement to obtain a corresponding instruction execution sequence.

[0040] A second aspect of the present invention provides a device for optimizing the deployment of an AI model to improve hardware throughput, and the device for optimizing the deployment of an AI model to improve hardware throughput includes:

[0041] The model segmentation module performs model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain corresponding multiple model sub-graphs;

[0042] A model compilation module, used for compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions;

[0043] A resource scheduling module, used to monitor the resource usage of the current scenario in real time;

[0044] A task allocation module, used to determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene;

[0045] The performance recording module is used to execute the executable instructions corresponding to the target task and record the execution parameters of the executable instructions corresponding to the target task.

[0046] The third aspect of the present invention provides an apparatus for optimizing the deployment of AI models to improve hardware throughput, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected through lines; the at least one processor calling the instructions in the memory so that the apparatus for optimizing the deployment of AI models to improve hardware throughput performs the steps of the above-mentioned method for optimizing the deployment of AI models to improve hardware throughput.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the steps of the above-mentioned method for optimizing deployment of an AI model to improve hardware throughput.

[0048] The above-mentioned method for optimizing the deployment of AI models to improve hardware throughput is to segment the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain multiple corresponding model subgraphs; compile the target model and model subgraphs at different optimization levels according to the environment configuration to obtain multiple corresponding executable instructions; monitor the resource usage of the current scene in real time; determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation of the target model corresponding to the current scene; execute the executable instructions corresponding to the target task and record the execution parameters of the executable instructions. This method realizes the efficient use of computing resources through model segmentation and dynamic task allocation, while ensuring the flexibility and performance of model execution, and is suitable for the optimized deployment of various deep learning models in resource-constrained or multi-task environments.

[0049] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a first embodiment of a method for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a second embodiment of a method for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of an embodiment of a device for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of an embodiment of a device for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0057] To facilitate understanding of this embodiment, a method for optimizing the deployment of an AI model for improving hardware throughput disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0058] 101. According to the preset environment configuration and segmentation requirements, the target model corresponding to the current scene is segmented to obtain corresponding multiple model sub-graphs;

[0059] In one embodiment of the present invention, the target model corresponding to the current scene is segmented according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model sub-graphs, including: determining the priority level of each target model corresponding to the current scene according to the environment configuration and the segmentation requirements; determining the division granularity of each target model corresponding to the current scene according to the priority level; and segmenting the corresponding model according to the division granularity to obtain the corresponding multiple model sub-graphs.

[0060] Specifically, we first need to consider the preset environment configuration and segmentation requirements. The environment configuration usually includes hardware resource information, such as the available processor type, memory capacity, storage space, etc., while the segmentation requirements involve factors such as performance goals and resource utilization. Based on this information, the system analyzes each target model in the current scene to determine their priority. The priority determination process involves many aspects, including the importance of the model, execution frequency, real-time requirements, etc. For example, in an autonomous driving scenario, an obstacle detection model may be given a higher priority because it is directly related to driving safety. Once the priority of each target model is determined, the system will determine the granularity of each model based on these levels. Models with higher priority levels usually have coarser granularity, which means that they will be divided into larger subgraphs. The reason for this is that high-priority models usually require more computing resources and faster response times. In contrast, models with lower priority are divided into finer granularity to make more flexible use of the remaining computing resources. The determination of granularity is a trade-off process. Coarse-grained subgraphs can better maintain the overall performance of the model and reduce the communication overhead between subgraphs, but may lead to uneven resource utilization. Fine-grained subgraphs provide greater flexibility, allowing the system to allocate resources more finely, but may increase scheduling complexity and communication overhead. The system needs to select the most suitable granularity for each model based on the preset environment configuration and priority level. After determining the granularity of the partition, the system will perform the actual partitioning operation on each model. This process usually involves analyzing and reorganizing the model calculation graph. The system will identify the key nodes and data flows in the model, and then divide the model into multiple subgraphs according to the predetermined granularity criteria. For coarse-grained partitioning, multiple adjacent layers or operations may be combined into a subgraph; for fine-grained partitioning, even a single layer or operation may be used as a subgraph. During the partitioning process, the system also needs to consider the dependencies and data flows between subgraphs to ensure that the partitioned subgraphs can be correctly executed in series to maintain the functional integrity of the original model. At the same time, it is necessary to evaluate the execution efficiency of each subgraph on the target hardware, and some local optimizations, such as operator fusion, may be performed to improve the execution performance of the subgraph. Ultimately, this process will produce a series of model subgraphs, each of which corresponds to a part of the original model's functionality. These subgraphs will serve as the basic units for subsequent optimization and deployment, providing the system with more flexible resource allocation and task scheduling capabilities.

[0061] Furthermore, the model segmentation is performed on the corresponding model according to the division granularity to obtain the corresponding multiple model subgraphs, including: performing operator analysis on the target model to obtain an operator sequence in the target model; performing hardware adaptability analysis on the operators in the operator sequence to obtain hardware analysis results; grouping the operators in the operator sequence according to the hardware analysis results and the division granularity to obtain multiple operator combinations; and generating corresponding multiple model subgraphs according to the multiple operator combinations.

[0062] Specifically, we first conduct an in-depth operator analysis of the target model. This process involves a comprehensive analysis of the model structure, identifying and extracting all the operators that make up the model. Operator analysis usually starts from the input layer of the model and goes deeper layer by layer until the output layer, identifying the operations in each layer as corresponding operators. These operators may include convolution, pooling, activation functions, fully connected layers, etc. During the analysis process, we must not only identify the type of operator, but also record the specific parameters of the operator, such as convolution kernel size, step size, padding method, etc. Ultimately, this step will generate a complete operator sequence that reflects the computational process of the entire model. Next, hardware adaptability analysis is performed on each operator in the operator sequence. This step aims to evaluate the execution efficiency of each operator on the target hardware platform. Hardware adaptability analysis considers multiple factors, including the computational complexity of the operator, memory access pattern, and characteristics of the target hardware (such as processor type, memory hierarchy, dedicated acceleration unit, etc.). During the analysis process, the execution time, energy consumption, resource usage, and other indicators of each operator on different hardware units are estimated or measured. For example, some convolution operations may be more efficient on the GPU, while some simple activation functions may be more suitable for execution on the CPU. Through this step, a detailed hardware analysis result is obtained, including the optimal execution unit and expected performance indicators for each operator. Based on the hardware analysis results and the predetermined partitioning granularity, the system begins to group the operators in the operator sequence. The partitioning process needs to balance multiple goals, including maximizing hardware utilization, minimizing communication overhead between subgraphs, and meeting the predetermined granularity requirements. For coarse-grained partitioning, the system tends to group together multiple consecutive operators that are efficient on the same hardware unit. For fine-grained partitioning, the operator sequence may be divided into smaller units, or even a single operator may be retained as a group. During the partitioning process, the system also needs to consider the data dependencies between operators to ensure that the divided groups can be correctly executed in series. In addition, the partitioning process also considers some optimization opportunities, such as operator fusion. For example, if several consecutive operators (such as convolution, batch normalization, and activation functions) can be efficiently executed on the same hardware unit and can significantly reduce memory access after fusion, then these operators will be divided into the same group. In this way, the system generates multiple operator combinations, each of which represents a local computing unit of the model. Finally, multiple corresponding model subgraphs are generated based on these operator combinations. This step involves converting each operator combination into an independently executable computational graph segment. The generation process includes determining the input and output interfaces of the subgraph, allocating the necessary memory space, generating the topological structure of the subgraph, etc. At the same time, metadata needs to be created for each subgraph to record information such as the operators it contains, the expected execution hardware, and the estimated computational complexity. This information will play an important role in the subsequent scheduling and execution process.

[0063] 102. Compile the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions;

[0064] In one embodiment of the present invention, the compiling of the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions includes: performing intermediate code generation processing on the target model and the model subgraph to obtain corresponding intermediate representations; optimizing the intermediate representation according to the environment configuration to obtain optimized intermediate codes; performing target code generation processing on the optimized intermediate codes to obtain assembly codes; and performing machine code generation processing on the assembly codes to obtain corresponding multiple executable instructions.

[0065] Specifically, the target model and model subgraphs are first processed for intermediate code generation. This process starts with a high-level representation of the model, usually a framework-specific computational graph or model description file. The compiler traverses these high-level representations and converts each operation or layer into a common intermediate representation (IR). The intermediate representation is an abstract representation independent of the source language and the target machine. It captures the essence of the computation while retaining enough information to support subsequent optimization and code generation. At this stage, the compiler generates a control flow graph (CFG) and a data dependency graph (DDG), which reflect the logical relationship and data flow between operations. Next, the intermediate representation is optimized according to the environment configuration. The environment configuration contains information such as the characteristics of the target hardware, available memory resources, and optimization level. The optimization process involves multiple stages and multiple technologies. The first is general optimization, such as constant folding, dead code elimination, loop unrolling, etc., which do not depend on specific hardware. Then there are architecture-related optimizations, such as instruction scheduling, register allocation, vectorization, etc., which take into account the characteristics of the target hardware. For deep learning models, some specific optimizations are also performed, such as operator fusion, memory layout optimization, quantization, etc. The optimization process is iterative, and the effect of each optimization step is evaluated after each optimization step to ensure that the optimization has indeed brought performance improvement. Different optimization levels will apply different degrees of optimization. Higher optimization levels will try more aggressive optimization strategies, but may also increase compilation time. The optimized intermediate code then enters the target code generation stage. In this stage, the compiler will map the optimized intermediate representation to the instruction set of the target architecture. This process involves subtasks such as instruction selection, addressing mode selection, and register allocation. Instruction selection aims to select the most suitable machine instruction sequence for each IR operation. Addressing mode selection determines how to access operands most efficiently. Register allocation determines which values ​​are stored in registers and which need to be stored in memory. The output of this stage is assembly code, which is a human-readable low-level code representation with one machine instruction per line. The last step is to convert the assembly code into machine code. This process is relatively straightforward. The assembler will translate each assembly instruction into the corresponding binary machine instruction. At this stage, tasks such as symbol resolution and relocation are also handled to ensure that the generated machine code can be loaded and executed correctly. For different target hardware or execution environments, different versions of machine code may be generated to adapt to different instruction set architectures or runtime environments.

[0066] 103. Real-time monitoring of resource usage in the current scene;

[0067] In one embodiment of the present invention, real-time monitoring of resource usage in the current scene is a complex and continuous process involving data collection and analysis at multiple levels. First, the system needs to set monitoring points at the underlying hardware and operating system levels. This usually includes the collection of key indicators such as CPU usage, memory usage, storage I / O, and network bandwidth. For the CPU, monitoring not only includes overall usage, but also needs to be broken down into the load of each core. Memory monitoring requires tracking the usage of physical and virtual memory, including page swap frequency. Storage I / O monitoring focuses on read and write speeds and queue lengths, while network monitoring tracks packet throughput and latency. These raw data are collected through interfaces provided by the operating system or dedicated monitoring drivers. For example, in a Linux system, this information can be obtained by reading the / proc file system. In a Windows system, the performance counter API may be used. For specialized deep learning acceleration hardware, such as GPUs or TPUs, specific APIs are also required to obtain their usage. For example, NVIDIA's NVML library can be used to monitor GPU usage, memory usage, temperature and other parameters. The collected raw data is then transmitted to a central monitoring module. This module is responsible for data preprocessing, aggregation and preliminary analysis. Preprocessing includes data cleaning, formatting, and standardization to ensure that data from different sources can be processed uniformly. The aggregation process compresses the data collected at a high frequency into a meaningful time series, such as aggregating the CPU usage data collected every second into an average value per minute. The preliminary analysis includes calculating some derived indicators, such as CPU load balance, memory pressure index, etc. In order to achieve real-time monitoring, this process needs to be efficient and low-overhead. A circular buffer is usually used to store the most recent monitoring data, and the old data will be overwritten by the new data. At the same time, multi-threading or asynchronous I / O technology is used to ensure that data collection does not block the main computing tasks. In addition, an adaptive sampling rate strategy is adopted to reduce the sampling frequency when the system load is light and increase the sampling frequency when the load is heavy to balance the monitoring accuracy and system overhead. The monitoring data also needs to be associated with the currently running tasks. This requires setting hooks at the task scheduling and execution level to record the start and end time of each task and the resources they use. For example, when a model subgraph starts to execute, the system will mark the GPU number it uses, the start time, and other information. This information is combined with the underlying resource usage data to build a resource usage profile accurate to the task level.

[0068] 104. Determine the target task of the current scenario and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scenario;

[0069] In one embodiment of the present invention, determining the target task of the current scene and the executable instructions corresponding to the target task based on the resource usage and the operation status of the target model corresponding to the current scene includes: analyzing and processing the resource usage to obtain currently available hardware resource information; performing task matching on multiple model subgraphs of the target model according to the requirements of the current scene to obtain the target task of the current scene; selecting from the multiple executable instructions based on the target task and the available hardware resource information to obtain the executable instructions corresponding to the target task; and associating the target task and the executable instructions to obtain the target task of the current scene and the executable instructions corresponding to the target task.

[0070] In one embodiment of the present invention, the target task of the current scene and the executable instructions corresponding to the target task can also be determined by performing dynamic timing analysis on the resource usage and the operation of the target model to obtain a multi-dimensional time-varying feature sequence, performing state update processing on a preset deep reinforcement learning model according to the multi-dimensional time-varying feature sequence to obtain a state representation of the current scene, performing task strategy reasoning processing on the state representation to obtain a candidate task decision tree, performing Monte Carlo tree search processing on the candidate task decision tree to obtain an optimal task path, performing adaptive instruction generation processing on key node tasks in the optimal task path to obtain a set of executable instructions corresponding to the target task.

[0071] Specifically, first, dynamic time series analysis is performed on resource usage and the operation of the target model. This step uses sliding window technology to divide the continuous time series data into several overlapping time segments. For each time segment, the system applies multi-scale wavelet transform to extract features at different time scales. These features include but are not limited to the fluctuation pattern of resource utilization, the distribution characteristics of task execution time, and the time series changes of the computing load of each layer of the model. Through this processing, the original time series data is converted into a multi-dimensional time-varying feature sequence, each dimension represents an important aspect of the system state. Next, this multi-dimensional time-varying feature sequence is used to update the state of the preset deep reinforcement learning model. This deep reinforcement learning model adopts a two-stream network structure, one stream processes the time series features of resource usage, and the other stream processes the state information of the model operation. The outputs of the two streams are fused through the attention mechanism to generate a comprehensive state representation. This state representation not only contains the system state at the current moment, but also encodes historical information and predictions of future trends. The model update adopts the asynchronous advantage Actor-Critic (A3C) algorithm, which allows simultaneous learning in multiple parallel environments, accelerates the learning process and improves the generalization ability of the model. After obtaining the state representation, the system performs task policy reasoning to generate a candidate task decision tree. This step uses a policy network based on a graph neural network. The network represents the dependencies between tasks as a graph structure, where each node represents a potential task and the edges represent the dependencies or resource competition between tasks. The policy network passes messages on this graph, taking into account the local features and global context of the nodes, and generates an execution probability for each task node. Based on these probabilities, the system constructs a candidate task decision tree, where each layer of the tree represents a decision time step and each branch represents a possible task execution sequence. After generating the candidate task decision tree, the system uses the Monte Carlo Tree Search (MCTS) method to determine the optimal task path. The implementation of MCTS uses the Upper Confidence Bound (UCB) strategy for node selection, balancing exploration and exploitation. In the simulation phase, the system uses a lightweight environment model to quickly evaluate the potential results of different decision sequences. This environment model is obtained by modeling historical execution data and can simulate the impact of different task combinations on the system state. Each iteration of MCTS updates the value estimates of nodes in the decision tree, eventually converging to an optimal task path that balances short-term benefits and long-term impacts. After determining the optimal task path, the system performs adaptive instruction generation processing on key node tasks in the path. This process first identifies the key nodes in the path, which are usually computationally intensive or resource-sensitive tasks. For each key node, the system calls a neural network compiler.This compiler accepts the computational graph description of the task and the current hardware status as input, and outputs an instruction sequence optimized for specific hardware. The compiler uses an instruction scheduling model based on reinforcement learning, which can dynamically adjust the instruction generation strategy according to the real-time status of the hardware. For example, when the GPU memory is close to saturation, it will generate more memory optimization instructions; when the CPU load is light, it will generate more parallel execution instructions. This adaptive instruction generation not only considers the optimization of a single task, but also the mutual influence between tasks. The system uses an inter-task interference prediction model to predict the resource competition and performance impact that may arise when different tasks are executed in parallel. Based on these predictions, the instruction generator will appropriately adjust the instruction sequence, such as inserting synchronization points or adjusting memory access patterns to reduce negative interference between tasks. Finally, the system combines the generated instruction sequences into an executable instruction set. This set is not static, but contains multiple conditional branches and dynamic adjustment points. In the actual execution process, the system can dynamically switch different instruction branches or adjust parameters according to the execution status monitored in real time, further improving the flexibility and efficiency of execution.

[0072] 105. Execute the executable instruction corresponding to the target task, and record the execution parameters of the executable instruction corresponding to the target task.

[0073] In one embodiment of the present invention, executing the executable instructions corresponding to the target task and recording the execution parameters of the executable instructions corresponding to the target task include: parsing the executable instructions corresponding to the target task to obtain a corresponding instruction execution sequence; executing the executable instructions corresponding to the target task one by one according to the instruction execution sequence to obtain the execution result of each executable instruction corresponding to the target task; and monitoring and recording the execution results in real time to obtain corresponding execution parameters.

[0074] In one embodiment of the present invention, the process of executing the executable instructions corresponding to the target task and recording the execution parameters can also be carried out by performing dependency analysis on the executable instructions to obtain an instruction dependency graph, dynamically allocating computing resources according to the instruction dependency graph to obtain a resource scheduling plan, performing instruction-level parallelism analysis on the resource scheduling plan to obtain a parallel execution strategy, grouping the executable instructions according to the parallel execution strategy to obtain phased execution results, sampling key parameters in the phased execution results to obtain a parameter sampling set, performing data compression and index construction on the parameter sampling set to obtain a structured execution parameter record.

[0075] Specifically, in the process of executing the executable instructions corresponding to the target task and recording the execution parameters, the executable instructions are first analyzed for dependencies. This step uses an improved program dependency graph (PDG) analysis technique, which not only considers the data dependency and control dependency between instructions, but also introduces the concept of resource dependency. The system constructs a multi-level instruction dependency graph by combining static analysis and dynamic detection. In this dependency graph, nodes represent single instructions or instruction blocks, and edges represent different types of dependencies. In order to improve the accuracy of the analysis, the system also uses symbolic execution technology to perform more detailed dependency analysis on different paths of conditional branches. Based on the obtained instruction dependency graph, the system dynamically allocates computing resources. This process uses a resource allocation model based on a graph neural network (GNN). The model takes the instruction dependency graph as input and learns the local and global features of instruction nodes through multiple rounds of message passing. At the same time, the system monitors the status of hardware resources in real time and encodes this information as global attributes of the graph. The output of the GNN model is the resource demand prediction and optimal execution unit recommendation for each instruction node. Based on these predictions and recommendations, the system uses a heuristic algorithm to generate a preliminary resource scheduling plan. This scheme is not static, but contains multiple dynamic adjustment points, allowing fine-tuning according to actual conditions during execution. Next, the system analyzes the resource scheduling scheme for instruction-level parallelism. This step uses an innovative hierarchical parallel analysis method. At the lowest level, the system analyzes the potential for instruction-level parallelism (ILP) and identifies instruction sequences that can be executed in parallel. At the middle level, the system considers opportunities for data-level parallelism (DLP) and task-level parallelism (TLP), such as SIMD operations and multi-threaded execution. At the highest level, the system analyzes the possibility of cross-device parallelism and considers the optimal solution for allocating tasks between CPUs, GPUs, and dedicated accelerators. Through this multi-level analysis, the system obtains a comprehensive parallel execution strategy. According to the parallel execution strategy, the system performs group execution processing on executable instructions. The execution process adopts a dynamic pipeline mechanism that divides the instruction stream into multiple stages, each with its own independent execution unit. The system uses an intelligent scheduler to monitor the execution status of each stage in real time and dynamically adjust the allocation and execution order of instructions. In order to deal with the uncertainty in the execution process, the system also implements a speculative execution mechanism. For conditional branches, multiple possible paths are executed simultaneously, and the correct result is selected after the condition is determined. This method greatly reduces the performance loss caused by branch prediction errors. During the execution process, the system samples the key parameters in the staged execution results. The sampling strategy adopts the Adaptive Importance Sampling technology. The system maintains a dynamically updated parameter importance model and dynamically adjusts the sampling frequency according to the degree of influence of the parameters on the final result.For parameters that change dramatically or have a significant impact on the results, the system increases the sampling frequency; for parameters that change slowly or have a small impact, the sampling frequency is reduced. This method ensures the capture of key information while reducing the overhead of data storage and processing. Finally, the system performs data compression and index construction on the sampled parameter set. Data compression uses a context-based compression algorithm that uses the correlation and timing characteristics between parameters to achieve efficient lossless compression. For some scenarios where slight precision loss is acceptable, the system also provides a lossy compression method based on an autoencoder, which greatly reduces storage space while retaining the main features of the data. Based on the compressed data, the system builds a multidimensional index structure. This index not only supports fast parameter queries, but also supports complex pattern matching and trend analysis. The index structure adopts a hierarchical design, using B+ trees at the bottom layer to ensure basic query efficiency, and using skip lists at the upper layer to support more complex range queries and pattern matching.

[0076] In this embodiment, the target model corresponding to the current scene is segmented according to the preset environment configuration and segmentation requirements to obtain multiple corresponding model subgraphs; the target model and the model subgraph are compiled at different optimization levels according to the environment configuration to obtain multiple corresponding executable instructions; the resource usage of the current scene is monitored in real time; the target task of the current scene and the executable instructions corresponding to the target task are determined according to the resource usage and the operation of the target model corresponding to the current scene; the executable instructions corresponding to the target task are executed, and the execution parameters of the executable instructions are recorded. This method realizes the efficient use of computing resources through model segmentation and dynamic task allocation, while ensuring the flexibility and performance of model execution, and is suitable for the optimized deployment of various deep learning models in resource-constrained or multi-task environments.

[0077] See also Figure 2 A second embodiment of a method for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention includes:

[0078] 201. According to the preset environment configuration and segmentation requirements, the target model corresponding to the current scene is segmented to obtain corresponding multiple model sub-graphs;

[0079] In one embodiment of the present invention, model segmentation can also be performed by semantically parsing the environment configuration and the segmentation requirements to obtain a segmentation constraint set, performing topological structure analysis on the target model according to the segmentation constraint set to obtain a model calculation graph, performing critical path identification on the model calculation graph to obtain a model key substructure sequence, performing hierarchical clustering on the target model according to the model key substructure sequence to obtain an initial segmentation scheme, performing computational complexity evaluation on the initial segmentation scheme to obtain a complexity distribution graph, performing boundary optimization on the initial segmentation scheme according to the complexity distribution graph to obtain corresponding multiple model subgraphs.

[0080] Specifically, the environment configuration and segmentation requirements are first semantically parsed. This step uses advanced natural language processing technology, combined with pre-trained large-scale language models and domain-specific knowledge graphs. The system uses an attention mechanism to perform multi-level semantic understanding of the input text and extract key configuration parameters and constraints. For the environment configuration, the system identifies key information such as hardware specifications, memory limitations, and network bandwidth; for segmentation requirements, the system extracts constraints such as performance goals, resource utilization thresholds, and model accuracy requirements. This extracted information is organized into a structured segmentation constraint set, which provides a clear guidance framework for subsequent model analysis and optimization. Next, the system performs topological structure analysis on the target model based on the segmentation constraint set. This process uses an improved graph analysis algorithm to represent the deep learning model as a directed acyclic graph (DAG). The system not only considers the connection relationship between layers, but also analyzes detailed information such as the characteristics of data flow, computing density, and memory access patterns. By applying the connectivity analysis and flow calculation methods in graph theory, the system generates a detailed model calculation graph, which not only reflects the static structure of the model, but also contains the resource requirements and data dependency information during dynamic execution. After obtaining the model calculation graph, the system performs critical path identification. This step adopts an innovative multi-objective critical path analysis method. Traditional critical path analysis mainly considers execution time, while this system also considers multiple factors such as computational complexity, memory usage, and energy consumption. The system uses a dynamic programming algorithm combined with Monte Carlo simulation to calculate multiple critical paths under different optimization objectives. These paths are combined into a sequence of model key substructures, each of which represents a part of the model that has a significant impact on the overall performance. Based on the sequence of model key substructures, the system performs hierarchical clustering on the target model. This process uses a hierarchical spectral clustering algorithm that considers the computational similarity, data dependency strength, and resource requirement similarity between substructures. The clustering process is adaptive and dynamically adjusts the clustering parameters according to the previously defined segmentation constraints. For example, if the constraints require a more uniform resource allocation, the system will tend to generate clusters of more balanced sizes; if the emphasis is on reducing communication overhead, the system will prioritize clustering structures with strong data dependencies. The output of this process is an initial partitioning scheme that divides the model into multiple relatively independent parts. The system performs a computational complexity evaluation on the obtained initial partitioning scheme. This step uses a complexity prediction model based on machine learning. The model predicts the execution time, memory usage, and energy consumption on the target hardware by analyzing factors such as the structural characteristics, operation type, and parameter scale of each partition. The prediction results are visualized as a multi-dimensional complexity distribution diagram, which intuitively shows the demand distribution of each partition in different resource dimensions. Finally, the system performs boundary optimization on the initial partitioning scheme based on the complexity distribution diagram.This process uses an iterative boundary adjustment algorithm. The algorithm starts from the part with the most uneven complexity distribution and gradually adjusts the split boundaries. At each adjustment, the system re-evaluates the adjusted complexity distribution and checks whether all the split constraints are met. If it is found that a certain adjustment causes the constraints to be unsatisfied, the system will backtrack to the last valid state and try other adjustment schemes. This process continues until an equilibrium point is reached, that is, further adjustments cannot significantly improve the complexity distribution or meet more constraints.

[0081] 202. Compile the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions;

[0082] 203. Real-time monitoring of resource usage in the current scene;

[0083] In one embodiment of the present invention, steps 202-203 are similar to steps 102-103 in the first embodiment and are not described in detail herein.

[0084] 204. Analyze and process the resource usage to obtain currently available hardware resource information;

[0085] In one embodiment of the present invention, a real-time monitoring system is used to continuously collect the usage status of each hardware component. The system interacts directly with the hardware through a low-level API to collect indicators such as CPU utilization, GPU computing unit occupancy, memory usage, network bandwidth, etc. The collected data is preliminarily processed to remove noise and standardize. Then, a timing analysis algorithm, such as moving average or exponential smoothing, is used to predict resource change trends in the short term. Finally, the system combines static hardware configuration and dynamic usage to generate a detailed hardware resource map, including information such as the current load, available memory, and expected idle time of each computing unit. This mapping provides an accurate resource basis for subsequent task allocation.

[0086] 205. Perform task matching on multiple model subgraphs of the target model according to the requirements of the current scene to obtain the target task of the current scene;

[0087] In one embodiment of the present invention, task matching is performed on multiple model subgraphs of the target model according to the requirements of the current scene to obtain the target task of the current scene. The system first analyzes the specific requirements of the current scene, such as real-time requirements, accuracy requirements, energy consumption constraints, etc. Then, these requirements are matched with the pre-segmented model subgraphs. The matching process uses a heuristic algorithm to consider factors such as the function of the subgraph, computational complexity, and output accuracy. The system also uses historical execution data to predict the performance of each subgraph in the current scene. Based on this information, the system builds a decision tree or dynamic programming model to select the subgraph combination that best suits the current needs. The selection process also considers the dependencies between subgraphs to ensure that the selected subgraphs can be correctly connected in series to form a complete processing flow. Finally, the system determines a set of optimal model subgraphs, which together constitute the target task of the current scene.

[0088] 206. According to the target task and the available hardware resource information, select from a plurality of executable instructions to obtain an executable instruction corresponding to the target task;

[0089] In one embodiment of the present invention, in one embodiment of the present invention, according to the target task and the available hardware resource information, a selection is made from a variety of executable instructions to obtain executable instructions corresponding to the target task. The system first generates multiple executable instruction versions for each selected subgraph, which are targeted at different hardware configurations and optimization levels. Next, the system uses a resource allocation algorithm, such as a greedy algorithm or linear programming, to match the subgraph with the available hardware resources. The matching process takes into account the resource requirements, execution efficiency, and available status of the current hardware of the instruction version. The system also takes into account the data transmission cost between instructions to minimize data movement across devices. For some key subgraphs, the system considers multiple alternatives to cope with dynamic changes during execution. Finally, the system selects the most appropriate executable instruction version for each subgraph to form a complete execution plan.

[0090] 207. Associating the target task with the executable instruction to obtain the target task of the current scene and the executable instruction corresponding to the target task;

[0091] In one embodiment of the present invention, the target task and the executable instructions are associated to obtain the target task of the current scene and the executable instructions corresponding to the target task. The system constructs a task-instruction mapping table to record in detail the association between each subgraph task and its corresponding executable instruction. This mapping includes not only static correspondences, but also dynamic adjustment rules. The system uses workflow management technologies, such as directed acyclic graph (DAG) schedulers, to determine the execution order and parallel strategy of tasks. At the same time, the system establishes a feedback mechanism to monitor the performance of each task in real time during the execution process, and dynamically adjust the task-instruction mapping according to actual conditions. In addition, the system also generates synchronization points and data exchange protocols between tasks to ensure that the entire target task can be executed smoothly.

[0092] 208. Execute the executable instruction corresponding to the target task, and record the execution parameters of the executable instruction corresponding to the target task.

[0093] In one embodiment of the present invention, executing the executable instructions corresponding to the target task and recording the execution parameters of the executable instructions corresponding to the target task include: parsing the executable instructions corresponding to the target task to obtain a corresponding instruction execution sequence; executing the executable instructions corresponding to the target task one by one according to the instruction execution sequence to obtain the execution result of each executable instruction corresponding to the target task; and monitoring and recording the execution results in real time to obtain corresponding execution parameters.

[0094] Specifically, the executable instructions corresponding to the target task are first parsed to obtain the corresponding instruction execution sequence. This process involves syntax analysis and semantic parsing of the instructions, converting high-level instructions into more fine-grained operation sequences. Next, the system executes the executable instructions corresponding to the target task one by one according to the instruction execution sequence obtained by parsing. This process adopts an efficient instruction scheduling mechanism that combines the advantages of static scheduling and dynamic scheduling. Static scheduling performs preliminary optimization of the instruction sequence before execution, such as instruction reordering and pipeline planning, to improve instruction-level parallelism. Dynamic scheduling makes real-time adjustments according to the actual execution situation at runtime to cope with runtime uncertainties. During execution, the system uses a multi-level cache strategy to store frequently accessed data and intermediate results in fast-access memory to reduce data reading and writing delays. For computationally intensive instructions, the system automatically decomposes and parallelizes tasks to make full use of parallel computing resources such as multi-core processors or GPUs. At the same time, the system also implements an intelligent memory management mechanism to minimize memory usage and avoid memory leaks and fragmentation problems through dynamic memory allocation and recycling. In order to handle conditional branches and loop structures, the system uses branch prediction technology and loop unrolling optimization. Branch prediction predicts the possible results of conditional branches by analyzing historical execution data, reducing pipeline pauses caused by branch jumps. Loop unrolling increases the number of instructions per iteration, reduces loop control overhead, and improves the utilization of the instruction pipeline. During execution, the system monitors the execution of each instruction in real time and records the corresponding execution parameters. This monitoring process uses lightweight probe technology to insert monitoring code at key instruction points to collect multi-dimensional data such as execution time, memory usage, cache hit rate, and I / O operations. In order to minimize the impact of monitoring on performance, the system uses sampling technology to perform detailed monitoring only at specific time points or on specific instructions. The collected raw execution data undergoes preliminary processing, including denoising, normalization, and compression. The system uses sliding window technology to calculate the moving average of key indicators to smooth short-term fluctuations. For long-running tasks, the system adopts a segmented storage strategy to periodically write monitoring data to persistent storage to prevent data loss due to system crashes. In order to better understand the global situation of instruction execution, the system implements a visualization module to convert execution parameters into intuitive charts and heat maps. These visualization results help to quickly identify performance bottlenecks and abnormal behaviors. At the same time, the system has also established an anomaly detection model based on machine learning to analyze execution parameters in real time and promptly detect and mark abnormal situations. In addition, the system has also implemented an adaptive parameter tuning mechanism. By analyzing historical execution data and current execution parameters, the system can dynamically adjust certain configurable execution parameters, such as cache size, parallelism, memory allocation strategy, etc., to optimize the execution efficiency of subsequent instructions.Finally, the system organizes all collected execution parameters into structured logs, including timestamps, instruction identifiers, execution time, resource usage, output results, and other information. These logs are not only used for immediate performance analysis, but also provide valuable data support for long-term performance optimization and system tuning. The system also provides a query interface that allows other modules or external tools to easily access and analyze these execution parameters.

[0095] Furthermore, the parsing and processing of the executable instructions corresponding to the target task to obtain the corresponding instruction execution sequence includes: performing a demand analysis on the target task to obtain the timeliness requirements of different target models corresponding to the target task in the current scenario; prioritizing the executable instructions corresponding to the model subgraphs of each target model according to the timeliness requirements to obtain a preliminary execution order; estimating the running time of the executable instructions in the preliminary execution order to obtain the estimated execution time of each executable instruction; adjusting the preliminary execution order according to the estimated execution time and the timeliness requirements to obtain the corresponding instruction execution sequence.

[0096] Specifically, the requirements of the target task are first analyzed to obtain the timeliness requirements of different target models corresponding to the target task in the current scenario. This process involves an in-depth understanding of the task context and performance requirements. The system uses a multi-dimensional task feature extractor to analyze factors such as the type, complexity, data volume, and expected output of the task. At the same time, considering the particularity of the current scenario, such as real-time requirements and resource constraints, the system uses a scenario adaptability analysis module to match the task requirements with the scenario features. This matching process uses machine learning technology to establish a task-scenario-requirement mapping model by analyzing historical task execution data. The model can output the timeliness requirement score of each target model based on the input task description and scenario parameters. This score reflects the importance and urgency of different models in the current task and scenario. Next, the system prioritizes the executable instructions corresponding to the model subgraphs of each target model according to the obtained timeliness requirements to obtain a preliminary execution order. The sorting process uses an improved multi-objective sorting algorithm that comprehensively considers the timeliness requirement score, the dependency between subgraphs, and the availability of hardware resources. The algorithm first constructs a task dependency graph to represent the execution order constraints between subgraphs. Then, a variation of topological sorting is used to calculate the priority score of each subgraph with timeliness requirements as node weights. In this process, the system also considers resource competition factors and uses resource reservation strategies to prevent high-priority tasks from being blocked by low-priority tasks. The sorting results are conflict-detected and adjusted to ensure that deadlocks or resource starvation will not occur. Finally, the system obtains a preliminary execution order that balances timeliness requirements and execution efficiency. Then, the system estimates the running time of the executable instructions in the preliminary execution order to obtain the estimated execution time of each executable instruction. This estimation process uses a performance prediction model based on historical data and machine learning. The model considers multiple factors, including the complexity of the instruction, the size of the input data, and the performance characteristics of the target hardware. In order to improve the prediction accuracy, the system uses an ensemble learning method to combine the prediction results of multiple sub-models. These sub-models include decision tree-based regression models, neural network models, and case-based reasoning models. Each sub-model is optimized for different types of instructions and execution environments. The system also implements a dynamic update mechanism to fine-tune the prediction model with the real execution time after each actual execution to continuously improve the accuracy of the prediction. Finally, the system adjusts the preliminary execution order based on the expected execution time and timeliness requirements to obtain the final instruction execution sequence. This adjustment process uses a dynamic programming algorithm, the goal of which is to minimize the total execution time while meeting timeliness requirements. The algorithm first calculates the latest start time and the earliest end time of each instruction, and then adjusts the execution order of the instructions by traversing in reverse. In this process, the system considers parallel execution opportunities between instructions and uses critical path analysis to identify groups of instructions that can be parallelized.For some key time nodes, the system sets checkpoints to ensure that important intermediate results can be produced on time. In addition, the system also implements a dynamic scheduling mechanism that allows fine-tuning of the execution order according to actual conditions during execution. This mechanism is particularly suitable for processing instructions with uncertain execution time.

[0097] In this embodiment, the target model corresponding to the current scene is segmented according to the preset environment configuration and segmentation requirements to obtain multiple corresponding model subgraphs; the target model and the model subgraph are compiled at different optimization levels according to the environment configuration to obtain multiple corresponding executable instructions; the resource usage of the current scene is monitored in real time; the target task of the current scene and the executable instructions corresponding to the target task are determined according to the resource usage and the operation of the target model corresponding to the current scene; the executable instructions corresponding to the target task are executed, and the execution parameters of the executable instructions are recorded. This method realizes the efficient use of computing resources through model segmentation and dynamic task allocation, while ensuring the flexibility and performance of model execution, and is suitable for the optimized deployment of various deep learning models in resource-constrained or multi-task environments.

[0098] The above describes a method for optimizing the deployment of an AI model to improve hardware throughput in an embodiment of the present invention. The following describes a device for optimizing the deployment of an AI model to improve hardware throughput in an embodiment of the present invention. Figure 3 , an embodiment of a device for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention includes:

[0099] The model segmentation module 301 performs model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain corresponding multiple model sub-graphs;

[0100] A model compiling module 302, configured to compile the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions;

[0101] The resource scheduling module 303 is used to monitor the resource usage of the current scene in real time;

[0102] The task allocation module 304 is used to determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene;

[0103] The performance recording module 306 is used to execute the executable instructions corresponding to the target task and record the execution parameters of the executable instructions corresponding to the target task.

[0104] In an embodiment of the present invention, the device for optimizing the deployment of AI models to improve hardware throughput runs the above-mentioned method for optimizing the deployment of AI models to improve hardware throughput, and the device for optimizing the deployment of AI models to improve hardware throughput performs model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain multiple corresponding model subgraphs; compiles the target model and the model subgraphs at different optimization levels according to the environment configuration to obtain multiple corresponding executable instructions; monitors the resource usage of the current scene in real time; determines the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation of the target model corresponding to the current scene; executes the executable instructions corresponding to the target task, and records the execution parameters of the executable instructions. This method realizes the efficient use of computing resources through model segmentation and dynamic task allocation, while ensuring the flexibility and performance of model execution, and is suitable for the optimized deployment of various deep learning models in resource-constrained or multi-task environments.

[0105] above Figure 3 From the perspective of modular functional entities, an apparatus for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention is described in detail. Next, from the perspective of hardware processing, a device for optimizing deployment of an AI model for improving hardware throughput in an embodiment of the present invention is described in detail.

[0106] Figure 4 It is a structural schematic diagram of a device for optimizing the deployment of an AI model to improve hardware throughput provided by an embodiment of the present invention. The device 400 for optimizing the deployment of an AI model to improve hardware throughput may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more mass storage device terminals) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 can be short-term storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in a device 400 for optimizing the deployment of an AI model to improve hardware throughput. Furthermore, the processor 410 may be configured to communicate with the storage medium 430, and execute a series of instruction operations in the storage medium 430 on a device 400 for optimizing the deployment of an AI model to improve hardware throughput, so as to implement the steps of the above-mentioned method for optimizing the deployment of an AI model to improve hardware throughput.

[0107] A device 400 for optimizing deployment of AI models to improve hardware throughput may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 4 The device structure shown for optimizing deployment of an AI model for improving hardware throughput does not constitute a limitation on the device for optimizing deployment of an AI model for improving hardware throughput provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0108] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for optimizing the deployment of an AI model to improve hardware throughput.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0111] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing deployment of an AI model to improve hardware throughput, characterized in that: The method for optimizing deployment of an AI model to improve hardware throughput includes: According to the preset environment configuration and segmentation requirements, the target model corresponding to the current scene is segmented to obtain corresponding multiple model subgraphs; the environment configuration includes processor type, memory capacity and storage space, and the segmentation requirements include performance targets and resource utilization; The step of performing model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model subgraphs includes: determining the priority level of each target model corresponding to the current scene according to the environment configuration and the segmentation requirements; determining the division granularity of each target model corresponding to the current scene according to the priority level; performing model segmentation on the corresponding model according to the division granularity to obtain the corresponding multiple model subgraphs; The method further comprises: performing model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model subgraphs, and also includes performing semantic analysis processing on the environment configuration and the segmentation requirements to obtain a segmentation constraint condition set, performing topological structure analysis processing on the target model according to the segmentation constraint condition set to obtain a model calculation graph, performing key path identification processing on the model calculation graph to obtain a model key substructure sequence, performing hierarchical clustering processing on the target model according to the model key substructure sequence to obtain an initial segmentation scheme, performing computational complexity evaluation processing on the initial segmentation scheme to obtain a complexity distribution graph, and performing boundary optimization processing on the initial segmentation scheme according to the complexity distribution graph to obtain the corresponding multiple model subgraphs; Compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain a corresponding plurality of executable instructions; Compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain a corresponding plurality of executable instructions includes: performing intermediate code generation processing on the target model and the model subgraph to obtain a corresponding intermediate representation; optimizing the intermediate representation according to the environment configuration to obtain an optimized intermediate code; performing target code generation processing on the optimized intermediate code to obtain an assembly code; performing machine code generation processing on the assembly code to obtain a corresponding plurality of executable instructions; Real-time monitoring of resource usage of the current scenario; Determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene; Determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene includes: performing dynamic timing analysis processing on the resource usage and the operation status of the target model to obtain a multi-dimensional time-varying feature sequence, performing state update processing on a preset deep reinforcement learning model according to the multi-dimensional time-varying feature sequence to obtain a state representation of the current scene, performing task strategy reasoning processing on the state representation to obtain a candidate task decision tree, performing Monte Carlo tree search processing on the candidate task decision tree to obtain an optimal task path, performing adaptive instruction generation processing on key node tasks in the optimal task path to obtain an executable instruction set corresponding to the target task; Execute the executable instruction corresponding to the target task, and record the execution parameters of the executable instruction corresponding to the target task.

2. The method for optimizing deployment of AI models to improve hardware throughput according to claim 1, characterized in that: The performing model segmentation on the corresponding model according to the division granularity to obtain the corresponding multiple model subgraphs includes: Performing operator analysis on the target model to obtain an operator sequence in the target model; Performing hardware adaptability analysis on operators in the operator sequence to obtain hardware analysis results; Dividing the operators in the operator sequence into groups according to the hardware analysis result and the division granularity to obtain a plurality of operator combinations; Generate corresponding multiple model subgraphs according to the multiple operator combinations.

3. The method for optimizing the deployment of an AI model to improve hardware throughput according to claim 1, characterized in that: Determining the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene includes: Analyzing and processing the resource usage to obtain currently available hardware resource information; Performing task matching on multiple model subgraphs of the target model according to the requirements of the current scene to obtain the target task of the current scene; According to the target task and the available hardware resource information, selecting from the multiple executable instructions to obtain an executable instruction corresponding to the target task; The target task and the executable instruction are associated with each other to obtain the target task of the current scene and the executable instruction corresponding to the target task.

4. The method for optimizing the deployment of an AI model to improve hardware throughput according to claim 1, characterized in that: The executing the executable instruction corresponding to the target task and recording the execution parameters of the executable instruction corresponding to the target task includes: Parsing the executable instructions corresponding to the target task to obtain a corresponding instruction execution sequence; Executing the executable instructions corresponding to the target task one by one according to the instruction execution sequence to obtain the execution result of each executable instruction corresponding to the target task; The execution results are monitored and recorded in real time to obtain the corresponding execution parameters.

5. The method for optimizing the deployment of an AI model to improve hardware throughput according to claim 4, characterized in that: The parsing of the executable instructions corresponding to the target task to obtain the corresponding instruction execution sequence includes: Performing a demand analysis on the target task to obtain the timeliness requirements of different target models corresponding to the target task in the current scenario; Prioritize the executable instructions corresponding to the model subgraphs of each target model according to the timeliness requirement to obtain a preliminary execution order; Estimating the running time of the executable instructions in the preliminary execution sequence to obtain an estimated execution time of each executable instruction; The preliminary execution order is adjusted according to the estimated execution time and the timeliness requirement to obtain a corresponding instruction execution sequence.

6. A device for optimizing the deployment of an AI model to improve hardware throughput, characterized in that: The device for optimizing the deployment of the AI ​​model for improving hardware throughput includes: The model segmentation module performs model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain corresponding multiple model subgraphs; the environment configuration includes the processor type, memory capacity and storage space, and the segmentation requirements include performance targets and resource utilization; The step of performing model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model subgraphs includes: determining the priority level of each target model corresponding to the current scene according to the environment configuration and the segmentation requirements; determining the division granularity of each target model corresponding to the current scene according to the priority level; performing model segmentation on the corresponding model according to the division granularity to obtain the corresponding multiple model subgraphs; The method further comprises: performing model segmentation on the target model corresponding to the current scene according to the preset environment configuration and segmentation requirements to obtain the corresponding multiple model subgraphs, and also includes performing semantic analysis processing on the environment configuration and the segmentation requirements to obtain a segmentation constraint condition set, performing topological structure analysis processing on the target model according to the segmentation constraint condition set to obtain a model calculation graph, performing key path identification processing on the model calculation graph to obtain a model key substructure sequence, performing hierarchical clustering processing on the target model according to the model key substructure sequence to obtain an initial segmentation scheme, performing computational complexity evaluation processing on the initial segmentation scheme to obtain a complexity distribution graph, and performing boundary optimization processing on the initial segmentation scheme according to the complexity distribution graph to obtain the corresponding multiple model subgraphs; A model compilation module, used for compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions; the compiling the target model and the model subgraph at different optimization levels according to the environment configuration to obtain corresponding multiple executable instructions includes: performing intermediate code generation processing on the target model and the model subgraph to obtain corresponding intermediate representation; optimizing the intermediate representation according to the environment configuration to obtain optimized intermediate code; performing target code generation processing on the optimized intermediate code to obtain assembly code; performing machine code generation processing on the assembly code to obtain corresponding multiple executable instructions; A resource scheduling module, used to monitor the resource usage of the current scenario in real time; A task allocation module, used to determine the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene; the determination of the target task of the current scene and the executable instructions corresponding to the target task according to the resource usage and the operation status of the target model corresponding to the current scene includes: performing dynamic timing analysis processing on the resource usage and the operation status of the target model to obtain a multi-dimensional time-varying feature sequence, performing state update processing on a preset deep reinforcement learning model according to the multi-dimensional time-varying feature sequence to obtain a state representation of the current scene, performing task strategy reasoning processing on the state representation to obtain a candidate task decision tree, performing Monte Carlo tree search processing on the candidate task decision tree to obtain an optimal task path, and performing adaptive instruction generation processing on key node tasks in the optimal task path to obtain an executable instruction set corresponding to the target task; The performance recording module is used to execute the executable instructions corresponding to the target task and record the execution parameters of the executable instructions corresponding to the target task.

7. A device for optimizing deployment of AI models to improve hardware throughput, characterized in that: The device for optimizing deployment of AI models for improving hardware throughput includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the device for optimizing the deployment of the AI ​​model to improve the hardware throughput performs the steps of a method for optimizing the deployment of the AI ​​model to improve the hardware throughput as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of a method for optimizing deployment of an AI model for improving hardware throughput as described in any one of claims 1-5 are implemented.

Images