Deep learning reasoning acceleration method based on calculation sharing
By fine-grainedly decomposing the deep learning model computation graph, generating common computation subgraphs and building a computational shared cache, the problems of computational resource waste and repeated computation in existing technologies are solved, efficient collaboration and resource optimization of heterogeneous hardware are achieved, and the acceleration effect of deep learning reasoning is improved.
Patent Information
- Application Number
- CN202510614193.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-13
- Publication Date
- 2025-09-16
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing deep learning inference acceleration methods have problems such as the contradiction between the granularity of computational graph decomposition and the efficiency of sub-graph sharing, serious repeated computing, and low efficiency of heterogeneous hardware collaboration, and fail to effectively utilize computing resources and cache sharing mechanisms.
By fine-grained decomposition of the deep learning model computation graph, generating common computation subgraph fingerprints, building a computational shared cache, and combining neural network latent variable analysis, performing computational resource-aware measurement and load balancing scheduling, we design a dynamic migration strategy to optimize computational resource utilization.
It improves the computing subgraph reuse capability and resource utilization, enhances the acceleration ratio and reasoning efficiency of deep learning models, and supports efficient processing of dynamic tasks.
Smart Images
Figure CN120653426A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to deep learning model reasoning acceleration technology. Background Art
[0002] With the development of deep learning and intelligent hardware, deep learning inference acceleration has received widespread attention in the industry. Current deep learning inference acceleration methods mainly focus on areas such as computational graph optimization, cache sharing mechanisms, and load balancing strategies. Distributed inference methods based on model sharding divide the model into multiple sub-modules for parallel execution, but do not solve the problem of unbalanced computational load between heterogeneous devices; and computational graph decomposition mostly adopts a static layering strategy, which is difficult to adapt to dynamic task topology. There are also methods that use batch processing strategies based on GPU memory reuse to reduce memory usage by merging requests, but lack real-time support for high-concurrency scenarios; or introduce cache sharing mechanisms, but ignore the common characteristics of computing tasks and the correlation between latent variables, resulting in redundant calculations.
[0003] In summary, existing deep learning inference acceleration methods have three problems: 1) The granularity of computational graph decomposition is inconsistent with the efficiency of subgraph sharing. Coarse-grained decomposition leads to waste of computing resources, while fine-grained decomposition introduces additional scheduling overhead; 2) The phenomenon of repeated computing is serious. The intermediate features of similar input data cannot be effectively reused. The latent variable cache lacks multi-task correlation modeling and fails to achieve cross-task weight reuse and memory compression; 3) Traditional scheduling strategies do not take into account the density of computing tasks, the differences in hardware computing power, and the dynamic load status of different computing power units. The collaborative efficiency of heterogeneous hardware (CPU / GPU / TPU) is low, and the problem of communication bandwidth contention is significant. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to propose a deep learning reasoning acceleration optimization method based on computing sharing to overcome the defects of existing deep learning reasoning scenarios such as computing redundancy, low resource utilization, and poor dynamic adaptability.
[0005] To solve the above technical problems, the present invention proposes a deep learning inference acceleration method based on computing sharing, comprising the following steps: Perform fine-grained decomposition of the computational graph of the deep learning model, screen and generate the optimal common computational subgraph fingerprint, extract common computational subgraphs for multiple scenarios and tasks, and reduce repeated calculations; Analyze the neural network hidden variables of the deep learning model, combine common computational subgraphs, build a computational shared cache, and achieve cross-task weight reuse; Perceive and measure the transfer computing resources of deep learning inference tasks and predict them in advance to balance the computing sharing load; Balanced scheduling is performed based on the computing shared load on each node.
[0006] Furthermore, the deep learning inference acceleration method based on computing sharing also includes switching task migration strategies and optimizing communication-computing pipelines to reduce strategy switching delays; Furthermore, the deep learning inference acceleration method based on computing sharing also includes updating sharing strategy parameters based on runtime indicator feedback.
[0007] Preferably, the fine-grained decomposition of the computational graph of the deep learning model, screening and generating the optimal common computational subgraph fingerprint, and extracting the common computational subgraphs for multiple scenarios and multiple tasks include: Analyze the deep learning computational graph structure and dependency matrix; Construct the data flow and control flow dependency matrix between operators; By performing eigendecomposition on the normalized Laplace matrix, the first k eigenvectors are selected to form the embedding space, where k is an empirical value. The operator units are clustered and grouped to generate a set of parallel candidate subgraphs. Calculate the commonality and calculate the cost of the subgraph; The cost size is calculated based on the commonality calculation subgraph, and the optimal commonality subgraph fingerprint and hash index table are screened and generated.
[0008] Preferably, analyzing the neural network latent variables of the deep learning model, combining common computational subgraphs, and constructing a computational shared cache comprises the following steps: According to the operation of the deep learning model, the tensor features of its neural network latent variables are extracted; Calculate the cache value function of the latent variable tensor; Design a cache replacement strategy based on the cache value function.
[0009] Preferably, the perceptual measurement of the transfer computing resources of the deep learning reasoning task and the prediction thereof in advance include the sub-steps of: Deploy perception probes on deep learning hardware to collect GPU / CPU utilization, memory usage, and bandwidth throughput data, and construct a joint task-resource feature space vector. LSTM-based time series regression training and prediction of computing task resource demand model; Preferably, the balanced scheduling according to the computing shared load on each node includes the following sub-steps: Calculate the load difference between nodes and set the optimization goal to minimize the load difference between nodes; Design a hybrid scheduling strategy and use the first adaptation algorithm to generate the initial solution; Based on the initial solution, the simulated annealing optimization algorithm is introduced to further optimize the solution.
[0010] Based on the technical architecture of dynamic computing sharing, this paper mainly constructs a shared computing layered cache driven by shared latent variables, implements balanced scheduling of computing shared loads, and implements a closed-loop prediction feedback mechanism for computing sharing strategies. This optimizes deep learning models, improves the reuse capability of computing subgraphs and operator levels, and the cache hit rate of forward reasoning weights. It also supports dynamic subgraph partitioning and migration, thereby improving the speedup ratio and resource utilization of typical models. Combining parallel computing, dynamic resource scheduling, and computing resource sharing mechanisms, this paper optimizes reasoning efficiency in heterogeneous computing environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.
[0012] Figure 1 This is an overall flow chart of a specific implementation method of the present invention. DETAILED DESCRIPTION
[0013] The deep learning inference acceleration method based on computing sharing proposed in this invention has the overall technical idea of identifying reusable operators by building a dynamic computing subgraph fingerprint library, designing an elastic scheduler to dynamically allocate heterogeneous computing resources, and establishing a two-level caching mechanism to realize cross-request sharing of computing results.
[0014] Specific implementation method, combined with Figure 1 As shown, the following specific steps are included: Step 1: Perform fine-grained decomposition of the computational graph of the deep learning model and extract common computational subgraphs for multiple scenarios and tasks. This includes the following substeps: Step 1-1: Analyze the deep learning computational graph structure and dependency matrix. Use graph theory algorithms to decompose the computational graph G = (V, E), where V represents the set of operator nodes and E represents the set of data flow edges. Define the parallelism evaluation function: ; in To calculate the time consumption, It takes time for adjacent nodes to communicate.
[0015] when It is a parallel unit.
[0016] Step 1-2: Construct the data flow and control flow dependency matrix between operators , where the elements , represents the data transmission time from operator i to j, is the computation time of operator i.
[0017] Step 1-3: Perform eigendecomposition by normalizing the Laplace matrix, select the first k eigenvectors to form the embedding space, k is an empirical value, and use the k-means method to cluster and group the operator units to generate a parallel candidate subgraph set. , Where I is the unit diagonal matrix, D is the connectivity matrix, and W is the data flow and control flow dependency matrix between operators.
[0018] Step 1-4: According to the following formula, the computational cost S(g) of the commonality computation subgraph is obtained, ; Where g is a subgraph, To calculate the time consumption, is the communication time of adjacent nodes, α is the granularity adjustment coefficient, is the changing subgraph.
[0019] Step 1-5: Based on the subgraph calculation cost in step 1-4, the optimal common subgraph fingerprint and hash index table are screened and generated.
[0020] Constructing feature vectors Contains features such as operator type, input dimension, parameter hash value, etc. Fingerprint generation formula: in is a trainable weight matrix and b is a bias term to achieve semantically sensitive feature encoding.
[0021] Step 2: Analyze the neural network hidden variables of the deep learning model and design and calculate the shared cache based on the shared subgraph generated in step 1. The specific substeps are as follows: Step 2-1: Based on the operation of the deep learning model, extract the tensor features of its neural network latent variables, including but not limited to the access frequency and life cycle of the tensor.
[0022] Step 2-2: Calculate the cache value function of the latent variable tensor t ; in : The access frequency per unit time of the latent variable tensor t; : The life cycle of the latent variable tensor t (ms); : Video memory usage (MB).
[0023] Step 2-3: Design a cache replacement strategy based on the cache value function.
[0024] When cache space is insufficient, prioritize elimination tensor, θ is the dynamic threshold.
[0025] Based on this, a double-layer cache structure is designed, where the L1 cache meets (Upper quartile) hot variables, L2 cache Low-frequency, long-period variables (between the upper and lower quartiles). The upper and lower quartiles are statistical indicators, with the upper quartile representing 3 / 4 of the total number and the lower quartile representing 1 / 4 of the total number.
[0026] Step 3: Further sense and measure the transfer, storage, and computing resources of deep learning inference tasks, and make predictions in advance to balance the computing and sharing load. The specific sub-steps are as follows: Step 3-1: Deploy perception probes on the deep learning hardware to collect GPU / CPU utilization, memory usage, and bandwidth throughput data, and construct a task-resource joint feature space vector. ; in is the computing efficiency of device r (FLOPS), is the available video memory (MB), FLOP S (t) is the computational requirement of task t, MEM(t) is the video memory requirement of task t, is the I / O requirement of task t (MB / s), is the available bandwidth of device r.
[0027] 3-2 LSTM-based time series regression training and prediction of computing task resource demand model ; ; ; in, is the resource occupancy rate of the next three time slices, σ is the sigmoid function, and ⊙ represents the Hadamard product. is the weight matrix, is a hidden variable, is the bias, is the eigenvector of step 3-1, is the gate vector, is the forget gate vector, is the input gate vector, These are the standard notation definitions of LSTM Step 4: Perform balanced scheduling based on the computing shared load on each node to prevent overloading of some nodes and thus degrading the overall system performance. The specific sub-steps are as follows: Step 4-1: Establish a dynamic load balancing model and define the optimization goal as minimizing the load difference between nodes. ; in, , represents the comprehensive load of node i, They are the dynamic weight coefficients of GPU computing power and video memory occupancy, is the GPU load of the i-th node, is the memory load of the i-th node, N is the total number of nodes, and j is the sequence number of the j-th node.
[0028] Step 4-2: Design a hybrid scheduling strategy and use the first adaptation algorithm to generate an initial solution ; in represents the task requirement vector, is the occupied resource vector of the kth task of node s, and K is the total number of tasks.
[0029] Step 4-3: Based on the initial solution, introduce the simulated annealing optimization algorithm to further optimize the solution ; in, To optimize the solution, is the error relative to the initial solution, and the temperature parameter T decays with the number of iterations t: , to achieve global optimal search.
[0030] Step 5: When the node load continuously exceeds the threshold, migration is triggered. The computing sharing strategy is switched during task migration. The specific sub-steps are as follows: Step 5-1: Monitor the node load threshold. For example, when the CPU load exceeds 80%, or the GPU memory exceeds 90% for more than 5 seconds, the load migration of computing sharing is triggered, as shown in the following formula: ; in, 、 The specific thresholds of CPU>80% or GPU memory>90% for 5 seconds are empirical values for example and optional for those skilled in the art, and do not limit the present invention.
[0031] Step 5-2: Select task subgraphs with communication overhead < 200ms to generate migration candidate sets ; in is the task candidate set, is the i-th task subgraph, is the amount of data to be migrated for the i-th task subgraph, is the available bandwidth, Similarly, the above communication overhead <200ms is an empirical value used for example, which can be selected by those skilled in the art and does not constitute a limitation to the present invention.
[0032] Step 5-3: Use a two-phase migration approach when calculating shared migration, freezing the subgraph state in the preparatory phase ; is the i-th task state, is a serialized snapshot, Atomic migration during commit ; is an atomic collection of snapshot tasks. The node after migration.
[0033] Step 6: For the overlapping parts of the computing tasks, align the communication computing resources to optimize them. The specific sub-steps are as follows: Step 6-1: Based on the spatiotemporal dependency between the computational flow and the communication flow, a spatiotemporal overlap optimization model is constructed and the pipeline cycle is defined as: ; in To calculate the time consumption, Communication takes time.
[0034] Step 6-2: Design a pipeline parallel controller to separate PCIe and NVLink channels. ; Step 6-3: Dynamically expand the computing instruction prefetch and communication buffer pools. The adjustment formula is: ; in, is the cache pool size, is the current queue length, is the queue threshold, is the adaptive learning rate.
[0035] Step 7: Feedback and update the computing sharing strategy. The specific sub-steps are as follows: Step 7-1: Collect runtime metrics, including task completion time , cache hit rate , migration times .
[0036] Step 7-2: Build a strategy evaluation model (multi-objective weighted scoring mechanism) ; is the weight of the i-th strategy.
[0037] Step 7-3: The update strategy of the parameter subgraph granularity coefficient α and migration threshold β is ; The error , is the adjustment coefficient, and finally generates a new strategy parameter combination .
[0038] Finally, it should be noted that the above specific implementation methods are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A deep learning reasoning acceleration method based on computing sharing, characterized in that: The following steps are involved: Perform fine-grained decomposition of the computational graph of the deep learning model, screen and generate the optimal common computational subgraph fingerprint, and extract common computational subgraphs for multiple scenarios and tasks; Analyze the neural network hidden variables of deep learning models, combine common computational subgraphs, and build a computational shared cache; Perceive and measure the computing resources required for deep learning reasoning tasks and predict them in advance; Balanced scheduling is performed based on the computing shared load on each node.
2. The deep learning reasoning acceleration method based on computing sharing according to claim 1 is characterized in that: The inference acceleration method also includes switching task migration strategies and optimizing communication-computing pipelines to reduce strategy switching delays.
3. The deep learning reasoning acceleration method based on computing sharing according to claim 2 is characterized in that: The inference acceleration method further includes updating shared strategy parameters based on runtime indicator feedback.
4. The deep learning reasoning acceleration method based on computing sharing according to claim 1 is characterized in that The method of performing fine-grained decomposition on the computational graph of the deep learning model, screening and generating the optimal common computational subgraph fingerprint, and extracting common computational subgraphs for multiple scenarios and multiple tasks includes: Analyze the deep learning computational graph structure and dependency matrix; Construct the data flow and control flow dependency matrix between operators; By performing eigendecomposition on the normalized Laplace matrix, the first k eigenvectors are selected to form the embedding space, where k is an empirical value. The operator units are clustered and grouped to generate a set of parallel candidate subgraphs. Calculate the computational cost of the common computation subgraph; The cost size is calculated based on the commonality calculation subgraph, and the optimal commonality subgraph fingerprint and hash index table are screened and generated.
5. The deep learning reasoning acceleration method based on computing sharing according to claim 1 is characterized in that: The method of analyzing the neural network latent variables of the deep learning model, combining common computational subgraphs, and building a computational shared cache includes the following steps: According to the operation of the deep learning model, the tensor features of its neural network latent variables are extracted; Calculate the cache value function of the latent variable tensor; Design a cache replacement strategy based on the cache value function.
6. The deep learning reasoning acceleration method based on computing sharing according to claim 1 is characterized in that: The process of perceiving and measuring the transfer computing resources of deep learning inference tasks and predicting them in advance includes the following sub-steps: Deploy perception probes on deep learning hardware to collect GPU / CPU utilization, memory usage, and bandwidth throughput data, and construct a joint task-resource feature space vector. A model for training and predicting resource requirements of computing tasks based on LSTM-based time series regression.
7. The deep learning reasoning acceleration method based on computing sharing according to claim 1 is characterized in that: The balanced scheduling based on the computing shared load on each node includes the following sub-steps: Calculate the load difference between nodes and set the optimization goal to minimize the load difference between nodes; Design a hybrid scheduling strategy and use the first adaptation algorithm to generate the initial solution; Based on the initial solution, the simulated annealing optimization algorithm is introduced to further optimize the solution.