A heterogeneous hybrid acceleration method, device and medium based on multiple accelerator cards

Abstract

The present application discloses a heterogeneous hybrid acceleration method, device and medium based on multiple accelerator cards, and relates to the field of parallel computing technology. The method includes: taking each operator in the deep learning model and the kernel combination of the corresponding target platform as the initial situation, and unloading the kernel combination to the hardware for execution; taking the score of the kernel combination as the initial cost of the cost model, and optimizing the cost model by the gradient descent method to determine whether it is an extreme value; comparing the extreme value corresponding to the kernel combination with the historical record to determine whether to update the tuning record, so as to determine the corresponding target tuning record; dividing the computational subgraph of the deep learning model analysis according to the target tuning record, and unloading the divided deep learning model to the corresponding hardware device for reasoning, so as to realize heterogeneous hybrid acceleration. The present application realizes efficient coordination and performance maximization of model reasoning, and improves the computing efficiency and response speed of the overall system.

Description

Technical Field

[0001] The present application relates to the field of parallel computing technology, and in particular to a heterogeneous hybrid acceleration method, device and medium based on multiple acceleration cards. Background Art

[0002] At present, with the rapid development and widespread application of artificial intelligence (AI) technology, the demand for high-performance computing in various fields, from deep learning, machine learning to big data processing, has increased dramatically, which has posed unprecedented challenges to the computing industry. The complexity of AI algorithms and the explosive growth of data volume have driven the demand for more efficient and powerful computing power. However, in the face of this surging wave of demand, major chip manufacturers around the world are still unable to meet the market's immediate and comprehensive supply of all types of acceleration hardware despite their continuous efforts to increase production capacity and optimize design.

[0003] At the same time, in actual application scenarios, users are often faced with diverse computing task requirements, which may involve multiple fields such as image processing, natural language processing, scientific computing, etc., and each task has different requirements for computing resources. Therefore, there are often a variety of accelerated computing cards of different forms and architectures coexisting in the user environment, including but not limited to GPU (graphics processing unit), FPGA (field programmable gate array), TPU (tensor processing unit) and various dedicated AI acceleration chips. These heterogeneous acceleration cards each have their own unique advantages and applicable scenarios, but they also bring complexity to resource integration and collaborative work.

[0004] In view of the above background, how to find a balance between limited hardware resources and diverse user needs to achieve efficient and flexible computing power deployment has become an urgent problem to be solved. Traditional single-type acceleration card solutions often fail to fully utilize existing hardware resources, resulting in computing power waste or performance bottlenecks. Summary of the invention

[0005] The embodiments of the present application provide a heterogeneous hybrid acceleration method, device and medium based on multiple acceleration cards to solve the above-mentioned technical problems.

[0006] On the one hand, an embodiment of the present application provides a heterogeneous hybrid acceleration method based on multiple accelerator cards, including:

[0007] Each operator in the deep learning model is combined with the kernel of the corresponding target platform as the initial situation, and the kernel combination is offloaded to the hardware for execution;

[0008] The score of the Kernel combination is used as the initial cost of the cost model, and the cost model is optimized by the gradient descent method to determine whether it is an extreme value;

[0009] Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record;

[0010] The computational subgraphs analyzed by the deep learning model are segmented according to the target tuning records, and the segmented deep learning models are unloaded to corresponding hardware devices for reasoning to achieve heterogeneous hybrid acceleration.

[0011] In one implementation of the present application, before combining each operator in the deep learning model with the Kernel of the corresponding target platform as an initial situation, the method further includes:

[0012] Receive an input deep learning model and parse a computational graph corresponding to a model structure of the deep learning model;

[0013] Determine the corresponding operator call relationship and the weight coefficient corresponding to each operator, and store the operator call relationship and the weight coefficient in a hash table;

[0014] According to the operator call relationship in the hash table, the computation graph is segmented to obtain segmented computation subgraphs;

[0015] Determine the operator corresponding to the computational subgraph and the attribute tag information corresponding to each operator, and determine the Kernel combination of each operator and the corresponding target platform according to the attribute tag information.

[0016] In one implementation of the present application, the score of the Kernel combination is used as the initial cost of the cost model, specifically including:

[0017] A cost model is established according to the calculation time, the hardware computing power and the data transmission time, and weight coefficients corresponding to the calculation time, the hardware computing power and the data transmission time are determined respectively;

[0018] The execution time and score of each kernel in the kernel combination are recorded, and the score of the kernel combination is used as the initial cost of the cost model.

[0019] In one implementation of the present application, the specific calculation method of the cost model is:

[0020]

[0021] in, represents the calculation time, represents the power corresponding to the calculation time, Indicates the hardware computing power. Indicates the power corresponding to the hardware computing power. Indicates the data transmission time. represents the power corresponding to the data transmission time, represents the weight coefficient corresponding to the calculation time, Indicates the weight coefficient corresponding to the hardware computing power situation, Indicates the weight coefficient corresponding to the data transmission time.

[0022] In one implementation of the present application, the cost model is optimized by a gradient descent method, specifically including:

[0023] Based on the initial cost of the cost model, calculating a model score corresponding to the cost model, and calculating a gradient function corresponding to the cost model;

[0024] When the gradient is greater than the preset minimum threshold, an initial Kernel combination is selected from the pre-built operator library, and the initial Kernel combination is reversely iterated in the opposite direction of the gradient;

[0025] When the gradient is equal to the preset minimum threshold, the iteration is stopped and the local minimum value of the cost model score is determined.

[0026] In one implementation of the present application, the extreme value corresponding to the Kernel combination is compared with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record, specifically including:

[0027] Acquire historical cache tuning records, and compare the local minimum value of the cost model score with the historical cache tuning records;

[0028] When the local minimum value is smaller than the historical cache tuning record, the historical cache tuning record is updated and a corresponding target tuning record is determined;

[0029] In the case where the local minimum value is greater than the historical cache tuning record, continue to determine whether the local minimum value meets the current threshold;

[0030] If not, the parameters of the cost model are adjusted to be greater than the local minimum value; if so, the historical cache tuning record is determined to be the target tuning record.

[0031] In one implementation of the present application, the computational subgraphs of the deep learning model analysis are segmented according to the target tuning records, and the segmented deep learning models are unloaded to corresponding hardware devices for reasoning to achieve heterogeneous hybrid acceleration, specifically including:

[0032] According to the target strategy in the target tuning record, the computational subgraph parsed by the deep learning model is segmented, and the hardware device corresponding to each operator after segmentation is determined;

[0033] Each operator after segmentation corresponds to a corresponding hardware device, and the hardware information of each operator after segmentation is stored in the deep learning model; wherein the hardware information is used to represent the operator call relationship between the operator and the hardware device;

[0034] Receive an operator to be processed, and determine a target hardware device corresponding to the operator to be processed according to an operator call relationship corresponding to the hardware information in the deep learning model;

[0035] The deep learning model is unloaded to the target hardware device corresponding to the operator to be processed to perform reasoning on the operator to be processed and realize heterogeneous hybrid acceleration.

[0036] In one implementation of the present application, before taking the kernel combination of each operator in the input model corresponding to the target platform as the initial situation, the method further includes:

[0037] Unify the interfaces corresponding to various accelerator cards of different forms, and encapsulate all the unified interfaces;

[0038] Attribute tag information corresponding to each target platform is obtained, and the attribute tag information is added to the corresponding target platform.

[0039] On the other hand, an embodiment of the present application further provides a heterogeneous hybrid acceleration device based on multiple acceleration cards, the device comprising:

[0040] at least one processor;

[0041] and, a memory communicatively coupled to the at least one processor;

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a heterogeneous hybrid acceleration method based on multiple acceleration cards as described above.

[0043] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, which, when executed, implement a heterogeneous hybrid acceleration method based on multiple acceleration cards as described above.

[0044] The embodiments of the present application provide a heterogeneous hybrid acceleration method, device, and medium based on multiple accelerator cards, which at least have the following beneficial effects:

[0045] By determining the binding status between the operator and the hardware kernel, the actual performance data of the operator on the specific hardware can be directly obtained, providing an accurate benchmark for subsequent optimization; through optimization by cost model and gradient descent method, the optimal operator and hardware combination strategy can be efficiently searched, so that the performance of the deep learning model in a heterogeneous hardware environment is maximized; by continuously comparing and updating the tuning records, it is ensured that each optimization is based on the historical optimal result, thus avoiding duplication of work and possible performance regression, and ensuring the continuity and effectiveness of the optimization process; through sub-graph segmentation and heterogeneous deployment based on tuning records, the advantages of different hardware devices can be fully utilized to achieve efficient coordination and performance maximization of model reasoning, improve the computing efficiency and response speed of the overall system, and achieve efficient heterogeneous hybrid acceleration, thereby providing faster response speed and better user experience in actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 A schematic diagram of a flow chart of a heterogeneous hybrid acceleration method based on multiple accelerator cards provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the internal structure of a heterogeneous hybrid acceleration device based on multiple acceleration cards provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0051] Figure 1 A schematic flow chart of a heterogeneous hybrid acceleration method based on multiple acceleration cards provided in an embodiment of the present application.

[0052] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For the convenience of understanding and description, the following embodiments are described in detail by taking a server as an example.

[0053] It should be noted that the server may be a single device or a system consisting of multiple devices, that is, a distributed server, and this application does not make any specific limitation on this.

[0054] like Figure 1 As shown, an embodiment of the present application provides a heterogeneous hybrid acceleration method based on multiple acceleration cards, including:

[0055] 101. Each operator in the deep learning model is combined with the Kernel of the corresponding target platform as the initial situation, and the Kernel combination is unloaded to the hardware for execution.

[0056] This application discloses a heterogeneous hybrid acceleration method based on multiple acceleration cards, which is mainly implemented based on existing FPGA (Xilinx U280), GPGPU (A100, Ascend 910), and NPU (MLU220) computing acceleration cards, implements operators for running tasks based on various hardware platforms, and builds an operator library for the entire platform system.

[0057] In one embodiment of the present application, before combining each operator in the deep learning model with the Kernel of the corresponding target platform as the initial situation, the following steps are included:

[0058] Receive an input deep learning model and parse the computational graph corresponding to the model structure of the deep learning model;

[0059] Determine the corresponding operator call relationship and the weight coefficient corresponding to each operator, and store the operator call relationship and weight coefficient in a hash table;

[0060] According to the operator call relationship in the hash table, the computation graph is segmented to obtain the segmented computation subgraphs;

[0061] Determine the operator corresponding to the computational subgraph and the attribute tag information corresponding to each operator, and determine the Kernel combination of each operator and the corresponding target platform based on the attribute tag information.

[0062] In one embodiment, it is assumed that a deep learning model based on the TensorFlow framework is received, which is used for an image classification task. The structure of the model is defined as a computational graph, which contains multiple layers and nodes, each node representing an operation or operator, such as convolution, full connection, activation function, etc.

[0063] Load the model file and use the API provided by the TensorFlow framework to read the model, such as tf.keras.models.load_model. Use TensorFlow's graph tools to obtain the model's computational graph representation, such as tf.Graph and tf.Session, or use the compatibility mode of tf.function and tf.Graph in TensorFlow 2.x. Parse the computational graph to identify all nodes and their interconnections.

[0064] For each node, record its operator type, such as Conv2D, MatMul, ReLU, etc. Determine the input and output tensors of each operator, as well as the calling relationship with other operators, and extract the weight coefficients of each operator, such as the weight matrix of the convolution kernel and the fully connected layer. Store this information in a hash table. The key of the hash table is the unique identifier of the operator, which can be the name or ID of the node, and the value is a structure containing the operator type, calling relationship, and weight coefficient.

[0065] In order to optimize the calculation or perform parallel processing, the calculation graph is divided into multiple calculation subgraphs. The division strategy can be based on specific needs, such as minimizing data transmission between subgraphs, balancing the calculation load, etc. For example, the calculation graph can be divided according to the convolution layer and the fully connected layer, and each convolution layer and its subsequent activation function is regarded as a calculation subgraph. Using the call relationship in the hash table, the boundaries of these nodes are identified and new subgraph objects are created.

[0066] For each computational subgraph, further analyze the operators it contains and collect attribute tag information of each operator, such as data type, tensor shape, step size, filling method, etc. Assuming that the target platform is a GPU, you need to select the optimal GPU Kernel for each operator. This usually involves finding a Kernel implementation that matches the operator attributes and the target platform. For example, for a Conv2D operator, you may need to select the most appropriate implementation in the GPU's Kernel library based on information such as its step size, convolution kernel size, and data types of input and output tensors. Finally, combine these Kernel components with the computational subgraph. Figure 1 Together, they form an optimized model representation that can be efficiently executed on the target platform.

[0067] In one embodiment of the present application, since different heterogeneous acceleration cards use different interfaces, it brings great challenges to the integration and unification of the upper layer. Therefore, before taking the Kernel combination of each operator in the input model corresponding to the target platform as the initial situation, the present application unifies the interfaces corresponding to various different forms of acceleration cards, and encapsulates all the unified interfaces; obtains the attribute tag information corresponding to each target platform, and adds the attribute tag information to the corresponding target platform.

[0068] In one embodiment, in order to unify the interfaces corresponding to various different forms of acceleration cards and encapsulate all the unified interfaces, obtain the attribute marking information corresponding to each target platform, and add the attribute marking information to the corresponding target platform, first, the system automatically detects and identifies various acceleration cards connected in the current system through a preset identification module, including but not limited to different types of acceleration cards such as GPU, FPGA, ASIC, etc.

[0069] Secondly, according to the identified accelerator card type, a unified interface specification is designed. This specification covers the basic functions of the accelerator card, such as data transmission, computing task submission, status query, etc. After that, for each type of accelerator card, a corresponding interface adapter is developed. These adapters are responsible for converting the native interface of the accelerator card into an interface under the unified interface specification, thereby achieving interface unification. Then, the unified interface is encapsulated to form an easy-to-use API library. This API library can shield the differences between the underlying accelerator cards, so that the upper-level applications do not need to care about which specific accelerator card is used at the bottom.

[0070] A set of attribute tag information is defined for each target platform, including but not limited to the platform's hardware configuration, operating system version, installed accelerator card type and quantity, etc. The target platform can be different servers, workstations, etc. The attribute tag information of each target platform is automatically obtained by system query, configuration file reading, etc. For example, hardware information can be obtained by querying the system's hardware configuration file, and the operating system version can be obtained by reading the operating system version information.

[0071] Add the acquired attribute tag information to the corresponding target platform. This is achieved by creating or updating a configuration file in the system, which records each target platform and its corresponding attribute tag information. When the upper-layer application needs to call the accelerator card, it can first query the attribute tag information of the target platform to determine what types of accelerator cards the platform supports and the number of accelerator cards. Then, select the appropriate accelerator card based on this information and call its function through a unified interface.

[0072] Assume that there is a server that contains two accelerator cards, GPU and FPGA. The system first detects these two accelerator cards through the identification module and develops interface adapters for them respectively. Then, the interfaces of these adapters are uniformly encapsulated into a set of API libraries. At the same time, the system obtains the attribute tag information of the server, such as hardware configuration (including GPU and FPGA), operating system version, etc., and adds this information to the configuration file of the server. When the upper-level application needs to call the accelerator card for calculation, it can first query the attribute tag information of the server to determine which accelerator cards are available on the server, and then call the functions of these accelerator cards through a unified interface.

[0073] 102. The score of the Kernel combination is used as the initial cost of the cost model, and the cost model is optimized through the gradient descent method to determine whether it is an extreme value.

[0074] Specifically, in one embodiment of the present application, the score of the Kernel combination is used as the initial cost of the cost model, which specifically includes:

[0075] According to the computing time, hardware computing power and data transmission time, a cost model is established, and the weight coefficients corresponding to the computing time, hardware computing power and data transmission time are determined respectively;

[0076] Record the execution time and score of each kernel in the kernel combination, and use the score of the kernel combination as the initial cost of the cost model.

[0077] In one embodiment, during the deep learning model optimization and deployment process, multiple factors need to be considered comprehensively to evaluate the performance of different kernel combinations on specific hardware. To this end, a cost model is established, which includes three main components: computing time, hardware computing power, and data transmission time.

[0078] It should be noted that the computing time in the embodiments of the present application refers to the time required to execute a specific Kernel, which is usually related to the complexity of the Kernel, the size of the input data, and the floating-point computing capability of the hardware. Hardware computing power refers to the efficiency of the hardware in performing specific types of calculations, such as floating-point operations, integer operations, matrix multiplication, etc., which can be measured by the number of operations per second (OPS). Data transmission time refers to the time required to transfer data between hardware or between hardware and memory, which depends on the amount of data, transmission bandwidth, and transmission protocol.

[0079] In order to combine these three factors, weight coefficients are assigned to each of them, which reflect the importance of each factor in a specific application scenario. For example, in scenarios with high real-time requirements, the weight of computing time may be higher; while in data-intensive applications, the weight of data transmission time may be more critical.

[0080] Assume that through expert evaluation, experimental measurement or historical data analysis, the following weight coefficients are determined: computing time weight: 0.5, hardware computing power weight: 0.3, data transmission time weight: 0.2. The cost model can be expressed as the weighted sum of these factors: cost = computing time * 0.5 + hardware computing power * 0.3 + data transmission time * 0.2.

[0081] After determining the cost model, different kernel combinations need to be evaluated. To this end, first record the execution time and score of each kernel on the target hardware. Unload all kernels in this group to the hardware device for execution, and record the execution time of each kernel. Then use the stochastic gradient descent method to select another group of corresponding kernels in the operator library, unload this group of kernels to the hardware device for execution, and record the corresponding scores.

[0082] In one embodiment of the present application, the specific calculation method of the cost model is:

[0083]

[0084] It should be noted that the embodiments of the present application represents the calculation time, Represents the power corresponding to the calculation time, Indicates the hardware computing power. Indicates the power corresponding to the hardware computing power. Indicates the data transmission time. Indicates the power corresponding to the data transmission time, Indicates the weight coefficient corresponding to the calculation time, Indicates the weight coefficient corresponding to the hardware computing power. Indicates the weight coefficient corresponding to the data transmission time. The weight coefficient in this application can be set according to the actual situation, and the sum of the weight coefficient corresponding to the calculation time, the weight coefficient corresponding to the hardware computing power, and the weight coefficient corresponding to the data transmission time is 1, that is, .

[0085] In one embodiment, the power corresponding to the cost model , the power corresponding to the hardware computing power , and the power corresponding to the data transmission time The power can be selected according to the actual situation, and this application does not make any specific restrictions on this. For example, , , , then the cost model corresponds to .

[0086] In one embodiment of the present application, the cost model is optimized by a gradient descent method, specifically including:

[0087] Based on the initial cost of the cost model, calculate the model score corresponding to the cost model, and calculate the gradient function corresponding to the cost model;

[0088] When the gradient is greater than the preset minimum threshold, an initial Kernel combination is selected from the pre-built operator library, and the initial Kernel combination is reversely iterated in the opposite direction of the gradient.

[0089] When the gradient is equal to the preset minimum threshold, the iteration stops and the local minimum of the cost model score is determined.

[0090] In one embodiment, the cost model is optimized using a gradient descent algorithm. The specific steps are as follows: select a set of initial Kernel and obtain Parameters, use this set of parameters as the initial value, calculate the cost model score f(x), and calculate the gradient function . When the gradient is small enough, stop the iteration and let , then a set of local minimum values ​​is found. Represents an intermediate value, i can be 1, 2, 3, , , They represent temporary recorded values ​​for computing time, hardware computing power, and transmission time, respectively. , , They respectively represent a set of current data records corresponding to the calculation time, hardware computing power, and transmission time at time t.

[0091] Otherwise, according to the formula Seek Specifically, let Iterate in the opposite direction of the gradient, that is, the time t+1 is calculated based on the x and gradient at time t.

[0092] In one embodiment, assume that a machine learning model optimization system is being developed that aims to minimize a specific cost function, such as mean square error, cross entropy loss, etc., by adjusting model parameters. First, a cost model is defined that calculates an initial cost based on the current model parameters. For example, in a linear regression problem, the cost model can be the sum of the squares of the differences between the predicted value and the true value. Next, based on the initial cost, a predefined scoring mechanism, such as accuracy, F1 score, or the negative value of the cost itself, is used to calculate the model score corresponding to the cost model. Assume that the scoring mechanism is to simply replace the negative value of the price as the score, that is, score = - initial cost. Using the partial derivative method in calculus, the gradient of the cost model relative to each model parameter is calculated to form a gradient vector.

[0093] Next, check each element in the gradient vector to determine whether its absolute value is greater than the preset minimum threshold. This threshold is used to decide when to stop the iteration to avoid wasting computing resources on gradients that are numerically close to zero but theoretically not zero. If the gradient is greater than the preset minimum threshold, select a set of initial Kernel combinations from the pre-built operator library. These Kernels are pre-optimized basic computing units, such as convolution kernels, activation functions, etc., which are used to accelerate parameter updates during gradient descent.

[0094] After that, the selected initial Kernel combination is used to perform reverse iteration in the opposite direction of the gradient (i.e., the negative gradient direction) to update the model parameters so that the cost function value of the next iteration is expected to decrease. During the iteration process, the gradient is recalculated at each step, and the Kernel combination is dynamically adjusted to adapt to the current gradient state to ensure the efficiency and stability of the optimization process.

[0095] When the absolute values ​​of all gradients are equal to or less than the preset minimum threshold, the iteration stops. At this point, it is considered that the cost model score has reached a local minimum, and further parameter adjustments can no longer bring significant cost reductions. Record the model parameter configuration and the corresponding cost model score at this time as the result of the optimization process.

[0096] 103. Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record.

[0097] Specifically, in one embodiment of the present application, the extreme value corresponding to the Kernel combination is compared with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record, specifically including:

[0098] Obtain historical cache tuning records, and compare the local minimum value of the cost model score with the historical cache tuning records;

[0099] When the local minimum value is smaller than the historical cache tuning record, the historical cache tuning record is updated and the corresponding target tuning record is determined;

[0100] When the local minimum value is greater than the historical cache tuning record, continue to determine whether the local minimum value meets the current threshold;

[0101] If not, the parameters of the cost model are adjusted to be greater than the local minimum value. If so, the historical cache tuning record is determined as the target tuning record.

[0102] In one embodiment, in a scenario aimed at optimizing the performance of a deep learning model, an automated tuning process is implemented, which includes calculating a cost model score, comparing it with historical cache tuning records, and making decisions based on the comparison results.

[0103] First, retrieve the historical tuning records from a dedicated database or cache system. These records contain the minimum value of the cost model score in the previous tuning process, the corresponding model parameter configuration, and the timestamp of the tuning. In the current tuning iteration, a new local minimum has been calculated by the cost model. Next, compare this local minimum with the minimum value in the historical cache tuning record.

[0104] If the current local minimum value is smaller than the minimum value in the historical cache tuning record, it means that a better solution has been found. At this time, the historical cache tuning record is updated, and the current local minimum value, the corresponding model parameter configuration, and the current timestamp are stored in the database. At the same time, the current record is determined as the target tuning record.

[0105] If the current local minimum is greater than the minimum in the historical cache tuning record, the current solution is not abandoned immediately, but further evaluated to see if it meets the currently set threshold. This threshold may be dynamically determined based on business needs, computing resource limitations, or tuning strategies.

[0106] If the current local minimum does not meet the current threshold (that is, it is larger than the threshold), the current solution is not optimal enough and needs to be further tuned. At this time, you can adopt strategies, such as adjusting the parameters of the cost model, to make it jump out of the current local minimum and explore other possible solution spaces. If the current local minimum meets the current threshold (that is, it is smaller than or equal to the threshold), although it is not the historical optimal, it is good enough under the current conditions. At this time, the historical cache tuning record (that is, the previous optimal solution) is determined as the target tuning record, and the current tuning iteration may be ended, or the next round of tuning may be entered according to business needs.

[0107] 104. Divide the computational subgraphs of deep learning model analysis according to the target tuning records, and unload the divided deep learning models to the corresponding hardware devices for reasoning to achieve heterogeneous hybrid acceleration.

[0108] Specifically, in one embodiment of the present application, the computational subgraphs of the deep learning model analysis are segmented according to the target tuning records, and the segmented deep learning models are unloaded to the corresponding hardware devices for reasoning to achieve heterogeneous hybrid acceleration, which specifically includes:

[0109] According to the target strategy in the target tuning record, the computational subgraph of the deep learning model analysis is segmented, and the hardware device corresponding to each operator after segmentation is determined;

[0110] Each operator after segmentation is mapped to a corresponding hardware device, and the hardware information of each operator after segmentation is stored in the deep learning model; wherein the hardware information is used to represent the operator call relationship between the operator and the hardware device;

[0111] Receive the operator to be processed, and determine the target hardware device corresponding to the operator to be processed according to the operator call relationship corresponding to the hardware information in the deep learning model;

[0112] Offload the deep learning model to the target hardware device corresponding to the operator to be processed to perform reasoning on the operator to be processed and achieve heterogeneous hybrid acceleration.

[0113] In one embodiment, the server finds the optimal solution through the gradient descent method in the target tuning record, and adjusts the correspondence between the operator devices according to the set of optimal solutions, that is, the calculation subgraph is segmented to ensure that the correspondence between the segmented operator and the corresponding hardware device is the optimal solution.

[0114] In one embodiment, in a deep learning model optimization and deployment scenario, the goal is to achieve efficient reasoning of the model by rationally utilizing heterogeneous hardware resources (such as CPU, GPU, FPGA, etc.). First, the computational subgraph after parsing the deep learning model is segmented according to the target strategy in the target tuning record. This segmentation process is based on the computational structure and dependency of the model, splitting the model into multiple operators that can be executed independently. During the segmentation process, the computational amount, data dependency, and characteristics of the operator are considered to ensure that the segmented operator can be efficiently mapped to the hardware device.

[0115] For each operator after segmentation, its corresponding target hardware device is determined based on its computing characteristics and the performance characteristics of the hardware device. For example, computationally intensive operators may be assigned to the GPU, while data preprocessing or simple logical operations may be assigned to the CPU. The data transmission overhead between hardware devices is also considered to minimize cross-device data movement to improve overall inference efficiency.

[0116] After segmentation, each operator and its corresponding hardware device information are stored in the deep learning model. The hardware information includes not only the mapping relationship between the operator and the hardware device, but also the specific execution unit on the device (such as a core of a GPU), memory address and other detailed information. The hardware information is used to represent the operator call relationship between the operator and the hardware device, and is an important basis for scheduling the operator during subsequent model inference.

[0117] In the model inference phase, operators to be processed are received. Based on the hardware information stored in the deep learning model, the target hardware device corresponding to the operator to be processed can be quickly determined. This process is achieved by looking up the hardware information table or calling the corresponding API, ensuring that the operator can be accurately scheduled to the correct hardware device.

[0118] Offload the deep learning model (or part of the model) to the target hardware device corresponding to the operator to be processed. This usually involves transferring the model's weights, computational graphs, and necessary runtime environment to the target device. On the target device, use the corresponding inference engine or library to execute the operator to be processed to achieve inference of the deep learning model. Since the operator has been optimized and scheduled according to the hardware device, heterogeneous hybrid acceleration can be achieved to improve inference efficiency. Different types of operators are reasonably assigned to different hardware devices for execution, making full use of the advantages of each hardware device, improving the overall inference performance of the model, and achieving hybrid acceleration of deep learning models in heterogeneous hardware environments.

[0119] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a heterogeneous hybrid acceleration device based on multiple acceleration cards, whose structure is as follows: Figure 2 shown.

[0120] Figure 2 The internal structure diagram of a heterogeneous hybrid acceleration device based on multiple acceleration cards provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the device includes:

[0121] at least one processor;

[0122] and, a memory communicatively coupled to the at least one processor;

[0123] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to:

[0124] Each operator in the deep learning model is combined with the kernel of the corresponding target platform as the initial situation, and the kernel combination is offloaded to the hardware for execution;

[0125] The score of the Kernel combination is used as the initial cost of the cost model, and the cost model is optimized through the gradient descent method to determine whether it is an extreme value;

[0126] Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record;

[0127] The computational subgraphs of deep learning model analysis are segmented according to the target tuning records, and the segmented deep learning models are offloaded to the corresponding hardware devices for inference to achieve heterogeneous hybrid acceleration.

[0128] The present application also provides a non-volatile computer storage medium storing computer executable instructions. When the computer executable instructions are executed, they can:

[0129] Each operator in the deep learning model is combined with the kernel of the corresponding target platform as the initial situation, and the kernel combination is offloaded to the hardware for execution;

[0130] The score of the Kernel combination is used as the initial cost of the cost model, and the cost model is optimized through the gradient descent method to determine whether it is an extreme value;

[0131] Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record;

[0132] The computational subgraphs of deep learning model analysis are segmented according to the target tuning records, and the segmented deep learning models are offloaded to the corresponding hardware devices for inference to achieve heterogeneous hybrid acceleration.

[0133] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0134] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0141] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A heterogeneous hybrid acceleration method based on multiple accelerator cards, characterized in that: The method comprises: Each operator in the deep learning model is combined with the kernel of the corresponding target platform as the initial situation, and the kernel combination is offloaded to the hardware for execution; The score of the Kernel combination is used as the initial cost of the cost model, and the cost model is optimized by the gradient descent method to determine whether it is an extreme value; Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record; Splitting the computational subgraphs analyzed by the deep learning model according to the target tuning records, and unloading the split deep learning model to the corresponding hardware device for reasoning, thereby achieving heterogeneous hybrid acceleration; Before combining each operator in the deep learning model with the Kernel of the corresponding target platform as an initial situation, the method further includes: Receive an input deep learning model and parse a computational graph corresponding to a model structure of the deep learning model; Determine the corresponding operator call relationship and the weight coefficient corresponding to each operator, and store the operator call relationship and the weight coefficient in a hash table; According to the operator call relationship in the hash table, the computation graph is segmented to obtain segmented computation subgraphs; Determine the operator corresponding to the computational subgraph and the attribute tag information corresponding to each operator, and determine the Kernel combination of each operator and the corresponding target platform according to the attribute tag information; The specific calculation method of the cost model is: in, represents the calculation time, represents the power corresponding to the calculation time, Indicates the hardware computing power. Indicates the power corresponding to the hardware computing power. Indicates the data transmission time. represents the power corresponding to the data transmission time, represents the weight coefficient corresponding to the calculation time, Indicates the weight coefficient corresponding to the hardware computing power situation, Represents a weight coefficient corresponding to the data transmission time; Compare the extreme value corresponding to the Kernel combination with the historical record to determine whether to update the tuning record to determine the corresponding target tuning record, specifically including: Obtaining historical cache tuning records, and comparing the local minimum value of the cost model score with the historical cache tuning records; When the local minimum value is smaller than the historical cache tuning record, the historical cache tuning record is updated and a corresponding target tuning record is determined; In the case where the local minimum value is greater than the historical cache tuning record, continue to determine whether the local minimum value meets the current threshold; If not, adjusting the parameters of the cost model to be greater than the local minimum value; if so, determining the historical cache tuning record as the target tuning record; The computational subgraphs analyzed by the deep learning model are segmented according to the target tuning records, and the segmented deep learning models are unloaded to corresponding hardware devices for reasoning to achieve heterogeneous hybrid acceleration, specifically including: According to the target strategy in the target tuning record, the computational subgraph parsed by the deep learning model is segmented, and the hardware device corresponding to each operator after segmentation is determined; Each operator after segmentation corresponds to a corresponding hardware device, and the hardware information of each operator after segmentation is stored in the deep learning model; wherein the hardware information is used to represent the operator call relationship between the operator and the hardware device; Receive an operator to be processed, and determine a target hardware device corresponding to the operator to be processed according to an operator call relationship corresponding to the hardware information in the deep learning model; The deep learning model is unloaded to the target hardware device corresponding to the operator to be processed to perform reasoning on the operator to be processed and realize heterogeneous hybrid acceleration.

2. The heterogeneous hybrid acceleration method based on multiple accelerator cards according to claim 1, characterized in that: The score of the Kernel combination is used as the initial cost of the cost model, specifically including: A cost model is established according to the calculation time, the hardware computing power and the data transmission time, and weight coefficients corresponding to the calculation time, the hardware computing power and the data transmission time are determined respectively; The execution time and score of each kernel in the kernel combination are recorded, and the score of the kernel combination is used as the initial cost of the cost model.

3. The heterogeneous hybrid acceleration method based on multiple accelerator cards according to claim 1, characterized in that: The cost model is optimized by the gradient descent method, specifically including: Based on the initial cost of the cost model, calculating a model score corresponding to the cost model, and calculating a gradient function corresponding to the cost model; When the gradient is greater than the preset minimum threshold, an initial Kernel combination is selected from the pre-built operator library, and the initial Kernel combination is reversely iterated in the opposite direction of the gradient; When the gradient is equal to the preset minimum threshold, the iteration is stopped and the local minimum value of the cost model score is determined.

4. The heterogeneous hybrid acceleration method based on multiple accelerator cards according to claim 1, characterized in that: Before taking the kernel combination of each operator in the input model corresponding to the target platform as the initial situation, the method further includes: Unify the interfaces corresponding to various accelerator cards of different forms, and encapsulate all the unified interfaces; Attribute tag information corresponding to each target platform is obtained, and the attribute tag information is added to the corresponding target platform.

5. A heterogeneous hybrid acceleration device based on multiple acceleration cards, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a heterogeneous hybrid acceleration method based on multiple acceleration cards as described in any one of claims 1-4.

6. A non-volatile computer storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed, a heterogeneous hybrid acceleration method based on multiple acceleration cards as described in any one of claims 1 to 4 is implemented.

Images