Deep learning accelerator software and hardware cooperation method and device

Abstract

The invention discloses a deep learning accelerator software and hardware cooperation method and device, and the method comprises the steps: configuring a sample through a software optimizer based on an initial hardware parameter, determining the operation information of a candidate model, and carrying out the calculation of the operation information of the candidate model through a hardware optimizer based on a task efficiency representation value corresponding to the operation information of the candidate model, determining an initial hardware parameter configuration sample of the next round of iteration, and if it is monitored that the deviation between the task efficiency representation values corresponding to the initial hardware parameter configuration samples corresponding to the front and back rounds after the preset round is reached is smaller than a preset deviation, sending the initial hardware parameter configuration sample obtained when the preset iteration condition is met to the next round of iteration; and taking a candidate model operation mode corresponding to the candidate model operation information determined by the software optimizer based on the target hardware parameter configuration as a target model operation mode. And continuously carrying out cooperative configuration on software and hardware through multi-round iteration so as to realize optimal accelerator specification parameters for a given model set.

Description

Technical Field

[0001] This specification relates to the fields of computer technology and artificial intelligence, and in particular to a method and apparatus for hardware and software collaboration in deep learning accelerators. Background Technology

[0002] A deep learning accelerator is a piece of hardware specifically designed to accelerate deep learning computations. With the rapid development of deep learning technology, the amount of computation required to process large amounts of data and complex models is also constantly increasing. Therefore, deep learning accelerators play an important role in improving computational performance and efficiency.

[0003] Deep learning accelerators are a type of AI chip used to accelerate computation in deep learning and artificial intelligence tasks.

[0004] Currently, in the design of artificial intelligence (AI) chips, a hardware-software integrated design approach is typically adopted to fully utilize the computational and memory access characteristics of AI models in order to achieve more efficient computing and better performance.

[0005] In AI chip design, it is necessary to carefully configure hardware resources such as computing, storage and communication, and also to ensure that the AI ​​model has good performance and fast computing speed when running on the chip. Therefore, when deploying the model, it is necessary to design corresponding deployment schemes at both the hardware and software levels. However, the current deployment scheme strategies cannot effectively improve the running speed and performance of the model.

[0006] Furthermore, the simultaneous operation of multiple models and the success of AI algorithms in single tasks (such as hand tracking, depth estimation, and speech recognition) have led to the emergence of multi-model AI workloads. Efficient AI accelerator design is crucial to meet the ever-increasing computational demands and evolving application requirements of modern AI workloads, necessitating efficient dedicated chip design methodologies.

[0007] Faced with the above challenges, how to deploy the model more accurately is an urgent problem to be solved. Summary of the Invention

[0008] This specification provides a method and apparatus for hardware and software collaboration in deep learning accelerators, which partially solves the problems existing in the prior art.

[0009] The embodiments in this specification adopt the following technical solutions: This specification provides a hardware / software co-processing method for deep learning accelerators, including: Obtain the model parameters of the target model and the accelerator area constraint, wherein the accelerator area constraint is used to characterize the hardware configuration budget allocated to the target model by the accelerator; Based on the hardware parameter configuration contained in the accelerator area constraint, an initial hardware parameter configuration sample is determined; The hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters are input into a preset software optimizer, so that the software optimizer determines the running information of each initial model under the given hardware running environment corresponding to the initial hardware parameter configuration sample, and determines the running information of candidate models based on the running information of each initial model. The candidate model running information is input into a preset analysis model. The analysis model predicts the task efficiency representation value obtained when the target model's computation task is executed in the hardware environment corresponding to the initial hardware parameter configuration sample according to the candidate model running mode corresponding to the candidate model running information. Based on the task efficiency representation value corresponding to the candidate model running mode, the initial hardware parameter configuration sample for the next iteration is determined under the given candidate model running mode. The initial hardware parameter configuration sample for the next iteration is sent to the software optimizer so that the software optimizer determines the candidate model running mode corresponding to the next iteration based on the initial hardware parameter configuration sample for the next iteration and the model parameters. This process continues until the preset iteration conditions are met, resulting in the target model running mode and the target hardware parameter configuration. Configure the target hardware parameters on the target device to configure the corresponding target hardware operating environment, and deploy the target model on the target device based on the target hardware operating environment, so as to execute the computational tasks corresponding to the target model on the target device in accordance with the target model's operating mode.

[0010] Optionally, based on the initial model running information, candidate model running information is determined, specifically including: Sampling is performed on the initial model running information to obtain sampled model running information; The sampling model running information and the initial hardware parameter configuration sample are input into a preset evaluator so that the evaluator can predict the task efficiency characterization value obtained when the target model's computation task is executed in the hardware running environment corresponding to the initial hardware parameter configuration sample according to the sampling model running information, and use it as the sampling characterization value. The sampling characterization value, the sampling model running information, and the initial hardware parameter configuration sample are input into the software optimizer so that the sampling characterization value and the sampling model running information are fitted by the software optimizer in the hardware running environment corresponding to the initial hardware parameter configuration sample to obtain the first optimized model. Based on the running information of each initial model, the running information of the candidate model is determined using the first optimization model.

[0011] Optionally, based on the task efficiency representation value corresponding to the candidate model's operating mode, the initial hardware parameter configuration sample for the next iteration is determined under the given candidate model's operating mode, specifically including: The task efficiency characterization value corresponding to the candidate model running mode, the candidate model running mode, and the initial hardware parameter configuration sample are input into a preset hardware optimizer, so that the hardware optimizer can determine the initial hardware parameter configuration sample for the next iteration under the given candidate model running mode.

[0012] Optionally, the hardware optimizer determines the initial hardware parameter configuration sample for the next iteration under the given candidate model operating mode, specifically including: Based on the candidate model's operating mode, the hardware optimizer is used to fit the task efficiency representation value corresponding to the candidate model's operating mode and the initial hardware parameter configuration sample to obtain a second optimized model. Based on the hardware parameter configurations required when performing tasks using the target model, the second optimization model determines samples of basic hardware parameter configurations under the given operating mode of the candidate model. The basic hardware parameter configuration samples are input into a preset evaluator so that the evaluator can predict the task efficiency characterization value obtained when the target model is executed in the hardware running environment corresponding to each basic hardware parameter configuration sample. This value is used as the task efficiency characterization value corresponding to each basic hardware parameter configuration sample. Based on the task efficiency representation value corresponding to each basic hardware parameter configuration sample, the initial hardware parameter configuration sample for the next iteration is determined.

[0013] Optionally, until the preset iteration conditions are met, the target model operation mode and target hardware parameter configuration are obtained, specifically including: If the deviation between the task efficiency representation values ​​corresponding to the initial hardware parameter configuration samples of the two rounds before and after reaching the preset round is less than the preset deviation, then the preset iteration condition is determined to be met, and the initial hardware parameter configuration sample obtained under the preset iteration condition is taken as the target hardware parameter configuration, and the candidate model running mode determined by the software optimizer based on the target hardware parameter configuration is taken as the target model running mode.

[0014] Optionally, the target model includes at least one model for performing computational tasks.

[0015] This specification provides a hardware and software co-processing device for a deep learning accelerator, comprising: The acquisition module is used to acquire the model parameters of the target model and the accelerator area constraint, wherein the accelerator area constraint is used to characterize the hardware configuration budget allocated by the accelerator to the target model; The first determining module is used to determine an initial hardware parameter configuration sample based on the hardware parameter configuration contained in the accelerator area constraint. The second determining module is used to input the hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters into a preset software optimizer, so that the software optimizer determines the running information of each initial model under the given hardware running environment corresponding to the initial hardware parameter configuration strategy, and determines the running information of candidate models based on the running information of each initial model. An analysis module is used to input the candidate model running information into a preset analysis model, so as to predict the task efficiency characterization value obtained when the target model's computation task is executed in the hardware running environment corresponding to the initial hardware parameter configuration sample according to the candidate model running mode corresponding to the candidate model running information. Based on the task efficiency characterization value corresponding to the candidate model running mode, the module determines the initial hardware parameter configuration sample for the next iteration under the given candidate model running mode, and sends the initial hardware parameter configuration sample for the next iteration to the software optimizer, so that the software optimizer determines the candidate model running mode corresponding to the next iteration based on the initial hardware parameter configuration sample for the next iteration and the model parameters, until the preset iteration conditions are met, and obtains the target model running mode and the target hardware parameter configuration. The running module is used to configure the target hardware runtime environment corresponding to the target hardware parameter configuration sample on the target device, and deploy the target model on the target device based on the target hardware runtime environment, so as to execute the computational tasks corresponding to the target model on the target device according to the target model's running mode.

[0016] Optionally, the second determining module is specifically configured to: sample from the initial model running information to obtain sampled model running information; input the sampled model running information and the initial hardware parameter configuration sample into a preset evaluator, so that the evaluator predicts the task efficiency representation value obtained when the target model's computation task is executed according to the sampled model running information in the hardware running environment corresponding to the initial hardware parameter configuration sample, and uses this as the sampled representation value; input the sampled representation value, the sampled model running information, and the initial hardware parameter configuration sample into the software optimizer, so that the software optimizer fits the sampled representation value and the sampled model running information in the hardware running environment corresponding to the initial hardware parameter configuration sample to obtain a first optimized model; and determine candidate model running information based on the initial model running information using the first optimized model.

[0017] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned deep learning accelerator hardware-software co-processing method.

[0018] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned deep learning accelerator hardware-software co-operation method.

[0019] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: In the embodiments of this specification, each iteration uses a software optimizer to determine candidate model operation information based on the initial hardware parameter configuration sample and the initial model operation information. Then, the hardware optimizer determines the initial hardware parameter configuration sample for the next iteration based on the task efficiency representation value corresponding to the candidate model operation mode, given the candidate model operation mode. If the deviation between the task efficiency representation values ​​corresponding to the initial hardware parameter configuration samples of two iterations after reaching a preset number of iterations is less than a preset deviation, the initial hardware parameter configuration sample obtained when the preset iteration conditions are met is used as the target hardware parameter configuration, and the candidate model operation mode determined by the software optimizer based on the target hardware parameter configuration is used as the target model operation mode. Subsequently, the target hardware operating environment is configured on the target device according to the target hardware parameter configuration, and the target model is deployed on the target device according to the target model operation mode to achieve the optimal accelerator specifications for a given set of models.

[0020] In this approach, the model and accelerator hardware are continuously designed collaboratively through multiple iterations to obtain the optimal model deployment method and corresponding hardware parameter configuration. This achieves the optimal accelerator specifications for a given set of models, enabling more efficient execution of computational tasks on the accelerator and significantly improving the utilization of accelerator hardware resources and the computational efficiency of the model. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating a deep learning accelerator hardware-software co-operation method provided in an embodiment of this specification; Figure 2 A schematic diagram of a deep learning accelerator hardware and software co-processing system architecture is provided for embodiments of this specification; Figure 3 A schematic diagram of a hardware and software co-working device for a deep learning accelerator provided in the embodiments of this specification; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0023] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart illustrating a deep learning accelerator hardware-software co-operation method provided in this specification includes: S100: Obtain the model parameters of the target model and the accelerator area constraint, wherein the accelerator area constraint is used to characterize the hardware configuration budget allocated to the target model by the accelerator.

[0025] A deep learning accelerator is an AI chip used to accelerate the computation process in deep learning and artificial intelligence tasks by allocating the hardware resources of the AI ​​chip to improve computational efficiency.

[0026] In the design of artificial intelligence (AI) chips, a hardware-software integrated design approach is typically adopted to fully utilize the computational and memory access characteristics of AI models in order to achieve more efficient computing and better performance.

[0027] The design process requires careful configuration of hardware resources such as computing, storage, and communication. It is also necessary to ensure that the AI ​​model, after being deployed on the chip using this design method, has good model performance and fast computing speed.

[0028] Currently, the collaborative exploration of hardware and software using spatial search algorithms involves two aspects: design spatial search and mapping spatial search. Design spatial search targets the hardware aspect, while mapping spatial search focuses on how to better map the computational methods of AI model algorithms onto hardware resources for execution of computational tasks. Spatial search algorithms can find the optimal design scheme for AI chip performance within a reasonable timeframe, such as random search, grid search, and evolutionary search. Although all of these can yield relatively reasonable AI chip design schemes, they are only effective when designing for AI chips that meet specific requirements, thus having certain limitations.

[0029] In summary, while existing technologies can deploy models on deep learning accelerators, they still result in wasted hardware resources and low model computation efficiency. To address these issues, in the embodiments of this specification, each iteration uses a software optimizer to determine candidate model operation information based on initial hardware parameter configuration samples and initial model operation information. Then, the hardware optimizer determines the initial hardware parameter configuration sample for the next iteration based on the task efficiency representation value corresponding to the candidate model operation mode, given the candidate model operation mode. If the deviation between the task efficiency representation values ​​of the initial hardware parameter configuration samples from two consecutive iterations after a preset iteration is less than a preset deviation, the initial hardware parameter configuration sample obtained when the preset iteration conditions are met is used as the target hardware parameter configuration, and the candidate model operation mode determined by the software optimizer based on the target hardware parameter configuration is used as the target model operation mode. Furthermore, the target hardware runtime environment is configured on the target device according to the target hardware parameter configuration, and the target model is deployed on the target device according to the target model operation mode, enabling the target device to execute the computational tasks corresponding to the target model more efficiently.

[0030] In this approach, the model and accelerator hardware are continuously designed collaboratively through multiple rounds of iteration to obtain the optimal model deployment method and corresponding hardware parameter configuration. This achieves the optimal accelerator specifications for a given set of models, enabling more efficient execution of computational tasks on deep learning accelerators and significantly improving accelerator hardware resource utilization and model computational efficiency.

[0031] Next, the target device needs to obtain the model parameters of the target model and the accelerator area constraints. The accelerator area constraints are used to characterize the hardware configuration budget allocated to the target model by the accelerator.

[0032] In the embodiments of this specification, after the target device obtains the model parameters of the target model, it will determine the subsequent initial hardware parameter configuration sample based on the model parameters of the target model and the hardware parameter configuration contained in the accelerator area constraint. The target device mentioned in this specification can be a terminal device such as a desktop computer or a laptop computer, or a server, or a dedicated device specifically used for training and deploying the model.

[0033] The accelerator area constraint mentioned here is used to characterize the hardware configuration budget allocated by the accelerator to the target model. The hardware configuration budget includes: Processing Element (PE): responsible for performing basic operations in the deep learning model, such as convolution and rectangular arrays; On-chip network bandwidth: used to represent the efficiency and speed of data transmission between various components inside the accelerator; AI chip computational precision: used to represent the computational precision when the accelerator performs operations; On-chip cache size: used to represent the size of the buffer used to store data inside the accelerator, etc.

[0034] The various components included in the aforementioned hardware configuration budget can actually be reflected through specific information, such as on-chip network bandwidth and AI chip computing accuracy, which can be reflected through specific data or information.

[0035] It is important to note that when deploying the target model, the target model must include at least one model for performing computational tasks. In other words, the target device can perform software and hardware co-design for multiple models to enable a single chip to support application scenarios of multiple different models simultaneously.

[0036] In this specification, the model can take various forms. For example, it can be a large language model or a generative AI model. A large language model can infer from the user's input text to output the content the user needs. A generative AI model mainly synthesizes the content the user needs based on an existing database. Another example is an image recognition model, which can identify objects contained in an input image.

[0037] Therefore, the accelerator area constraints in this specification can actually provide a variety of selectable hardware parameter configurations for the accelerator, which in turn determine which hardware resources can be used in subsequent configuration samples.

[0038] S102: Determine an initial hardware parameter configuration sample based on the hardware parameter configuration contained in the accelerator area constraint.

[0039] In the embodiments of this specification, after the terminal device obtains the hardware parameter configurations required for the target model to perform tasks, it can select computing components, storage components, and communication components based on the model parameters of the target model and the fact that the target device can perform the computational tasks of the target model, thereby obtaining a preliminary hardware parameter configuration sample, that is, an initial hardware parameter configuration sample.

[0040] The initial hardware parameter configuration sample mentioned here is mainly used to reflect how much hardware resources are used in the hardware configuration budget within the accelerator area constraint to execute the target model's computational tasks. For example, in an initial hardware parameter configuration sample, it is specified that 200 PE execution units can be selected, the on-chip network bandwidth is 100MB, the AI ​​chip computation precision is int8, and the on-chip cache size is 180KB.

[0041] By selecting the hardware parameter configurations included in the accelerator area constraints, the requirements for the target model to perform computational tasks are met. Subsequent hardware parameter adjustments are made based on the initial hardware parameter configuration sample to obtain the target hardware parameter configuration, which represents the hardware resources of the accelerator used when the computational efficiency of the target model can reach its optimal level.

[0042] S104: Input the hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters into a preset software optimizer, so that the software optimizer determines the running information of each initial model under the given hardware running environment corresponding to the initial hardware parameter configuration sample, and determines the running information of the candidate model based on the running information of each initial model.

[0043] In this specification, before the target device inputs the initial hardware parameter configuration sample and the model parameters of the target model obtained above into the preset software optimizer, it can obtain various mapping relationships between the computational tasks executed by the target model and the hardware resources based on the initial hardware parameter configuration sample, that is, the hardware resources. That is, various initial model operation information. Then, the software optimizer obtains candidate model operation information based on the various initial model operations and the initial hardware parameter configuration sample, and through the evaluation results after the target model executes the computational tasks according to the various model operation information.

[0044] The initial model running information mentioned here is different from the initial hardware parameter configuration sample mentioned above. The initial model running information is mainly reflected in the execution mode of the computational tasks performed by the model at the software level. That is, the target device can obtain various initial model running modes based on the various mapping relationships between the cyclic computations contained in the computational tasks of the target model and the hardware resources.

[0045] It is important to note that there are various mapping relationships between the loop operations contained in the computational tasks executed by the target model and the hardware resources. These mapping relationships refer to the ability to group each loop in the loop operations contained in the computational tasks executed by the target model. For example, if a loop contains 10 loops, the hardware resources of each loop can be configured, or the 10 loops can be divided into 5 groups, and the accelerator hardware resources of each group can be configured.

[0046] Based on different mapping methods, multiple implementation methods for model computation tasks can be obtained. Executing the target model computation task according to each implementation method will result in different computational efficiencies. By continuously adjusting these implementation methods under the condition of initial hardware parameter configuration samples, candidate model running information, that is, the candidate model running method corresponding to the candidate model running information, can be obtained.

[0047] Specifically, the target device first samples various initial model operation information to obtain sampled model operation information. Then, the sampled model operation information and initial hardware parameter configuration samples are input into a preset evaluator. Through the preset evaluator, the corresponding task efficiency evaluation index is obtained after the target model performs the computation task according to the sampled model operation information in the hardware environment corresponding to the initial hardware parameter configuration samples. This index is used as a sampling representation value. Then, the sampling representation value, sampled model operation information, and initial hardware parameter configuration samples are input into the software optimizer. In the hardware operation environment corresponding to the initial hardware parameter configuration samples, the sampling representation value and sampled model operation information are fitted to obtain the first optimized model. The operation information of each basic candidate model is obtained by using the method of taking the extreme value of the first optimized model.

[0048] For example, after the software optimizer fits the relationship between the sampled representation values ​​and the sampling model operation mode, it obtains a curve. The appropriate basic candidate model operation information can be obtained from the extreme values ​​or the neighborhood of the extreme values ​​on the curve.

[0049] Furthermore, the running information of each basic candidate model and the initial hardware parameter configuration sample are input into the analysis model to predict the task efficiency characterization value obtained when the target model performs the computation task in the hardware running environment corresponding to the initial hardware parameter configuration sample according to the running information of each basic candidate model. This yields the task efficiency characterization value corresponding to the running information of each basic candidate model, so that the software optimizer can obtain the candidate model running information based on the task efficiency characterization value corresponding to the running information of each basic candidate model.

[0050] It is important to note that the task efficiency representation value and sampling representation value mentioned here are both indicators used to evaluate model performance, such as time consumption and energy consumption. Time consumption represents the time required for the model to complete the computation task, which can reflect the execution efficiency of the computation task. Energy consumption represents the amount of electricity used by the model to complete the computation task. Among them, the task efficiency representation value is the model performance evaluation of the analysis model based on the running information of each basic candidate model obtained by the software optimizer. The sampling representation value refers to the model performance evaluation value corresponding to the sampled model running information required by the software optimizer when performing fitting.

[0051] In addition, there are multiple preset evaluators available. For example, you can use the Maestro Evaluator (Maestro) or the TimeLoop Evaluator (TimeLoop) to perform evaluation tasks.

[0052] Secondly, there are multiple methods for the target device to sample various initial model operation information. For example, the Latin hypercube method can be used for uniform sampling to ensure that various initial model operation information can be sampled uniformly. This further ensures that when the target device makes software operation information adjustments, it is more comprehensive and can obtain the optimal target model operation mode corresponding to the optimal target model operation information.

[0053] S106: Input the candidate model running information into a preset analysis model, so as to predict the task efficiency characterization value obtained when the target model's computation task is executed in the hardware running environment corresponding to the initial hardware parameter configuration sample according to the candidate model running mode corresponding to the candidate model running information. Based on the task efficiency characterization value corresponding to the candidate model running mode, determine the initial hardware parameter configuration sample for the next iteration under the given candidate model running mode. Send the initial hardware parameter configuration sample for the next iteration to the software optimizer, so that the software optimizer determines the candidate model running mode corresponding to the next iteration based on the initial hardware parameter configuration sample for the next iteration and the model parameters, until the preset iteration conditions are met, and obtain the target model running mode and the target hardware parameter configuration.

[0054] In this specification, the target device obtains the initial hardware parameter configuration sample for the next iteration under the given candidate model operation mode by inputting the candidate model operation mode determined by the software optimizer, the task efficiency characterization value corresponding to the candidate model operation mode, and the initial hardware parameter configuration sample into a preset hardware optimizer. Through cyclic iteration between the software optimizer and the hardware optimizer, the target hardware parameter configuration that meets the preset iteration conditions and the target model operation mode determined based on the target hardware parameter configuration are obtained.

[0055] Specifically, the hardware optimizer fits the task efficiency representation value corresponding to the candidate model's running mode and the initial hardware parameter configuration sample to obtain the second optimized model. Then, given the candidate model's running mode, the second optimized model is used to adjust the initial hardware parameter configuration sample to obtain each basic hardware parameter configuration sample. Each basic hardware parameter configuration sample is then input into a preset evaluator to predict the task efficiency representation value of the target model performing computation tasks in the hardware running environment corresponding to each basic hardware parameter configuration sample. Based on the task efficiency representation value corresponding to each basic hardware parameter configuration sample, the initial hardware parameter configuration sample for the next iteration is obtained.

[0056] It is important to note that the hardware optimizer mentioned here can fit the task efficiency representation value corresponding to the candidate model's running mode and the initial hardware parameter configuration sample in various ways. One method is to use Heterogeneous Variance Evolutionary Bayesian Optimization (HEBO) to fit this process.

[0057] S108: Configure the target hardware parameters on the target device to configure the target hardware operating environment, and deploy the target model on the target device based on the target hardware operating environment, so as to execute the computational tasks corresponding to the target model on the target device in accordance with the target model's operating mode.

[0058] In this specification, the initial hardware parameter configuration sample is adjusted with the goal of achieving a deviation between the task efficiency representation values ​​corresponding to the initial hardware parameter configuration samples of the two rounds after the preset number of rounds being less than a preset deviation. Based on the adjusted initial hardware parameter configuration sample, the model running mode is selected and adjusted to obtain the target model running mode and the corresponding target hardware parameter configuration sample.

[0059] After obtaining the target model's operating mode and target hardware parameter configuration through the above methods, the target device is configured with its target hardware operating environment according to the target hardware parameter configuration sample, and the target model is executed on the target hardware according to the target model's operating mode. Therefore, in the embodiments of this specification, the final target model's operating mode and target hardware parameter configuration can be considered as the optimal target hardware parameters and model deployment method obtained after hardware and software co-design, enabling the target device to execute the computational tasks corresponding to the target model more efficiently.

[0060] As can be seen from the above method, by selecting the model operation mode of the target model based on the initial hardware parameter configuration sample, and then adjusting the initial hardware parameter configuration sample used in the next iteration based on the selected candidate model operation mode, the hardware parameter configuration can be further optimized. Each round of adjustment is based on the previous round of adjustment. This not only ensures that the more rounds of adjustment, the better the effect of the final initial hardware parameter configuration sample, but also allows for the selection and adjustment of the target model's model operation mode. This achieves hardware and software co-design, ensuring a higher degree of matching between the target model and the target device during runtime, and greatly improving model running performance and accelerator hardware resource utilization.

[0061] Figure 2 This is a schematic diagram of a deep learning accelerator hardware and software co-processing system architecture provided for an embodiment of this specification.

[0062] like Figure 2 As shown, the process of co-configuring the target model and the target device's software and hardware includes mapping space search and design space search. Mapping space search selects and adjusts model operation information based on the initial hardware parameter configuration sample. Design space search, on the other hand, adjusts the initial hardware parameter configuration sample based on the candidate model operation modes corresponding to the candidate model operation information obtained after one round of selection and adjustment of model operation information in mapping space search. The adjusted initial hardware parameter configuration sample is then used as the initial hardware configuration input to the software optimizer of mapping space search. Based on the adjusted initial hardware parameter configuration sample, mapping space search adjusts the model operation mode. That is, the software optimizer determines the initial model operation information according to the input hardware configuration and inputs the determined initial model operation information and the input hardware configuration into the analysis model to obtain the task efficiency characterization value corresponding to each initial model operation information based on the determined hardware configuration. Thus, the software optimizer adjusts the model operation information based on the task efficiency characterization value corresponding to each basic candidate model operation information.

[0063] In this method, each round of mapping space search is based on the adjusted initial hardware parameter configuration sample to obtain the candidate model running mode corresponding to the candidate model running information. The design space search is based on the candidate model running mode to adjust the initial hardware parameter configuration sample, and each round of adjustment is based on the previous round of adjustment. After multiple iterations, the target hardware parameter configuration that meets the preset iteration conditions and the target model running mode determined based on the target hardware parameter configuration sample can be obtained. That is, the optimal deployment mode of the target model on the target device and the allocation of hardware resources corresponding to the optimal computational efficiency of the target model can be obtained.

[0064] By configuring the hardware runtime environment on the target device according to the target hardware parameters, the target model is deployed using the corresponding target model runtime mode, thereby realizing the target model's computational tasks and greatly improving the model's computational efficiency and accelerator hardware resource utilization.

[0065] The above is an embodiment of a deep learning accelerator hardware and software collaborative method provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0066] Figure 3 This is a schematic diagram of a hardware / software co-processing device for a deep learning accelerator, provided as an embodiment of this specification. The device includes: The acquisition module 301 is used to acquire the model parameters of the target model and the accelerator area constraint, wherein the accelerator area constraint is used to characterize the hardware configuration budget allocated by the accelerator to the target model. The first determining module 302 is used to determine an initial hardware parameter configuration sample based on the hardware parameter configuration contained in the accelerator area constraint. The second determining module 303 is used to input the hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters into a preset software optimizer, so that the software optimizer determines the running information of each initial model under the given hardware running environment corresponding to the initial hardware parameter configuration sample, and determines the running information of the candidate model based on the running information of each initial model. Analysis module 304 is used to input candidate model running information into a preset analysis model, so as to predict the task efficiency characterization value obtained when the target model's computation task is executed in the hardware running environment corresponding to the initial hardware parameter configuration method according to the candidate model running mode corresponding to the candidate model running information. Based on the task efficiency characterization value corresponding to the candidate model running mode, it determines the initial hardware parameter configuration sample for the next iteration under the given candidate model running mode, and sends the initial hardware parameter configuration sample for the next iteration to the software optimizer, so that the software optimizer determines the candidate model running mode corresponding to the next iteration based on the initial hardware parameter configuration sample for the next iteration and the model parameters, until the preset iteration conditions are met, and obtains the target model running mode and the target hardware parameter configuration. The running module 305 is used to configure the target hardware parameters on the target device to form a target hardware running environment, and based on the target hardware running environment, to deploy the target model on the target device to execute the computational tasks corresponding to the target model on the target device in accordance with the target model's running mode.

[0067] Optionally, the second determining module 303 is specifically configured to: sample from the initial model running information to obtain sampled model running information; input the sampled model running information and the initial hardware parameter configuration sample into a preset evaluator, so as to predict, through the evaluator, the task efficiency representation value obtained when the target model's computation task is executed according to the sampled model running information in the hardware running environment corresponding to the initial hardware parameter configuration sample, as the sampled representation value; input the sampled representation value, the sampled model running information, and the initial hardware parameter configuration sample into the software optimizer, so as to fit the sampled representation value and the sampled model running information through the software optimizer in the hardware running environment corresponding to the initial hardware parameter configuration sample to obtain a first optimized model; and determine candidate model running information based on the initial model running information using the first optimized model.

[0068] Optionally, the analysis module 304 is specifically used to input the task efficiency characterization value corresponding to the candidate model running mode, the candidate model running mode, and the initial hardware parameter configuration sample into a preset hardware optimizer, so as to determine the initial hardware parameter configuration sample for the next iteration under the given candidate model running mode through the hardware optimizer.

[0069] Optionally, the analysis module 304 is specifically used to, based on the candidate model running mode, fit the task efficiency characterization value corresponding to the candidate model running mode and the initial hardware parameter configuration sample through the hardware optimizer to obtain a second optimized model; Based on the hardware parameter options required when performing the task using the target model, the second optimization model determines the configuration samples of each basic hardware parameter under the given running mode of the candidate model. The basic hardware parameter configuration samples are input into a preset evaluator so that the evaluator can predict the task efficiency characterization value obtained when the target model is executed in the hardware running environment corresponding to each basic hardware parameter configuration sample. This value is used as the task efficiency characterization value corresponding to each basic hardware parameter configuration sample. Based on the task efficiency representation value corresponding to each basic hardware parameter configuration sample, the initial hardware parameter configuration sample for the next iteration is determined.

[0070] Optionally, the analysis module 304 is specifically used to: if the deviation between the task efficiency representation values ​​corresponding to the initial hardware parameter configuration samples of the two rounds before and after reaching the preset round is less than the preset deviation, then determine that the preset iteration condition is met, and take the initial hardware parameter configuration sample obtained under the preset iteration condition as the target hardware parameter configuration, and take the candidate model running mode determined by the software optimizer based on the target hardware parameter configuration as the target model running mode.

[0071] Optionally, the target model includes at least one model for performing computational tasks.

[0072] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the above-described actions. Figure 1 This paper presents a hardware and software co-processing method for deep learning accelerators.

[0073] based on Figure 1 The embodiment of this specification also provides a method for hardware and software collaboration in deep learning accelerators. Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 The aforementioned method for hardware and software collaboration in deep learning accelerators.

[0074] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0075] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0076] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0077] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0078] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0079] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0085] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0087] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0089] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0090] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A deep learning accelerator software and hardware collaboration method, characterized in that: include: Acquire model parameters of the target model and an accelerator area constraint, where the accelerator area constraint is used to characterize a hardware configuration budget allocated by the accelerator to the target model; Determining an initial hardware parameter configuration sample according to the hardware parameter configuration included in the accelerator area constraint; Inputting the hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters into a preset software optimizer, so that the software optimizer determines each initial model operation information under a given hardware operation environment corresponding to the initial hardware parameter configuration sample, and determines the candidate model operation information based on each initial model operation information; The candidate model operation information is input into a preset analysis model, so as to predict, through the analysis model, the task efficiency characterization value obtained when the operation task of the target model is executed in the hardware operation environment corresponding to the initial hardware parameter configuration sample according to the candidate model operation mode corresponding to the candidate model operation information; according to the task efficiency characterization value corresponding to the candidate model operation mode, the initial hardware parameter configuration sample of the next iteration is determined under the condition of the given candidate model operation mode; the initial hardware parameter configuration sample of the next iteration is sent to the software optimizer, so that the software optimizer determines the candidate model operation mode corresponding to the next iteration according to the initial hardware parameter configuration sample of the next iteration and the model parameters, until the preset iteration conditions are met, and the target model operation mode and the target hardware parameter configuration are obtained; A target hardware operating environment corresponding to the target hardware parameter configuration is configured on the target device, and based on the target hardware operating environment, the target model is deployed on the target device to execute a computing task corresponding to the target model on the target device according to the target model operating mode.

2. The method according to claim 1, characterized in that Based on the initial model operation information, determining the candidate model operation information specifically includes: Sampling from each of the initial model operation information to obtain sampled model operation information; Inputting the sampling model operation information and the initial hardware parameter configuration sample into a preset evaluator, so as to predict, through the evaluator, a task efficiency characterization value obtained when the computing task of the target model is executed in the hardware operation environment corresponding to the initial hardware parameter configuration sample according to the sampling model operation information, as a sampling characterization value; Inputting the sampled characterization value, the sampled model operation information, and the initial hardware parameter configuration sample into the software optimizer, so as to fit the sampled characterization value and the sampled model operation information through the software optimizer under the hardware operation environment corresponding to the initial hardware parameter configuration sample, and obtain a first optimization model; The candidate model operation information is determined based on the operation information of each initial model through the first optimization model.

3. The method according to claim 1, characterized in that According to the task efficiency characterization value corresponding to the candidate model operation mode, determining the initial hardware parameter configuration sample for the next iteration under the condition of the given candidate model operation mode, specifically including: The task efficiency characterization value corresponding to the candidate model operation mode, the candidate model operation mode and the initial hardware parameter configuration sample are input into a preset hardware optimizer, so as to determine the initial hardware parameter configuration sample for the next iteration under the condition of the given candidate model operation mode through the hardware optimizer.

4. The method according to claim 3, characterized in that Determining, by the hardware optimizer, an initial hardware parameter configuration sample for the next iteration under the condition of the given operation mode of the candidate model, specifically including: Based on the candidate model operation mode, the hardware optimizer is used to fit the task efficiency characterization value corresponding to the candidate model operation mode and the initial hardware parameter configuration sample to obtain a second optimization model; Based on the hardware parameter configurations required by the target model to perform the task, determine, through the second optimization model, various basic hardware parameter configuration samples under the condition of the given candidate model operation mode; Inputting each of the basic hardware parameter configuration samples into a preset evaluator, so as to predict, through the evaluator, a task efficiency characterization value obtained when executing the computing task of the target model under the hardware operating environment corresponding to each of the basic hardware parameter configuration samples, as the task efficiency characterization value corresponding to each of the basic hardware parameter configuration samples; According to the task efficiency characterization value corresponding to each basic hardware parameter configuration sample, the initial hardware parameter configuration sample for the next iteration is determined.

5. The method according to claim 1, characterized in that Until the preset iteration conditions are met, the target model operation mode and target hardware parameter configuration are obtained, including: If it is monitored that the deviation between the task efficiency characterization values ​​corresponding to the initial hardware parameter configuration samples corresponding to the two rounds before and after the preset round is less than the preset deviation, it is determined that the preset iteration condition is met, and the initial hardware parameter configuration sample obtained when the preset iteration condition is met is used as the target hardware parameter configuration, and the candidate model operation mode determined by the software optimizer based on the target hardware parameter configuration is used as the target model operation mode.

6. The method according to any one of claims 1 to 5, characterized in that The target model includes at least one model for performing a computing task.

7. A deep learning accelerator hardware and software collaboration device, characterized in that: include: An acquisition module, used for acquiring model parameters of a target model and an accelerator area constraint, wherein the accelerator area constraint is used for characterizing a hardware configuration budget allocated by the accelerator to the target model; A first determination module, configured to determine an initial hardware parameter configuration sample according to the hardware parameter configuration included in the accelerator area constraint; A second determination module is used to input the hardware parameters corresponding to the initial hardware parameter configuration sample and the model parameters into a preset software optimizer, so that the software optimizer determines each initial model operation information under a given hardware operation environment corresponding to the initial hardware parameter configuration sample, and determines the candidate model operation information based on the each initial model operation information; An analysis module is used to input the candidate model operation information into a preset analysis model, so as to predict, through the analysis model, a task efficiency characterization value obtained when the operation task of the target model is executed in the hardware operation environment corresponding to the initial hardware parameter configuration sample according to the candidate model operation mode corresponding to the candidate model operation information, and determine, based on the task efficiency characterization value corresponding to the candidate model operation mode, the initial hardware parameter configuration sample of the next iteration under the condition of the given candidate model operation mode, so as to send the initial hardware parameter configuration sample of the next iteration to the software optimizer, so that the software optimizer determines the candidate model operation mode corresponding to the next iteration according to the initial hardware parameter configuration sample of the next iteration and the model parameters, until the preset iteration conditions are met, and the target model operation mode and the target hardware parameter configuration are obtained; An operation module is used to configure a target hardware operating environment corresponding to the target hardware parameter configuration on a target device, and deploy the target model on the target device based on the target hardware operating environment, so as to execute the computing tasks corresponding to the target model on the target device according to the target model operation mode.

8. The device according to claim 7, characterized in that The second determination module is specifically used to sample from the initial model operation information to obtain sampled model operation information; input the sampled model operation information and the initial hardware parameter configuration sample into a preset evaluator, so as to predict, through the evaluator, the task efficiency characterization value obtained when the computing task of the target model is executed according to the sampled model operation information under the hardware operation environment corresponding to the initial hardware parameter configuration sample, as a sampled characterization value; input the sampled characterization value, the sampled model operation information and the initial hardware parameter configuration sample into the software optimizer, so as to fit the sampled characterization value and the sampled model operation information through the software optimizer under the hardware operation environment corresponding to the initial hardware parameter configuration sample to obtain a first optimization model; and determine the candidate model operation information based on the initial model operation information through the first optimization model.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images