Model deployment method and device

Abstract

The invention provides a model deployment method and device. The method comprises the following steps: acquiring a plurality of candidate deployment schemes; the candidate deployment scheme is a parallel strategy of a model obtained by pre-analyzing different calculation hardware under different calculation parameter combinations; the parallel strategy at least comprises a tensor parallel strategy and a data parallel strategy; the candidate deployment scheme is determined based on performance test results of a calculation structure unit of a model under different calculation parameter combinations, and the calculation structure unit is a standard reasoning calculation structure used for pre-analysis; based on a target calculation parameter combination of a to-be-deployed model, determining at least one first deployment scheme matched with the to-be-deployed model from a plurality of candidate deployment schemes, so as to deploy the to-be-deployed model based on the at least one first deployment scheme; the to-be-deployed model at least comprises calculation structure units adjusted based on the target calculation parameter combination.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a model deployment method and apparatus. Background Technology

[0002] Model deployment involves migrating trained models to different computing hardware to achieve efficient model inference. As model size increases and computational demands rise, it becomes necessary to achieve optimal model performance within limited hardware resources. Existing model deployment methods typically require manual tuning for specific hardware environments, making them difficult to adapt to diverse model deployment needs. Summary of the Invention

[0003] This application provides a model deployment method and apparatus.

[0004] This application provides a model deployment method, comprising: acquiring multiple candidate deployment schemes; the candidate deployment schemes are parallel strategies for models obtained through pre-analysis of different computing hardware under different combinations of computing parameters; the parallel strategies include at least tensor parallel strategies and data parallel strategies; the candidate deployment schemes are determined based on performance test results of the model's computational structural units under different combinations of computing parameters, the computational structural units being standard inference computational structures used for the pre-analysis; based on the target combination of computing parameters of the model to be deployed, determining at least one first deployment scheme that matches the model to be deployed from the multiple candidate deployment schemes, and deploying the model to be deployed based on the at least one first deployment scheme; the model to be deployed includes at least computational structural units adjusted based on the target combination of computing parameters.

[0005] According to one embodiment of this application, the method further includes: obtaining the video memory usage parameters of the model to be deployed; determining the video memory constraints of the model to be deployed based on the video memory usage parameters; determining a target deployment scheme from the at least one first deployment scheme based on the video memory usage parameters of the model to be deployed and the video memory constraints; and deploying the model to be deployed based on the target deployment scheme.

[0006] According to one embodiment of this application, the performance test results of the model-based computing structure unit under different combinations of computing parameters are determined by: constructing a model of a single-layer computing structure unit; configuring a parallel strategy for the computing structure unit based on a pre-set combination of computing parameters to obtain a configured computing structure unit; the combination of computing parameters includes at least: hidden layer size, intermediate layer size, and attention head size; performing performance tests on the configured computing structure unit to obtain performance test results corresponding to the combination of computing parameters; and determining the corresponding parallel strategy as the candidate deployment scheme in response to the performance test results meeting the set performance conditions.

[0007] According to one embodiment of this application, the step of performing performance testing on the configured computing structure unit to obtain the performance test results corresponding to the combination of computing parameters includes: obtaining the application scenario of the set model; different application scenarios have different requirements for the performance indicators of the model; performing performance testing on the configured computing structure unit based on the application scenario to obtain the performance test results corresponding to the combination of computing parameters in the application scenario.

[0008] According to one embodiment of this application, the method further includes: obtaining a configuration file of the model to be deployed; determining a target combination of computational parameters for the model to be deployed based on the configuration file; the target combination of computational parameters includes at least the hidden layer size, the intermediate layer size, and the attention head size.

[0009] According to one embodiment of this application, determining at least one first deployment scheme that matches the model to be deployed from multiple candidate deployment schemes based on the target combination of computational parameters of the model to be deployed includes: calculating the distance between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment schemes; and determining at least one first deployment scheme corresponding to the closest combination of computational parameters based on the shortest distance algorithm.

[0010] According to one embodiment of this application, determining a target deployment scheme from the at least one first deployment scheme includes: obtaining a target application scenario for the model to be deployed; determining a target performance index for the model to be deployed based on the target application scenario; the target performance index includes at least a throughput index and / or a latency index; selecting at least one second deployment scheme that satisfies the target performance index from the first deployment schemes based on the target performance index; and determining the second deployment scheme that satisfies the memory constraint as the target deployment scheme.

[0011] According to one embodiment of this application, determining the video memory constraints of the model to be deployed based on the video memory usage parameters includes: obtaining the device performance parameters of the deployment device of the model to be deployed; the device performance parameters include at least video memory reservation parameters and the total video memory of the device; calculating the video memory usage parameters corresponding to the model to be deployed based on the model parameters and the video memory reservation parameters; and determining the video memory constraints of the model to be deployed based on the video memory usage parameters and the total video memory of the device.

[0012] According to one embodiment of this application, the candidate deployment scheme is associated with the type of computing hardware, and the candidate deployment scheme corresponding to each type of computing hardware is determined based on the performance test results of the computing structure unit of the model deployed on the computing hardware under different combinations of computing parameters.

[0013] This application also provides a model deployment apparatus, comprising: an acquisition module for acquiring multiple candidate deployment schemes; the candidate deployment schemes are parallel strategies for models obtained through pre-analysis of different computing hardware under different combinations of computing parameters; the parallel strategies include at least tensor parallel strategies and data parallel strategies; the candidate deployment schemes are determined based on performance test results of the model's computational structural units under different combinations of computing parameters, the computational structural units being standard inference computational structures used for the pre-analysis; and a determination module for determining at least one first deployment scheme matching the model to be deployed from the multiple candidate deployment schemes based on the target combination of computing parameters of the model to be deployed, and deploying the model to be deployed based on the at least one first deployment scheme; the model to be deployed includes at least a computational structural unit adjusted based on the target combination of computing parameters. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0015] Figure 1 This paper illustrates the processing flow of the model deployment method provided in an embodiment of this application. Figure 1 ; Figure 2 This paper illustrates the processing flow of the model deployment method provided in an embodiment of this application. Figure 2 ; Figure 3 This paper illustrates the processing flow of the model deployment method provided in an embodiment of this application. Figure 3 ; Figure 4 This paper illustrates the processing flow of the model deployment method provided in an embodiment of this application. Figure 4 ; Figure 5 This illustration shows an application scenario diagram of the model deployment method provided in an embodiment of this application; Figure 6 This diagram illustrates another application scenario of the model deployment method provided in the embodiments of this application; Figure 7This illustration shows an optional schematic diagram of the model deployment apparatus provided in an embodiment of this application; Figure 8 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0016] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] The processing flow in the model deployment method provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the processing flow of the model deployment method provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S102 shown will be explained.

[0021] Step S101: Obtain multiple candidate deployment schemes; the candidate deployment schemes are parallel strategies of models obtained through pre-analysis for different computing hardware under different combinations of computing parameters; the parallel strategies include at least tensor parallel strategies and data parallel strategies; the candidate deployment schemes are determined based on the performance test results of the model's computing structure unit under different combinations of computing parameters, and the computing structure unit is a standard inference computing structure used for pre-analysis.

[0022] In some embodiments, candidate deployment schemes may include: multiple model parallelism strategies pre-analyzed for different combinations of computing hardware and computing parameters. Candidate deployment schemes can be used to provide selectable parallelism strategies when actually deploying models, adapting to different hardware environments and computing requirements. Computing hardware may include: physical devices for performing computing tasks, such as graphics processing units (GPUs), central processing units (CPUs), and field-programmable gate arrays (FPGAs). This application does not limit specific computing hardware parallelism strategies, which may include: tensor parallelism (TP) and data parallelism (DP). Parallelism strategy is one of the most important factors affecting the performance (throughput, latency) of large model inference. Tensor parallelism (TP) and data parallelism (DP) are two parallelism strategies used in large model inference scenarios. TP distributes model parameters across multiple GPUs, effectively utilizing multi-GPU computing power, but incurring significant inter-GPU communication overhead. DP copies the model across multiple GPUs, with low inter-GPU communication overhead but weaker single-request processing capability and high memory requirements. Given a fixed amount of GPU resources, the parameter selection for TP and DP depends on the model and the deployed hardware environment. The computational structural unit can include computational modules within the model, such as a single-layer structure in the Transformer architecture. This computational structural unit can serve as the standard inference computational structure during pre-analysis, and performance testing of the computational structural unit can determine candidate deployment schemes. The combination of computational parameters refers to the values ​​of multiple parameters during model deployment. This combination of computational parameters can include hidden layer size (hidden_size), intermediate layer size (intermediate_size), attention head size (attention_head_size), etc. The embodiments in this application do not limit the specific combination of computational parameters.

[0023] Step S102: Based on the target computational parameter combination of the model to be deployed, at least one first deployment scheme that matches the model to be deployed is determined from multiple candidate deployment schemes, so as to deploy the model to be deployed based on at least one first deployment scheme; the model to be deployed includes at least a computational structure unit adjusted based on the target computational parameter combination.

[0024] In some embodiments, the target computational parameter combination may include a specific combination of parameters such as the hidden layer size, intermediate layer size, and attention head size of the model to be deployed. The first deployment scheme may include one or more parallel strategies selected from candidate deployment schemes that match the computational parameter combination of the model to be deployed. The model to be deployed may include a model that needs to be deployed to achieve inference functionality. The model to be deployed may include a single-layer structure in the Transformer architecture adjusted based on the target computational parameter combination.

[0025] As an example, let's take the deployment of an inference model as an example. The inference model to be deployed is based on the Transformer architecture, and the target computational parameter combination is a hidden layer size of 5120, an intermediate layer size of 25600, and an attention head size of 5120. Candidate deployment schemes can be generated in advance for several common GPU models.

[0026] First, multiple candidate deployment schemes are obtained. These schemes are model parallelism strategies pre-analyzed for different GPUs under different combinations of computational parameters. They include various parallelism methods such as tensor parallelism and data parallelism. Each scheme is determined based on the performance test results of the model's computational structural unit (such as a single-layer Transformer structure) on the corresponding GPU.

[0027] Next, based on the target combination of computational parameters for the inference model to be deployed, at least one first deployment scheme that matches it is determined from these candidate deployment schemes. For example, by comparing the matching degree between the combination of computational parameters and each candidate scheme, several deployment schemes with the best performance under the same or similar parameter combinations are selected as the first deployment scheme.

[0028] Finally, a target deployment scheme that meets both performance requirements and memory constraints is selected from the first deployment scheme, and the inference model to be deployed is deployed according to the first deployment scheme, so that the inference model can run efficiently and stably on the GPU of the actual deployment device to complete the inference task.

[0029] The method in this application generates multiple candidate deployment schemes by analyzing different combinations of computing hardware and computing parameters in advance. Tensor parallelism and data parallelism strategies are considered to adapt to different hardware environments and model requirements. By using candidate deployment schemes determined based on performance test results, the optimal deployment scheme matching the model to be deployed can be quickly selected from multiple schemes during model deployment, achieving rapid and efficient model deployment without the need for manual model tuning, thus improving deployment efficiency and adapting to diverse model deployment needs.

[0030] In some embodiments, candidate deployment schemes are associated with types of computing hardware, and the candidate deployment schemes for each type of computing hardware are determined based on the performance test results of the computing structure units of the model deployed on the computing hardware under different combinations of computing parameters.

[0031] In some embodiments, the types of computing hardware may include different models of GPUs, CPUs, and FPGAs. Different computing hardware has different architectures and computing capabilities. Performance test results may include performance metrics such as throughput and latency obtained by testing the computational structural units of the model under a set combination of computing hardware and computing parameters.

[0032] As an example, in practical applications, if multiple instances of a Transformer-based natural language processing model need to be deployed, these instances can run on different types of computing hardware, including Type A, Type B, and Type C GPUs. For each type of computing hardware, candidate deployment schemes are generated separately. For Type A GPUs, performance tests are conducted on the model's computational structural units (such as a single-layer Transformer structure) under different combinations of computational parameters. The corresponding performance test results are obtained, and candidate deployment schemes including tensor parallelism and data parallelism strategies are generated based on these results. Similarly, for Type B and Type C GPUs, similar performance tests and candidate deployment scheme generation are performed based on their respective hardware characteristics. Finally, candidate deployment schemes corresponding to each type of GPU are obtained. When a model needs to be deployed, the most suitable scheme is selected from the pre-generated candidate deployment schemes based on the target combination of computational parameters for the model to be deployed and the type of computing hardware on the deployment device. For example, if a model needs to be deployed on a Type A GPU, the first deployment scheme can be directly determined from the candidate deployment schemes corresponding to Type A GPUs for model deployment.

[0033] The method in this application, by associating candidate deployment schemes with computing hardware types, can provide customized deployment schemes for different hardware environments. This improves the deployment adaptability and performance of the model in diverse hardware environments, and enhances the versatility and flexibility of model deployment.

[0034] In some embodiments, the processing flow of the model deployment method is illustrated. Figure 2 ,like Figure 2 As shown, the model deployment method may also include: Step S201: Obtain the video memory usage parameters of the model to be deployed.

[0035] Step S202: Determine the memory constraints of the model to be deployed based on the memory usage parameters.

[0036] In some embodiments, step S202 may include: obtaining the device performance parameters of the deployment device of the model to be deployed; the device performance parameters include at least the video memory reservation parameters and the total video memory of the device; calculating the video memory usage parameters corresponding to the model to be deployed based on the model parameters and video memory reservation parameters of the model to be deployed; and determining the video memory constraints of the model to be deployed based on the video memory usage parameters and the total video memory of the device.

[0037] In this embodiment, the video memory usage parameters may include: the amount of video memory occupied by the model during runtime. The video memory reservation parameters may include: KVCache and system video memory reservation values. The model parameters of the model to be deployed may include: the number of model parameters and the precision of the model parameters. The total device video memory may include: the total video memory capacity included in the deployment device.

[0038] As an example, obtain the memory reservation parameters and total memory of the deployment device for the model to be deployed. The memory reservation parameters include KVCache memory reservation and system memory reservation values, and the total memory includes the GPU card's memory size. Based on the model parameters and memory reservation parameters of the model to be deployed, calculate the model memory size in the memory usage parameters corresponding to the model to be deployed, which can be expressed by the following formula: Model memory size = number of model parameters Model parameter accuracy / n (1 + 0.2 ~ 0.5) The values ​​from 0.2 to 0.5 represent the reserved memory for KVCache and system video memory, and n represents the number of computing hardware devices, specifically the number of GPU cards. In heavy-load and long-sequence scenarios, the reserved video memory value can be appropriately increased.

[0039] Based on the memory usage parameters and the total memory of the device, the memory constraints of the model to be deployed are determined. Specifically, the memory constraints of TP in the parallel strategy can be calculated by the following formula, where ceil() is for rounding up: TP ≥ ceil (model memory size / GPU memory size) Taking the deployment of a deep learning model as an example, its calculated GPU memory size is 32GB. Now, the deep learning model is deployed on two GPU cards, each with 16GB of GPU memory. According to the formula TP≥ceil(model GPU memory size / GPU card GPU memory size), that is, TP≥ceil(32GB / 16GB)=ceil(2)=2. Therefore, the GPU memory constraint requires TP to be greater than or equal to 2, indicating that tensor parallel partitioning using two GPU cards can satisfy the model deployment requirement.

[0040] Step S203: Based on the memory usage parameters and memory constraints of the model to be deployed, determine the target deployment scheme from at least one first deployment scheme.

[0041] In some embodiments, determining the target deployment scheme from at least one first deployment scheme in step S203 may include: obtaining the target application scenario of the model to be deployed; determining the target performance indicators of the model to be deployed based on the target application scenario; the target performance indicators include at least throughput indicators and / or latency indicators; selecting at least one second deployment scheme that meets the target performance indicators from the first deployment schemes based on the target performance indicators; and determining the second deployment scheme that meets the memory constraints as the target deployment scheme.

[0042] In this embodiment, the target application scenario may include: the specific scenario in which the model is actually applied. Specifically, it may be in natural language processing, image recognition processing, etc. This application embodiment does not limit the specific target application scenario. The target performance metric may include: the performance standard that the model needs to achieve in the target application scenario. The target performance metric may specifically be the throughput metric (the number of requests processed per unit time) and / or the latency metric (the time required to process one request). The second deployment scheme may include: one or more deployment schemes that meet the target performance metric selected from the first deployment scheme.

[0043] As an example, taking the deployment of a deep learning model as an example, the target computational parameter combination of the deep learning model to be deployed corresponds to multiple first deployment schemes. The target application scenario of the deep learning model is real-time object detection in image recognition. The target application scenario requires the model to have high throughput. Based on the target application scenario, the target performance indicator of the model to be deployed is mainly the throughput indicator. From the first deployment schemes, second deployment schemes that meet the conditions are selected based on the target performance indicator. Taking the existence of 3 first deployment schemes as an example: Option A: TP=1, throughput=80 images / second Option B: TP=2, throughput=120 images / second Option C: TP=4, throughput=150 images / second Determine the target deployment scheme that satisfies the memory constraint (TP≥2). Scheme B has TP=2, satisfying the memory constraint; Scheme C has TP=4, also satisfying the memory constraint. Therefore, both Scheme B and Scheme C can be used as the target deployment scheme. Alternatively, Scheme C, which has the highest throughput, can be selected as the target deployment scheme.

[0044] Step S204: Deploy the model to be deployed based on the target deployment plan.

[0045] In this embodiment, the video memory usage parameters may include parameters related to the video memory capacity occupied by the model during runtime, such as the number of model parameters, the KVCache video memory reservation ratio, and the system video memory reservation value. This application embodiment does not limit specific video memory usage parameters. Video memory constraints can be used to limit the video memory capacity available during model deployment, ensuring that the model can run normally on the hardware device without exceeding the video memory capacity. The target deployment scheme may include: the optimal deployment scheme that meets both performance requirements and video memory constraints. The target deployment scheme can guide the actual deployment of the model on the computing hardware of the deployment device.

[0046] As an example, the memory usage parameters of the model to be deployed are obtained, including the number of model parameters, the KVCache memory reservation ratio, and the system memory reservation value. These parameters allow for the calculation of the memory capacity required by the model during runtime. Then, based on the memory capacity required by the model during runtime, combined with the total memory of the deployment device and other memory usage limitations, the memory constraints of the model to be deployed are determined, i.e., the maximum memory capacity that the model can use during deployment. Further screening is performed from at least one determined first deployment scheme to identify a deployment scheme that meets both performance requirements and memory constraints. Specifically, one or more optimal deployment schemes can be ultimately determined as the target deployment scheme from multiple first deployment schemes. Based on the determined target deployment scheme, the model to be deployed is deployed on the actual deployment device.

[0047] The method in this application, by analyzing memory usage parameters and determining memory constraints, ensures efficient model deployment under given memory resources. This avoids deployment failures due to insufficient memory and improves the stability of model deployment. Simultaneously, by acquiring the target application scenario and determining target performance indicators based on that scenario, this method can select a second deployment scheme that meets the performance requirements from the first deployment scheme, and determine the final deployment scheme by combining the memory constraints. This ensures optimal model performance in specific scenarios and enables efficient model deployment in various application scenarios.

[0048] In some embodiments, the processing flow of the model deployment method is illustrated. Figure 3 ,like Figure 3 As shown, the performance test results of the model-based computational structure unit under different combinations of computational parameters in step S101 are determined, which may specifically include: Step S301: Construct a model of a single-layer computational structural unit.

[0049] In some embodiments, a single-layer computational architecture unit is constructed. This single-layer computational architecture unit can be a layer in a Transformer architecture, containing a multi-head self-attention mechanism and a feedforward neural network. The single-layer computational architecture unit model can also be a model structure containing only one computational layer.

[0050] Step S302: Configure the parallel strategy of the computing structure unit based on the pre-set combination of computing parameters to obtain the configured computing structure unit.

[0051] Step S303: Perform performance testing on the configured computational structure unit to obtain the performance test results corresponding to the combination of computational parameters.

[0052] In some embodiments, step S303 may include: obtaining the application scenario of the set model; different application scenarios have different requirements for the performance indicators of the model; performing performance testing on the configured computing structure unit based on the application scenario, and obtaining the performance test results corresponding to the combination of computing parameters in the application scenario.

[0053] As an example, first determine the application scenario of the model. Specifically, the application scenario can be a real-time interactive scenario, which has high requirements for model latency, and the corresponding performance metric requirement is minimum latency; the application scenario can also be a large-scale data processing scenario, which has high requirements for model throughput, and the corresponding performance metric requirement is maximum throughput.

[0054] In performance testing for real-time interactive scenarios, the performance of the configured computational structure unit under different combinations of TP and DP is tested, and the performance test results corresponding to the combinations of computational parameters are obtained. From the performance test results, the performance test result corresponding to the minimum latency is determined.

[0055] In performance testing for large-scale data processing scenarios, the performance of the configured computing structure units under different combinations of TP and DP is tested, and the performance test results corresponding to the combinations of computing parameters are obtained. From the performance test results, the performance test result corresponding to the maximum throughput is determined.

[0056] Step S304: In response to the performance test results meeting the set performance conditions, the corresponding parallel strategy is determined as a candidate deployment scheme.

[0057] In this embodiment, performance testing may include: testing the configured computing architecture unit to determine its performance metrics, such as throughput and latency, under set combinations of computing parameters and parallel strategies. Performance conditions may include: pre-set performance standards, such as minimum throughput requirements or maximum latency limits.

[0058] As an example, in the pre-tuning phase of model deployment, candidate deployment schemes are determined for different combinations of computational parameters. First, a single-layer computational structure unit model is constructed, for example, this model includes only one layer of the Transformer structure. This single-layer computational structure unit can include a multi-head self-attention mechanism and a feedforward neural network. This computational structure unit is configured based on pre-defined combinations of computational parameters. Specifically, the parameters of the projected linear layer can be configured according to the hidden layer size, intermediate layer size, and attention head size. The parameters of the projected linear layer are related to the performance of TP splitting. Model weights do not need to be loaded during testing, and the initial values ​​can be random numbers. The performance of the configured computational structure unit under different TP and DP combinations is tested to obtain the performance test results corresponding to the computational parameter combinations. The performance test results can include throughput and / or latency. The set performance conditions can be a throughput of at least 1000 requests per second and a latency of no more than 200 milliseconds. If the performance test results corresponding to the configured computational structure unit meet the set performance conditions, the corresponding parallel strategy is determined as a candidate deployment scheme. The candidate deployment scheme can be selected and used when actually deploying the model.

[0059] The method in this application, by constructing a single-layer computational structure unit model and conducting performance tests, can accurately evaluate the performance under different combinations of computational parameters, thus improving the accuracy of deployment schemes. By acquiring the application scenario of the model and conducting performance tests based on that scenario, it ensures that the selected deployment scheme can meet the performance requirements of specific application scenarios, thereby improving the model's performance in practical applications. Customizing deployment schemes for different application needs enhances the model's adaptability and flexibility.

[0060] In some embodiments, the model deployment method may further include: obtaining a configuration file of the model to be deployed; determining a target combination of computational parameters for the model to be deployed based on the configuration file; the target combination of computational parameters includes at least the hidden layer size, the intermediate layer size, and the attention head size.

[0061] As an example, obtain the configuration file (config.json) of the model to be deployed. This configuration file contains the structure and related parameters of the model, such as the number of layers and the number of neurons in each layer. Based on the information in the configuration file, read or calculate the three parameters of the model to be deployed: hidden layer size, intermediate layer size, and attention head size.

[0062] In some embodiments, the processing flow of the model deployment method is illustrated. Figure 4 ,like Figure 4 As shown, step S102, based on the target calculation parameter combination of the model to be deployed, determines at least one first deployment scheme that matches the model to be deployed from multiple candidate deployment schemes, including: Step S401: Calculate the distance between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment scheme.

[0063] Step S402: Based on the shortest distance algorithm, determine at least one first deployment scheme corresponding to the closest combination of calculation parameters.

[0064] In this embodiment, the distance can include the degree of difference between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment scheme. Specifically, the distance can be Euclidean distance, Manhattan distance, or L-Inf distance. This embodiment does not limit the specific distance, where the L-Inf distance is defined as the maximum value of the absolute difference between the target combination of computational parameters and the combination of computational parameters. The distance can usually be calculated using mathematical formulas, and different mathematical formulas correspond to different distances. The distance can be used to measure the similarity between different combinations of computational parameters; the shorter the distance, the more similar the two are. The shortest distance algorithm can include an algorithm for determining the minimum distance between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment scheme. Specifically, the shortest distance algorithm can be used to determine the deployment scheme that best matches the target combination of computational parameters from multiple candidate deployment schemes.

[0065] As an example, during model deployment, the target computation parameter combination corresponding to the Transformer model to be deployed may include hidden_size=1024, intermediate_size=4096, and attention_head_size=64.

[0066] Based on the target computational parameters, the shortest distance algorithm is used to match the candidate deployment scheme with the closest parameters. The candidate deployment scheme is a set of computational parameter combinations and their corresponding optimal parallel strategies generated during the pre-tuning phase. Each candidate deployment scheme includes the values ​​of the hidden_size, intermediate_size, and attention_head_size parameters, as well as the corresponding performance test results and optimal parallel strategy.

[0067] Candidate deployment schemes may include: Scenario A: hidden_size=1000, intermediate_size=4000, attention_head_size=60; Scenario B: hidden_size=1024, intermediate_size=4096, attention_head_size=64; Scenario C: hidden_size=1050, intermediate_size=4150, attention_head_size=65; Using L-Inf distance as a distance metric, the distance between the target combination of computational parameters and the combination of computational parameters corresponding to each candidate deployment scheme is calculated.

[0068] For scenario A, the hidden_size difference = |1024-1000| = 24, the intermediate_size difference = |4096-4000| = 96, and the attention_head_size difference = |64-60| = 4. Therefore, the L-Inf distance is max(24,96,4) = 96.

[0069] For scenario B, the hidden_size difference = |1024-1024| = 0, the intermediate_size difference = |4096-4096| = 0, and the attention_head_size difference = |64-64| = 0. Therefore, the L-Inf distance is max(0,0,0) = 0.

[0070] For scenario C, the hidden_size difference = |1024-1050| = 26, the intermediate_size difference = |4096-4150| = 54, and the attention_head_size difference = |64-65| = 1. Therefore, the L-Inf distance is max(26,54,1) = 54.

[0071] Based on the L-Inf distance, the distance to scenario B is 0, so the deployment scheme corresponding to scenario B is determined as the first deployment scheme.

[0072] The method in this application calculates the distance between the target combination of computational parameters and candidate deployment schemes, and determines the first deployment scheme based on the shortest distance algorithm. This enables the rapid and accurate selection of the optimal scheme from multiple candidate schemes, improving deployment efficiency and reducing deployment time and resource consumption. Simultaneously, it enhances the overall performance of the model.

[0073] refer to Figure 5 The application scenario diagram of the model deployment method provided in this application embodiment is applied to implement TP splitting in the Transformer model.

[0074] The diagram illustrates a Transformer-based model deployment method, including a tensor-parallel splitting strategy. Each Transformer model comprises input embedding, positional encoding, multiple Transformer Block layers, and an output layer. Each Transformer Block layer further includes a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections. The dashed lines in the diagram indicate the locations of the tensor-parallel splitting (TP). The performance differences under different TP splitting strategies are primarily determined by the dimensions (hidden_size, intermediate_size, and attention_head_size) of the down-projected Linear layer. The attention_head_size is calculated as: attention_head_size = (num_attention_heads + 2) num_key_value_heads) `head_dim`. `num_attention_heads` is the number of attention heads, `num_key_value_heads` is the number of key-value heads, and `head_dim` is the width of the matrix output by each attention head.

[0075] refer to Figure 6 Another application scenario diagram of the model deployment method provided in this application embodiment is applied to automated tensor parallel optimization based on pre-profiling (analysis).

[0076] The pre-profiling phase is used to perform a performance test for each GPU card to determine the optimal parallel strategy.

[0077] Construct a single-layer Transformers model.

[0078] Modify Linear layer parameters: The Linear layer parameters of the Transformers model include hidden_size, intermediate_size, and attention_head_size.

[0079] Test the performance of different parallel strategies: Perform performance tests on the modified model based on different parallel strategies (such as tensor parallelism TP and data parallelism DP), and record performance indicators to obtain the performance test results corresponding to each parallel strategy, including Throughput, TTFT (Time To First Token) and TPOT (Time Per Output Token).

[0080] Generate profiling result file: Save the performance test results and corresponding parallel strategies to the Profile.yaml file.

[0081] The model deployment phase is used to determine the final parallel strategy for the model to be deployed based on the results of the pre-analysis.

[0082] Read the model configuration file (config.json): Obtain the configuration information of the model to be deployed, such as model type, number of GPUs, scene, etc.

[0083] Calculate key parameters of the Linear layer: Based on the model configuration, calculate the key parameters of the Linear layer of the model to be deployed.

[0084] Parameter shortest distance matching: Compare the calculated parameters with those obtained in the pre-analysis stage to find the closest parameter combination.

[0085] Determine the candidate parallel strategy (optimal computing power): Based on the shortest distance matching result, select the parallel strategy with the optimal computing power (such as TP8DP1>TP4DP2>TP2DP4).

[0086] Computational model memory usage: The memory usage of the computational model during deployment, based on model configuration and GPU information.

[0087] Determine the candidate parallel strategy (memory constraint): Determine the memory constraint conditions (such as TP>= 2) based on the memory usage of the model during deployment, and ensure that the selected parallel strategy meets the memory constraint conditions.

[0088] Comprehensive decision: Taking into account both computing power and memory constraints, the final parallel strategy (TP8DP1) is determined.

[0089] The following continues to describe an exemplary structure of the model deployment device 90 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 7 As shown, the model deployment device 90 may include: an acquisition module 901, used to acquire multiple candidate deployment schemes; the candidate deployment schemes are parallel strategies of models obtained through pre-analysis for different computing hardware under different combinations of computing parameters; the parallel strategies include at least tensor parallel strategies and data parallel strategies; the candidate deployment schemes are determined based on the performance test results of the model's computational structural units under different combinations of computing parameters, and the computational structural units are standard inference computational structures used for pre-analysis; and a determination module 902, used to determine at least one first deployment scheme that matches the model to be deployed from multiple candidate deployment schemes based on the target combination of computing parameters of the model to be deployed, so as to deploy the model to be deployed based on at least one first deployment scheme; the model to be deployed includes at least a computational structural unit adjusted based on the target combination of computing parameters.

[0090] In some embodiments, the model deployment apparatus 90 may further include a deployment module, which is configured to: obtain the video memory usage parameters of the model to be deployed; determine the video memory constraints of the model to be deployed based on the video memory usage parameters; determine a target deployment scheme from at least one first deployment scheme based on the video memory usage parameters and the video memory constraints of the model to be deployed; and deploy the model to be deployed based on the target deployment scheme.

[0091] In some embodiments, the model deployment apparatus 90 may further include a testing module, which is used to: construct a model of a single-layer computational structure unit; configure a parallel strategy for the computational structure unit based on a pre-defined combination of computational parameters to obtain the configured computational structure unit; the combination of computational parameters includes at least: hidden layer size, intermediate layer size, and attention head size; perform performance testing on the configured computational structure unit to obtain the performance test results corresponding to the combination of computational parameters; and, in response to the performance test results meeting the set performance conditions, determine the corresponding parallel strategy as a candidate deployment scheme.

[0092] In some embodiments, the testing module is used to: obtain the application scenario of the set model; different application scenarios have different requirements for the performance indicators of the model; perform performance testing on the configured computing structure unit based on the application scenario, and obtain the performance test results corresponding to the combination of computing parameters in the application scenario.

[0093] In some embodiments, the acquisition module 901 is further configured to: acquire the configuration file of the model to be deployed; and determine the target computational parameter combination of the model to be deployed based on the configuration file; the target computational parameter combination includes at least the hidden layer size, the intermediate layer size, and the attention head size.

[0094] In some embodiments, the determining module 902 is configured to: calculate the distance between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment scheme; and determine at least one first deployment scheme corresponding to the closest combination of computational parameters based on the shortest distance algorithm.

[0095] In some embodiments, the deployment module is configured to: obtain the target application scenario of the model to be deployed; determine the target performance indicators of the model to be deployed based on the target application scenario; the target performance indicators include at least throughput indicators and / or latency indicators; select at least one second deployment scheme that meets the target performance indicators from the first deployment scheme based on the target performance indicators; and determine the second deployment scheme that meets the memory constraints as the target deployment scheme.

[0096] In some embodiments, the deployment module is used to: obtain the device performance parameters of the deployment device of the model to be deployed; the device performance parameters include at least the video memory reservation parameters and the total video memory of the device; calculate the video memory usage parameters corresponding to the model to be deployed based on the model parameters and video memory reservation parameters of the model to be deployed; and determine the video memory constraints of the model to be deployed based on the video memory usage parameters and the total video memory of the device.

[0097] In some embodiments, candidate deployment schemes are associated with types of computing hardware, and the candidate deployment schemes for each type of computing hardware are determined based on the performance test results of the computing structure units of the model deployed on the computing hardware under different combinations of computing parameters.

[0098] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For any technical details not covered in the model deployment apparatus provided in this application embodiment, please refer to... Figures 1 to 7 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0099] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0100] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0101] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0102] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the model deployment method. For example, in some embodiments, the model deployment method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the model deployment method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the model deployment method by any other suitable means (e.g., by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable model deployment device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0108] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0109] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model deployment method, comprising: Multiple candidate deployment schemes are obtained; the candidate deployment schemes are parallel strategies for models obtained through pre-analysis of different computing hardware under different combinations of computing parameters. The parallel strategy includes at least tensor parallel strategy and data parallel strategy; the candidate deployment scheme is determined based on the performance test results of the model's computational structure unit under different combinations of computational parameters, and the computational structure unit is a standard inference computational structure used for the pre-analysis. Based on the target calculation parameter combination of the model to be deployed, at least one first deployment scheme that matches the model to be deployed is determined from multiple candidate deployment schemes, so as to deploy the model to be deployed based on the at least one first deployment scheme; The model to be deployed includes at least a computational structure unit adjusted based on the target computational parameter combination.

2. The method according to claim 1, further comprising: Obtain the video memory usage parameters of the model to be deployed; Based on the aforementioned video memory usage parameters, determine the video memory constraints of the model to be deployed. Based on the memory usage parameters of the model to be deployed and the memory constraints, a target deployment scheme is determined from the at least one first deployment scheme; Deploy the model to be deployed based on the target deployment scheme.

3. The method according to claim 1, wherein the performance test results of the model-based computational structural unit under different combinations of computational parameters are determined, including: Construct a model of a single-layer computational structural unit; The parallel strategy of the computing structure unit is configured based on the pre-set combination of computing parameters to obtain the configured computing structure unit. The combination of computational parameters includes at least: hidden layer size, intermediate layer size, and attention head size; The performance of the configured computing structure unit is tested to obtain the performance test results corresponding to the combination of computing parameters; In response to the performance test results meeting the set performance conditions, the corresponding parallel strategy is determined as the candidate deployment scheme.

4. The method according to claim 3, wherein performing performance testing on the configured computational structure unit to obtain performance test results corresponding to the combination of computational parameters includes: Obtain the application scenarios for the defined model; Different application scenarios have different requirements for the performance metrics of the model; Based on the application scenario, the performance of the configured computing structure unit is tested to obtain the performance test results corresponding to the combination of computing parameters in the application scenario.

5. The method according to claim 1, further comprising: Obtain the configuration file of the model to be deployed; Based on the configuration file, the target combination of computational parameters for the model to be deployed is determined; The target computational parameter combination includes at least the hidden layer size, the intermediate layer size, and the attention head size.

6. The method according to claim 1, wherein determining at least one first deployment scheme matching the model to be deployed from multiple candidate deployment schemes based on the target calculation parameter combination of the model to be deployed comprises: Calculate the distance between the target combination of computational parameters and the combination of computational parameters corresponding to the candidate deployment scheme; Based on the shortest distance algorithm, at least one first deployment scheme is determined corresponding to the closest combination of computational parameters.

7. The method according to claim 2, wherein determining the target deployment scheme from the at least one first deployment scheme comprises: Obtain the target application scenario for the model to be deployed; Based on the target application scenario, determine the target performance metrics of the model to be deployed; The target performance metrics include at least throughput metrics and / or latency metrics; Based on the target performance metric, at least one second deployment scheme that meets the target performance metric is selected from the first deployment scheme; The second deployment scheme that satisfies the aforementioned memory constraints is determined as the target deployment scheme.

8. The method according to claim 2, wherein determining the video memory constraints of the model to be deployed based on the video memory usage parameters includes: Obtain the device performance parameters of the deployment device for the model to be deployed; The device performance parameters include at least the video memory reservation parameters and the total video memory of the device; Based on the model parameters and memory reservation parameters of the model to be deployed, the memory usage parameters corresponding to the model to be deployed are calculated. Based on the memory usage parameters and the total memory of the device, the memory constraints of the model to be deployed are determined.

9. The method according to claim 1, wherein the candidate deployment scheme is associated with the type of computing hardware, and the candidate deployment scheme corresponding to each type of computing hardware is determined based on the performance test results of the computing structure unit of the model deployed on the computing hardware under different combinations of computing parameters.

10. A model deployment apparatus, comprising: The acquisition module is used to acquire multiple candidate deployment schemes. The candidate deployment schemes are parallel strategies of models obtained through pre-analysis for different computing hardware under different combinations of computing parameters. The parallel strategies include at least tensor parallel strategies and data parallel strategies. The candidate deployment schemes are determined based on the performance test results of the model's computational structure unit under different combinations of computing parameters. The computational structure unit is a standard inference computational structure used for the pre-analysis. The determination module is used to determine at least one first deployment scheme that matches the model to be deployed from multiple candidate deployment schemes based on the target computational parameter combination of the model to be deployed, so as to deploy the model to be deployed based on the at least one first deployment scheme; the model to be deployed includes at least a computational structure unit adjusted based on the target computational parameter combination.

Images