Model bottleneck identification methods, devices, electronic equipment, storage media, and programs

Abstract

This disclosure provides a method, apparatus, electronic device, storage medium, and program for determining model bottlenecks. One specific embodiment of the method includes: in response to receiving a task request, acquiring multiple operating parameters of the target model and the target hardware, under the condition that the target hardware carries the target model, wherein the target model is used to respond to the task request; acquiring multiple actual resource requirements requested by the task request, each actual resource requirement representing a requirement on a corresponding operating parameter; determining the degree of constraint of each operating parameter on the corresponding actual resource requirement; identifying the operating parameters whose constraint degree is greater than a corresponding preset constraint threshold as bottleneck parameters, and determining the location in the target hardware and / or target model pointed to by the bottleneck parameter as the bottleneck location. This embodiment can effectively and accurately estimate the bottleneck of the model.

Description

Technical Field

[0001] The embodiments of this application relate to the technical field of artificial intelligence, and in particular to a method, apparatus, electronic device, storage medium, and program for determining model bottlenecks. Background Technology

[0002] As the demand for artificial intelligence technology increases, in the process of applying and running artificial intelligence entities, such as artificial intelligence models, there are often situations where the corresponding tasks cannot be completed in a timely manner due to the operational bottlenecks of the artificial intelligence models.

[0003] The bottleneck in the operation of an artificial intelligence model may be due to insufficient or excessive precision of the model's architecture or configuration parameters, or it may be due to the hardware parameters of the hardware used to deploy the artificial intelligence model. The end result is that the artificial intelligence model has insufficient performance during operation.

[0004] However, in practice, it is often difficult to accurately locate the operational bottleneck of an artificial intelligence model. That is, it is impossible to determine whether the operational bottleneck is caused by model architecture parameters, configuration parameters or hardware parameters, and therefore it is also impossible to accurately determine the parameters corresponding to the operational bottleneck and their location in the corresponding hardware or artificial intelligence model. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a method, apparatus, electronic device, storage medium, and program for determining model bottlenecks.

[0006] In a first aspect, embodiments of this disclosure provide a method for determining model bottlenecks, the method comprising:

[0007] In response to receiving a task request, under the condition that the target model is deployed on the target hardware, multiple operating parameters of the target model and the target hardware are obtained, wherein the target model is used to respond to the task request;

[0008] Obtain the multiple actual resource requirements of the task request, where each actual resource requirement represents the requirement for the corresponding operating parameters;

[0009] Determine the degree of constraint that each operating parameter imposes on the corresponding actual resource requirements;

[0010] The operating parameters whose constraint level is greater than the corresponding preset constraint threshold are identified as bottleneck parameters, and the location in the target hardware and / or the target model pointed to by the bottleneck parameter is identified as the bottleneck location.

[0011] In some optional implementations, multiple operational parameters of the target model and the target hardware are obtained, including:

[0012] Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request;

[0013] Obtain multiple actual hardware parameters of the target hardware;

[0014] The various architecture parameters, configuration parameters, and actual hardware parameters are structured and then used as the operating parameters.

[0015] In some optional implementations, several actual hardware parameters of the target hardware are obtained, including:

[0016] Obtain multiple standard hardware parameters of the target hardware;

[0017] The corresponding actual hardware parameters are determined based on the preset compensation factors corresponding to each standard hardware parameter. Each compensation factor represents the degree of difference between the corresponding standard hardware parameter and the corresponding actual hardware parameter.

[0018] In some optional implementations, the actual resource requirements include the actual computing power requirements; the process of the target model responding to the task request includes a pre-filling stage and a decoding stage; the compensation factor corresponding to each standard hardware parameter includes a first efficiency compensation factor in the pre-filling stage and a second efficiency compensation factor in the decoding stage;

[0019] Accordingly, the actual resource requirements of the task request are obtained, including:

[0020] Determine the computational requirements of the task request in the pre-filling stage and the decoding stage respectively;

[0021] The computational requirements of the pre-filling stage are corrected according to the first efficiency compensation factor to obtain the first equivalent computational quantity, and the computational requirements of the decoding stage are corrected according to the second efficiency compensation factor to obtain the second equivalent computational quantity.

[0022] The first equivalent computational load and the second equivalent computational load are determined as the actual computing power requirement.

[0023] In some alternative implementations, actual resource requirements include actual memory requirements;

[0024] Accordingly, the actual resource requirements of the task request are obtained, including:

[0025] Obtain the static memory requirement and cache memory requirement of the target model, wherein the static memory requirement and the cache memory requirement have the same memory value in the pre-filling stage and the decoding stage;

[0026] Determine the peak memory usage of the task requests in the pre-filling and decoding phases, respectively.

[0027] The larger of the stage memory peak values ​​from the pre-filling stage and the decoding stage, the sum of the static memory requirement and the cache memory requirement, is determined as the actual memory requirement.

[0028] In some alternative implementations, actual resource requirements include actual bandwidth requirements;

[0029] Accordingly, the actual resource requirements of the task request are obtained, including:

[0030] Determine the bandwidth requirements of the task request in the pre-filling stage and the decoding stage respectively, and determine the sum of the bandwidth requirements of the pre-filling stage and the decoding stage as the actual bandwidth requirement;

[0031] The bandwidth requirement of the pre-filling stage is the ratio between the total data transmission volume of the pre-filling stage and the latency budget of the pre-filling stage, and the bandwidth requirement of the decoding stage is the product of the data transmission volume per word and the number of words per second.

[0032] In some alternative implementations, the degree of constraint that each operating parameter imposes on the corresponding actual resource requirements is determined, including:

[0033] Determine the parameter difference between each operating parameter and the corresponding actual resource requirement in the pre-filling stage and the decoding stage, respectively;

[0034] The ratio of the parameter difference between the pre-filling stage and the decoding stage to the corresponding operating parameters is determined as the corresponding constraint level.

[0035] Secondly, embodiments of this disclosure provide a model bottleneck determination apparatus, the apparatus comprising:

[0036] The parameter acquisition module is configured to, in response to receiving a task request, acquire multiple operating parameters of the target model and the target hardware, under the condition that the target hardware carries the target model, wherein the target model is used to respond to the task request;

[0037] The requirement acquisition module is configured to acquire multiple actual resource requirements of the task request, each actual resource requirement representing a requirement for the corresponding operating parameters.

[0038] The constraint degree calculation module is configured to determine the degree of constraint of each running parameter on the corresponding actual resource requirements;

[0039] The bottleneck determination module is configured to identify the operating parameters whose constraint degree is greater than the corresponding preset constraint threshold as bottleneck parameters, and to identify the location in the target hardware and / or the target model pointed to by the bottleneck parameter as the bottleneck location.

[0040] In some optional implementations, the parameter acquisition module is further configured to:

[0041] Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request;

[0042] Obtain multiple actual hardware parameters of the target hardware;

[0043] The various architecture parameters, configuration parameters, and actual hardware parameters are structured and then used as the operating parameters.

[0044] Among them, obtaining multiple actual hardware parameters of the target hardware includes:

[0045] Obtain multiple standard hardware parameters of the target hardware;

[0046] The corresponding actual hardware parameters are determined based on the preset compensation factors corresponding to each standard hardware parameter. Each compensation factor represents the degree of difference between the corresponding standard hardware parameter and the corresponding actual hardware parameter.

[0047] In some optional implementations, the actual resource requirements include the actual computing power requirements; the process by which the target model responds to the task request includes a pre-filling stage and a decoding stage; the compensation factor corresponding to each standard hardware parameter includes a first efficiency compensation factor in the pre-filling stage and a second efficiency compensation factor in the decoding stage.

[0048] Accordingly, the requirement elicitation module is further configured as follows:

[0049] Determine the computational requirements of the task request in the pre-filling stage and the decoding stage respectively;

[0050] The computational requirements of the pre-filling stage are corrected according to the first efficiency compensation factor to obtain the first equivalent computational quantity, and the computational requirements of the decoding stage are corrected according to the second efficiency compensation factor to obtain the second equivalent computational quantity.

[0051] The first equivalent computational load and the second equivalent computational load are determined as the actual computing power requirement.

[0052] In some alternative implementations, actual resource requirements include actual memory requirements.

[0053] Accordingly, the requirement elicitation module is further configured as follows:

[0054] Obtain the static memory requirement and cache memory requirement of the target model, wherein the static memory requirement and the cache memory requirement have the same memory value in the pre-filling stage and the decoding stage;

[0055] Determine the peak memory usage of the task requests in the pre-filling and decoding phases, respectively.

[0056] The larger of the stage memory peak values ​​from the pre-filling stage and the decoding stage, the sum of the static memory requirement and the cache memory requirement, is determined as the actual memory requirement.

[0057] In some alternative implementations, actual resource requirements include actual bandwidth requirements.

[0058] Accordingly, the requirement elicitation module is further configured as follows:

[0059] Determine the bandwidth requirements of the task request in the pre-filling stage and the decoding stage respectively, and determine the sum of the bandwidth requirements of the pre-filling stage and the decoding stage as the actual bandwidth requirement;

[0060] The bandwidth requirement of the pre-filling stage is the ratio between the total data transmission volume of the pre-filling stage and the latency budget of the pre-filling stage, and the bandwidth requirement of the decoding stage is the product of the data transmission volume per word and the number of words per second.

[0061] In some optional implementations, the constraint degree calculation module is further configured to:

[0062] Determine the parameter difference between each operating parameter and the corresponding actual resource requirement in the pre-filling stage and the decoding stage, respectively;

[0063] The ratio of the parameter difference between the pre-filling stage and the decoding stage to the corresponding operating parameters is determined as the corresponding constraint level.

[0064] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0065] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method as described in any implementation of the first aspect.

[0066] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the implementations of the first aspect.

[0067] To address the difficulty in identifying operational bottlenecks in artificial intelligence models, embodiments of this disclosure provide a method, apparatus, electronic device, storage medium, and program for determining model bottlenecks. Upon receiving a task request, for the target model to respond to the task request and the target hardware used to deploy the target model, multiple relevant operational parameters can be obtained. Each operational parameter reflects one or more dimensions of the target model or target hardware's performance in responding to the task request. By determining whether each operational parameter is sufficient to support the response to the task request, it can be determined whether the operational parameter is a bottleneck in the target model's operation. By obtaining multiple actual resource requirements required by the task request, it can be determined whether each operational parameter meets the corresponding actual resource requirements, i.e., whether the operational parameter forms a constraint bottleneck on the corresponding actual resource requirements. This allows for the determination of whether each operational parameter is a bottleneck parameter in the target model's operation. Furthermore, once the bottleneck parameter is determined, the location of the bottleneck parameter in the target model or target hardware can be identified as the bottleneck location, achieving the effect of accurately locating the operational bottleneck. Attached Figure Description

[0068] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0069] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure may be applied;

[0070] Figure 2 A flowchart illustrating one embodiment of the model bottleneck determination method according to this disclosure;

[0071] Figure 3 This is an exploded flowchart of one embodiment of step 201 according to the present disclosure;

[0072] Figure 4A This is an exploded flowchart of one embodiment of step 202 according to the present disclosure;

[0073] Figure 4B This is an exploded flowchart of another embodiment of step 202 according to the present disclosure;

[0074] Figure 5This is an exploded flowchart of one embodiment of step 203 according to the present disclosure;

[0075] Figure 6 A schematic diagram of the structure of one embodiment of the model bottleneck determination apparatus according to the present disclosure;

[0076] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0077] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0078] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0079] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the model bottleneck determination methods, apparatus, electronic devices, storage media, and programs of this disclosure can be applied.

[0080] like Figure 1 As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0081] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as natural language processing applications, large model applications, speech recognition applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0082] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be either hardware or software. When the first terminal device 101, the second terminal device 102, and the third terminal device 103 are hardware, they can be various electronic devices with sound acquisition devices (e.g., microphones), video acquisition devices (e.g., cameras), and displays, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When the first terminal device 101, the second terminal device 102, and the third terminal device 103 are software, they can be installed on the terminal devices listed above. They can be implemented as multiple software programs or software modules (e.g., to provide task management related services) or as a single software program or software module. No specific limitations are made here.

[0083] In some cases, the model bottleneck determination method provided in this disclosure can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the model bottleneck determination device can be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103. In this case, the system architecture 100 may not include the server 105.

[0084] In some cases, the model bottleneck determination method provided in this disclosure can be jointly executed by the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. For example, the step of "in response to receiving a task request, under the condition of deploying the target model on the target hardware, obtaining multiple operating parameters of the target model and the target hardware" can be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103, while the step of "obtaining multiple actual resource requirements requested by the task request" can be executed by the server 105. This disclosure does not limit this. Correspondingly, the model bottleneck determination device can also be respectively set in the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105.

[0085] In some cases, the model bottleneck determination method provided in this disclosure can be executed by server 105. Accordingly, the model bottleneck determination device can also be set in server 105. In this case, system architecture 100 may not include first terminal device 101, second terminal device 102, and third terminal device 103.

[0086] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0087] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0088] It should be noted that the model bottleneck determination method disclosed herein is used to predict the operational bottleneck of an artificial intelligence model. The artificial intelligence model, as the target model, can be, for example, a neural network model or a machine learning model, specifically, a model such as an LLM (Large Language Model).

[0089] The target model can be deployed in Figure 1 The first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105 can be deployed on the same device or server as this method, or they can be deployed on different devices or servers. In other cases, the target model can also be deployed on... Figure 1 Other devices besides the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105 shown; wherein, the target model and the device deploying the target model can interact with the device carrying the method, so that the device running the method can obtain relevant data of the target model and relevant data of the device deploying the target model, and then use the relevant data to implement the method.

[0090] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of the model bottleneck determination method of this disclosure, which includes the following steps 201 to 204:

[0091] Step 201: In response to receiving the task request, under the condition that the target model is deployed on the target hardware, obtain multiple operating parameters of the target model and the target hardware.

[0092] The target model is used to respond to the task request; the target hardware is the device that deploys the target model, and the target hardware provides supporting resources for the operation of the target model. The supporting resources provided by the target hardware for the target model may be, for example, computing power resources, memory resources, and bandwidth resources.

[0093] The target model can respond to the task requests it receives based on the supporting resources provided by the target hardware; that is, it can complete the task requests by calling various supporting resources to run.

[0094] In this step, when the target model receives a task request sent to it, before the target model runs and responds to the task request, the device running this method can obtain multiple operating parameters of the target model and multiple operating parameters of the target hardware.

[0095] The acquired operational parameters of the target model are all parameters that affect its performance when responding to task requests. For example, the operational parameters of the target model can be architectural parameters related to the target model's architecture, and the operational parameters of the target hardware can be hardware parameters related to the computing power, memory resources, and bandwidth resources that the target hardware can provide.

[0096] Step 202: Obtain the multiple actual resource requirements of the task request.

[0097] Each actual resource requirement represents the requirement for the corresponding operating parameters.

[0098] Based on the task request received by the target model, the device running this method can obtain multiple actual resource requirements of the task request before the target model runs and responds to the task request.

[0099] Each actual resource requirement represents the resource overhead required to complete the task request, and indicates the minimum requirements of the task request for each operating parameter. In other words, each actual resource requirement points to the requirement for one or more operating parameters.

[0100] Specifically, corresponding to the various operating parameters in step 201 above, the actual resource requirements may include, for example, the actual computing power requirements for computing power resources, the actual bandwidth requirements for bandwidth resources, and the actual memory requirements for memory resources.

[0101] Therefore, the actual resource requirement represents the resource requirement that the task demands on the target model during actual operation. This actual resource requirement differs from the resource requirement inherent in the task itself. Specifically, in practice, the supporting resources that the target hardware can actually provide when running the target model are often less than its pre-defined standard resources. For example, the target hardware may be rated to provide a bandwidth of 100MB / s, but when actually running the target model, it may only provide 80MB / s, which is 80% of the standard bandwidth. Therefore, if the task itself requires a bandwidth of 50MB / s, the target hardware must provide at least 62.5MB / s of standard bandwidth to provide at least 50MB / s of actual bandwidth when running the target model. In other words, in this case, the actual bandwidth requirement requested by the task is 62.5MB / s.

[0102] Step 203: Determine the degree of constraint that each operating parameter imposes on the corresponding actual resource requirements.

[0103] Based on the various operating parameters and corresponding actual resource requirements determined in steps 201 and 202 above, it can be determined whether the operating parameter can meet the corresponding actual resource requirements and the degree of constraint of the operating parameter on the corresponding actual resource requirements based on the value of each operating parameter and the value of the corresponding actual resource requirements.

[0104] Specifically, when the operating parameters meet the corresponding actual resource requirements, the remaining resource size can be regarded as reflecting the degree of constraint of the operating parameters on the actual resource requirements; or, when the operating parameters do not meet the corresponding actual resource requirements, the size of the resource gap can also be regarded as reflecting the degree of constraint of the operating parameters on the actual resource requirements.

[0105] Therefore, this degree of constraint indicates the extent to which the operating parameter, i.e., the supporting resources provided, constrains and limits the target model in responding to task requests. When the operating parameter cannot meet or just meets the corresponding actual resource requirements, then the operating parameter becomes a bottleneck that constrains and limits the task requests.

[0106] Therefore, the degree of constraint of each operating parameter on the actual resource demand can be determined based on the value of each operating parameter and the actual resource demand. Even if all operating parameters meet the actual resource demand, different degrees of constraint can be obtained due to the different amounts of resources remaining for each operating parameter.

[0107] In some alternative implementations, since different operating parameters have different metrics, the ratio between the remaining resource size or the resource gap size and the corresponding actual resource demand can also be regarded as the degree of constraint of the operating parameter on the actual resource demand.

[0108] Step 204: Determine the operating parameters whose constraint degree is greater than the corresponding preset constraint threshold as bottleneck parameters, and determine the location in the target hardware and / or the target model pointed to by the bottleneck parameters as the bottleneck location.

[0109] Based on the constraint degree of each operating parameter determined in step 203 above, corresponding constraint thresholds can be set for each operating parameter in advance, and each constraint degree can be compared with the corresponding constraint threshold.

[0110] Based on the comparison results, the operating parameters with a constraint degree greater than the corresponding constraint threshold can be identified, and these operating parameters can be regarded as bottleneck parameters. In other words, the bottleneck parameters can be regarded as the bottleneck that restricts the target model from responding to task requests smoothly and in a timely manner.

[0111] Based on this, the location in the target hardware or target model that the bottleneck parameter specifically points to can be taken as the bottleneck location.

[0112] Specifically, since each operating parameter points to a corresponding resource, when that operating parameter is a bottleneck parameter, the location of that resource in the target hardware, or the location in the target model where that resource is used, is the bottleneck location.

[0113] In other words, for a target model running on the target hardware, there may be a single bottleneck parameter and its corresponding bottleneck location, or there may be multiple bottleneck parameters and their respective bottleneck locations.

[0114] In some alternative implementations, the bottleneck parameter can also be the operating parameter with the greatest constraint among all operating parameters. That is, in this implementation, for the target model running on the target hardware, the operating parameter that most restricts the performance of the target model is taken as the bottleneck parameter, and the location in the target hardware that provides the corresponding resource, or the location in the target model that uses the corresponding resource, is taken as the bottleneck location.

[0115] As can be seen, upon receiving a task request, for the target model that will respond to the task request and the target hardware used to deploy the target model, multiple relevant operational parameters can be obtained first. Each operational parameter reflects one or more dimensions of the performance of the target model or target hardware in responding to the task request. Thus, by judging whether each operational parameter is sufficient to support the response to the task request, it can be determined whether the operational parameter is a bottleneck in the operation of the target model. By obtaining multiple actual resource requirements required by the task request, it can be judged whether each operational parameter meets the corresponding actual resource requirements, that is, whether the operational parameter forms a constraint bottleneck on the corresponding actual resource requirements. This allows it to be judged whether each operational parameter is a bottleneck parameter in the operation of the target model. Furthermore, once the bottleneck parameter is determined, the location of the bottleneck parameter in the target model or target hardware can be identified as the bottleneck location, achieving the effect of accurately locating the operational bottleneck.

[0116] Based on the received task request, in the process of obtaining multiple operating parameters of the target model and the target hardware, step 201 of this disclosure can obtain multiple architecture parameters and configuration parameters of the target model, and obtain the actual hardware parameters of the target hardware.

[0117] Specifically, please refer to Figure 3 This illustrates a decomposition process 300 of one embodiment of step 201 of this disclosure. The decomposition process 300 includes the following steps 301 to 303:

[0118] Step 301: Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request.

[0119] Among them, the architecture parameters can be the various parameters that the target model itself possesses, and the configuration parameters can be the various parameters that the target model needs to configure to execute task requests.

[0120] In the process of obtaining multiple architectural parameters and multiple configuration parameters of the target model, taking LLM as an example, the architectural parameters that can be obtained include, but are not limited to, the following:

[0121] Architecture parameters that affect the computational load of the target model include: context length, maximum sequence length, input prompt length, current processed context length, number of tokens generated in a single iteration, total number of tokens generated, hidden layer dimension, feedforward network intermediate layer dimension, number of query attention heads, number of key and / or value attention heads, number of model layers, vocabulary size, attention dimension, rotation dimension ratio of rotation position encoding, and implementation of the normalized exponential function.

[0122] The process of LLM executing a task request can include at least a pre-filling stage and a decoding stage. Therefore, the input prompt length mentioned above is the architecture parameter of the pre-filling stage, and the currently processed context length, the number of tokens generated in a single instance, and the total number of tokens generated are the architecture parameters of the decoding stage.

[0123] Step 302: Obtain multiple actual hardware parameters of the target hardware.

[0124] Among them, each actual hardware parameter represents the support resources that the target hardware for deploying the target model can actually provide in each corresponding dimension.

[0125] Specifically, in the process of obtaining multiple actual hardware parameters of the target model, taking LLM as an example, the actual hardware parameters that can be obtained include, but are not limited to, the following:

[0126] Target quantization type, such as FP16, BF16, and INT8; batch size; input / output sequence length; target throughput; peak computing power under different precision conditions; total memory or video memory capacity and bandwidth; capacity and bandwidth of each cache level; and interconnect bandwidth, etc.

[0127] In some alternative implementations, as mentioned above, since the actual support resources provided by the target hardware are often less than the standard resources it is calibrated, in the process of obtaining multiple actual hardware parameters of the target model, the standard hardware parameters calibrated by the target hardware can be obtained first, and then the actual hardware parameters corresponding to each standard hardware parameter can be calculated through a pre-set compensation factor.

[0128] Among them, the compensation factor quantitatively characterizes the degree of difference between the standard hardware parameters and the corresponding actual hardware parameters, as well as the effective utilization rate of the standard hardware parameters.

[0129] In determining the compensation factor between each standard hardware parameter and its corresponding actual hardware parameter, the standard hardware parameters of the target hardware can be tested using a general matrix multiplication standard micro-benchmark test to obtain the corresponding actual hardware parameters. The compensation factor can then be determined based on the ratio between the actual hardware parameters and the corresponding standard hardware parameters.

[0130] Specifically, the test matrix size can be determined based on the size of a typical LLM layer, for example, 4096×4096, 8192×4096 and / or 11008×4096, and the test precision can be set, for example, FP16, INT8 and / or INT4.

[0131] Furthermore, taking the standard hardware parameter TOPS (Tera Operations Per Second) as an example, under the above test conditions, the actual test results of TOPS can be recorded and used as the corresponding actual hardware parameter. Accordingly, the ratio between the actual test results of TOPS and the standard value of TOPS, i.e. the theoretical value, can be used as the compensation factor for this standard hardware parameter.

[0132] For other standard hardware parameters, such as on-chip storage performance, it can be tested by repeatedly accessing small data blocks (i.e., data blocks less than or equal to 32KB); for L2 cache performance, it can be tested by accessing medium datasets (i.e., datasets greater than or equal to 10MB and 1GB); and for global memory performance, it can be tested by accessing large datasets (i.e., datasets greater than or equal to 1GB).

[0133] Based on this, baseline calibration of various standard hardware parameters can be completed, and compensation factors for various standard hardware parameters can be obtained.

[0134] In the process of performing the above-mentioned matrix multiplication standard micro-benchmark test, we can also test actual hardware parameters such as the hardware average actual speedup ratio of the target quantization bits in the pre-filling stage relative to the reference floating-point precision, and the hardware average actual speedup ratio of the target quantization bits in the decoding stage relative to the reference floating-point precision.

[0135] Based on the obtained compensation factors and corresponding standard hardware parameters, the product of the compensation factor and the corresponding standard hardware parameter can be used to determine the corresponding actual hardware parameter.

[0136] In some alternative implementations, when determining the compensation factor, taking LLM as an example, since the process of LLM executing a task request may include at least a pre-filling stage and a decoding stage, and the pre-filling stage and the decoding stage have different performance on each actual hardware parameter, it is possible to calculate the compensation factor for each actual hardware parameter in the pre-filling stage and the decoding stage respectively.

[0137] Step 303: Perform structured processing on the various architecture parameters, configuration parameters, and actual hardware parameters, and determine the structured architecture parameters, configuration parameters, and actual hardware parameters as various operating parameters.

[0138] Based on the various architecture parameters, configuration parameters, and actual hardware parameters obtained in the preceding steps, these parameters can be converted into structured parameters, making the structured data format of the architecture parameters suitable for reasoning in subsequent steps.

[0139] Furthermore, the structured architectural parameters, configuration parameters, and actual hardware parameters are used as the runtime parameters when the target model runs on the target hardware in response to task requests.

[0140] Based on this, by collecting the architecture and configuration parameters of the target model and combining them with the actual hardware parameters of the target hardware, the actual resources that the target hardware can provide and the relevant parameters of the target model are accurately quantified. By structuring these parameters, the structured parameters can be directly adapted to the estimation of resource requirements at different stages.

[0141] In some alternative implementations, actual resource requirements may include actual computing power requirements; accordingly, further reference should be made to... Figure 4A This illustrates a decomposition process 410 of one embodiment of step 202 of this disclosure. The decomposition process 410 includes the following steps 411 to 413:

[0142] Step 411: Determine the computational requirements of the task request in the pre-filling stage and the decoding stage respectively.

[0143] As mentioned earlier, the process of the target model executing the task request can include at least a pre-filling stage and a decoding stage. The computational requirements of the pre-filling stage and the decoding stage are different. Therefore, the computational requirements of the pre-filling stage and the decoding stage can be determined separately.

[0144] Specifically, for the pre-filling stage, the computational requirements may include, but are not limited to, the following operating parameters:

[0145] The computational cost of Q-projection, K-projection, and V-projection (Query projection, Key projection, Value projection) for each layer of the target model; the computational cost of attention scores for each layer of the target model; the computational cost of context vector generation; the computational cost of output projection for each layer of the target model; the computational cost and / or computational cost of feedforward layers for each layer of the target model; the computational cost and / or computational cost of embedding layers; and the computational cost and / or computational cost of the head output layer of the target model, etc.

[0146] The computational complexity of Q-projection in the pre-filling stage can be calculated using the following formula:

[0147]

[0148] The computational complexity of K-projection and V-projection during the pre-filling stage can be calculated using the following formula:

[0149]

[0150] Where Q_prefill represents the computational cost of Q-projection in the prefilling stage, KV_prefill represents the computational cost of K-projection and V-projection in the prefilling stage, L_input_prefill represents the input cue length in the prefilling stage, d_model represents the hidden layer dimension, d_head represents the attention head dimension, and n_head_kv represents the number of attention heads for K-projection and V-projection.

[0151] Where d_head= .

[0152] n_head_q represents the number of Q-projection attention heads.

[0153] The computational complexity of the attention score during the pre-filling stage can be calculated using the formula shown below:

[0154]

[0155] Here, Flops_Attention_prefill represents the computational cost of the attention score during the prefilling phase.

[0156] The computational cost of generating the context vector during the pre-filling stage can also be calculated using the formula shown below:

[0157]

[0158] Here, Context_prefill represents the computational cost of generating the context vector during the prefilling phase.

[0159] The amount of computation for the output projection of each network layer during the pre-filling stage can also be calculated using the formula shown below:

[0160]

[0161] Flops_O_proj_prefill represents the amount of operation for the output projection of each layer of the network during the prefilling stage.

[0162] The number of operations and / or computations for each layer of the feedforward network in the pre-filling stage can be calculated using the following formula:

[0163]

[0164] Where Flops_FFN_prefill represents the amount of operation and / or computation of each layer of the feedforward network in the prefilling stage, and d_ff represents the dimension of the intermediate layer of the feedforward network.

[0165] The number of operations and / or computations per operation in the output layer of the target model head during the pre-filling stage can be calculated using the following formula:

[0166]

[0167] The above formula is used for cases where the logical values ​​of all tokens in the pre-filling stage are required;

[0168]

[0169] The above formula is used for cases where the logical value of the last word in the pre-filling stage is required;

[0170] Where Flops_LM_head_prefill represents the number of operations and / or computations per operation in the target model head output layer during the prefilling stage, B represents the batch size for batch processing, and vocab represents the vocabulary size.

[0171] Furthermore, the sum of the above-mentioned operations and / or computations can be used as the computational requirement for the pre-filling stage. This computational requirement can characterize the theoretical operations or computations required by the target model to generate one lexical unit in the pre-filling stage.

[0172] Specifically, it can be calculated using the formula shown below:

[0173]

[0174] Here, Total_Flops_prefill represents the computational requirements of the prefill stage, n_layer represents the number of network layers as n, and per_layer represents the number of network layers as a single layer.

[0175] In some alternative implementations, the ratio between the sum of the above-mentioned operations and / or computations and the average actual speedup of the hardware relative to the reference floating-point precision of the target quantization bits in the pre-filling stage can be determined as the quantization equivalent computing power requirement of the pre-filling stage.

[0176] The equivalent computing power requirement can be calculated using the following formula:

[0177]

[0178] Here, Y_prefill represents the quantization equivalent computing power requirement in the prefill stage, and Quant_acceleration_ratio_prefill represents the average actual hardware speedup ratio of the target quantization bits relative to the baseline floating-point precision in the prefill stage.

[0179] Furthermore, for the decoding stage, the computational requirements may include, but are not limited to, the following operating parameters:

[0180] The computational costs of Q-projection, K-projection, and V-projection (Query projection, Key projection, Value projection) for each layer of the target model; the computational costs of attention scores for each layer of the target model; the computational costs of context vector generation; the computational costs of output projection for each layer of the target model; the computational costs and / or operations of each feedforward layer of the target model; the computational costs and / or operations of the embedding layers; and the computational costs and / or operations of the head output layers of the target model, etc.

[0181] The computational complexity of Q-projection in the decoding stage can be calculated using the following formula:

[0182]

[0183] The computational complexity of K-projection and V-projection during the decoding phase can be calculated using the following formula:

[0184]

[0185] Where Q_decode represents the computational cost of Q-projection in the decoding stage, KV_decode represents the computational cost of K-projection and V-projection in the decoding stage, and L_gen_decode represents the number of tokens generated in a single operation.

[0186] The computational complexity of the attention score during the decoding phase can be calculated using the following formula:

[0187] Where Flops_Attention_decode represents the computational cost of the attention score during the decoding phase, and L_prompt_decode represents the length of the current processed context.

[0188] The computational complexity of generating the context vector during the decoding phase can also be calculated using the formula shown below:

[0189]

[0190] Here, Context_decode represents the computational cost of generating the context vector during the decoding phase.

[0191] The amount of computation involved in the output projection of each network layer during the decoding phase can also be calculated using the formula shown below:

[0192]

[0193] Flops_O_proj_decode represents the amount of computation involved in the output projection of each network layer during the decoding phase.

[0194] The number of operations and / or computations required for each layer of the feedforward network in the decoding phase can be calculated using the following formula:

[0195]

[0196] Wherein, Flops_FFN_decode represents the amount of operation and / or computation of each layer of the feedforward network in the decoding stage.

[0197] The number of operations and / or computations per operation in the output layer of the target model head during the decoding phase can be calculated using the following formula:

[0198]

[0199] Flops_LM_head_decode represents the amount of operation and / or computation in a single operation of the target model head output layer during the decoding phase.

[0200] Furthermore, the sum of the above-mentioned operations and / or computations can be used as the computational requirement for the decoding stage. This computational requirement can characterize the theoretical operations or computations required by the target model to generate one lexical unit during the decoding stage.

[0201] Specifically, it can be calculated using the formula shown below:

[0202]

[0203] Among them, Total_Flops_decode represents the computational requirements of the decoding stage.

[0204] In some alternative implementations, the ratio between the sum of the above-mentioned operations and / or computations and actual hardware parameters such as the target quantization bit depth of the decoding stage relative to the reference floating-point precision can be determined as the quantization equivalent computing power requirement of the decoding stage.

[0205] The equivalent computing power requirement can be calculated using the following formula:

[0206]

[0207] Here, Y_decode represents the quantization equivalent computing power requirement in the decoding stage, and Quant_acceleration_ratio_decode represents the hardware average actual speedup ratio of the target quantization bits relative to the baseline floating-point precision in the decoding stage.

[0208] Step 412: Correct the computational requirements of the pre-filling stage according to the first efficiency compensation factor to obtain the first equivalent computational requirements, and correct the computational requirements of the decoding stage according to the second efficiency compensation factor to obtain the second equivalent computational requirements.

[0209] As mentioned earlier, the process of the target model executing the task request can include at least a pre-filling stage and a decoding stage. The determined compensation factors can be divided into a first efficiency compensation factor in the pre-filling stage and a second efficiency compensation factor in the decoding stage.

[0210] Based on the quantization equivalent computing power requirements for the pre-filling stage and the decoding stage determined in step 411 above, since in practice, the actual support resources that the target hardware can provide when running the target model are often less than the standard resources it has pre-calibrated, the quantization equivalent computing power requirements for the pre-filling stage can be corrected using the first efficiency compensation factor to obtain the first equivalent computational load for the pre-filling stage, and the quantization equivalent computing power requirements for the decoding stage can be corrected using the second efficiency compensation factor to obtain the second equivalent computational load for the decoding stage.

[0211] Specifically, the ratio between the quantization equivalent computing power requirement in the pre-filling stage and the first efficiency compensation factor can be determined as the first equivalent computing power, and the ratio between the quantization equivalent computing power requirement in the decoding stage and the second efficiency compensation factor can be determined as the second equivalent computing power.

[0212] Step 413: Determine the first equivalent computational load and the second equivalent computational load as the actual computing power requirement.

[0213] Based on the first and second equivalent computational quantities determined above, they can be used as the actual computing power requirements of the target model for the task request.

[0214] Based on this, by distinguishing the computational requirements of the pre-filling stage and the decoding stage, and by combining compensation factors for correction, the actual computing power requirements of the task request at different stages can be accurately determined. By using the first efficiency compensation factor and the second efficiency compensation factor for correction, the problem of inaccurate computing power estimation caused by the hardware parameter estimation deviation of the target hardware is effectively solved, and the reliability of the actual computing power requirement estimation is significantly improved.

[0215] In some alternative implementations, actual resource requirements may also include actual memory requirements and actual bandwidth requirements; accordingly, further reference... Figure 4B This illustrates a decomposition process 420 of another embodiment of step 202 of this disclosure. The decomposition process 420 includes the following steps 421 to 423:

[0216] Step 421: Obtain the static memory requirements and cache memory requirements of the target model.

[0217] In determining actual memory requirements, since the process of the target model executing task requests can include a pre-filling stage and a decoding stage, various memory resources can be divided into memory resources with the same memory value in both the pre-filling and decoding stages, and stage memory with different memory values ​​in both stages. This allows for estimation of memory resources with different memory values ​​in the pre-filling and decoding stages according to different stages, achieving a more granular estimation of actual memory requirements and obtaining more accurate results when determining actual memory requirements.

[0218] Therefore, since the static memory requirements and cache memory requirements of the target model do not change during the pre-filling or decoding stages, the static memory requirements and cache memory requirements of the target model during the execution of the task request can be obtained first.

[0219] Specifically, static memory requirements can be, for example, the memory usage of quantized architecture parameters, and the static memory requirements can be determined by multiplying the total number of architecture parameters of the target model by the byte size of a single architecture parameter after quantization.

[0220] Furthermore, for cache memory requirements, twice the product of batch size, context length, number of model layers, attention header dimension, number of key-value headers, and number of bytes of quantization type can be used as the cache memory requirement.

[0221] Step 422: Determine the peak memory values ​​of the task request in the pre-filling stage and the decoding stage respectively, and determine the larger peak memory value of the pre-filling stage and the decoding stage, the sum of the static memory requirement and the cache memory requirement as the actual memory requirement.

[0222] In this step, the maximum value of the stage memory in the pre-filling stage can be determined and used as the stage memory peak value of the pre-filling stage; at the same time, the maximum value of the stage memory in the decoding stage can be determined and used as the stage memory peak value of the decoding stage.

[0223] The peak stage memory in the pre-filling stage is determined by the extended activation or large intermediate matrix inside the attention layer of the feedforward network layer, such as the Score matrix. It should be noted that the peak stage memory in the decoding stage is often smaller than that in the pre-filling stage.

[0224] Based on this, the total memory required for static memory usage, cache memory requirements, and the maximum peak memory usage at any given time can be calculated, and this total memory usage can be used as the actual memory requirement.

[0225] Step 423: Determine the bandwidth requirements of the task request in the pre-filling stage and the decoding stage respectively, and determine the sum of the bandwidth requirements of the pre-filling stage and the decoding stage as the actual bandwidth requirement.

[0226] The bandwidth requirement for the pre-filling stage is the ratio of the total data transmission volume of the pre-filling stage to the latency budget of the pre-filling stage, while the bandwidth requirement for the decoding stage is the product of the data transmission volume per word and the number of words per second.

[0227] In this step, since the amount of data transmitted during the pre-filling and decoding stages of the target model during the execution of the task request is different, the amount of data transmitted during the pre-filling and decoding stages can be determined separately, thereby further determining their respective bandwidth requirements, and the sum of the two bandwidth requirements can be determined as the actual bandwidth requirement.

[0228] Specifically, for the pre-filling stage, the amount of data transferred can include, but is not limited to, the following operating parameters:

[0229] The amount of data transfer required to write keys and values ​​to the cache, the amount of data transfer required to read keys and values ​​from the cache, the amount of data transfer required to read the architecture parameters, configuration parameters and / or other parameters of the target model, and the amount of data transfer required to read and write activation values.

[0230] The amount of data transfer required to write keys and values ​​to the cache during the pre-filling stage can be calculated using the following formula:

[0231] Here, KV_Cache(write_prefill) represents the amount of data transfer required to write keys and values ​​to the cache during the prefill phase, and sizeof represents the size of a single architecture parameter in bytes after quantization.

[0232] The bandwidth required to read keys and values ​​from the cache can be approximated as 0.

[0233] The amount of data transfer required to read the architectural parameters of the target model can be approximately equal to the total number of parameters in the target model, or approximately equal to the number of parameters in the active computing layer.

[0234] To determine the amount of data transfer required to read and write activation values, if the activation values ​​cannot reside entirely in the on-chip cache, the amount of data transfer required to read and write from main memory can be considered. For example, consider the amount of data transfer required for input / output activation of a feedforward network layer.

[0235] The amount of data transmission required for the activation of the input and output of the feedforward network layer can be calculated using the following formula:

[0236]

[0237]

[0238] Here, BW_Activation_rw_prefill(FFN) represents the amount of data transmission required to activate the input and output of the feedforward network layer.

[0239] Based on this, the total data transmission volume can be determined by summing the above-mentioned operating parameters regarding the data transmission volume required in the pre-filling phase, and the ratio between the total data transmission volume and the expected latency budget can be determined as the bandwidth requirement for the pre-filling phase.

[0240] For the decoding stage, the amount of data transmitted may include, but is not limited to, the following operating parameters:

[0241] The amount of data transfer required to read keys and values ​​from the cache, the amount of data transfer required to read the architecture parameters, configuration parameters and / or other parameters of the target model, and the amount of data transfer required to read and write activation values.

[0242] The amount of data transfer required for reading and writing activation values ​​during the decoding phase is usually less than the amount of data transfer required for reading and writing activation values ​​during the pre-filling phase.

[0243] The amount of data transfer required to read the key and value from the cache during the decoding stage can be calculated using the following formula:

[0244] Here, KV_Cache(read_decode) represents the amount of data transfer required to read keys and values ​​from the cache.

[0245] The amount of data transfer required to write the key and value to the cache during the decoding stage can be calculated using the following formula:

[0246] KV_Cache(write_decode) represents the amount of data transfer required to write the key and value to the cache during the decoding phase.

[0247] Based on this, the total data transmission volume can be determined by summing the above-mentioned operating parameters regarding the data transmission volume required for the decoding stage, and the product of the data transmission volume per word and the number of words per second can be determined as the bandwidth requirement for the decoding stage.

[0248] Based on this, the sum of the bandwidth requirements of the pre-filling stage and the decoding stage can be used as the actual bandwidth requirement of the target model in the process of responding to the task request.

[0249] Based on this, by separating static memory usage, cache memory usage, and peak memory usage at different stages, fine-grained modeling of memory usage is achieved. The peak memory usage at different stages of the pre-filling and decoding stages is treated differently, avoiding the coarse results of uniform estimation and ensuring the completeness of actual memory requirement prediction.

[0250] By separately calculating the data transfer volume in the pre-filling stage and the decoding stage, the differences between different stages during memory access of the target model are accurately reflected. The actual bandwidth requirement obtained by summing the data transfer volume of the two stages can reflect the resource requirements of the real task.

[0251] refer to Figure 5 The diagram illustrates a decomposition process 500 of one embodiment of step 203 of this disclosure. This decomposition process 500 includes the following steps 501 to 502:

[0252] Step 501: Determine the parameter difference between each operating parameter and the corresponding actual resource requirement in the pre-filling stage and the decoding stage, respectively.

[0253] Based on the aforementioned operating parameters and corresponding actual resource requirements, since the operating parameters exhibit different performance in the pre-filling and decoding stages, and the actual resource requirements also differ in the pre-filling and decoding stages, the parameter differences can be calculated separately for the pre-filling and decoding stages.

[0254] Specifically, the difference between each operating parameter and the corresponding actual resource requirement can be calculated in the pre-filling stage, and the difference between each operating parameter and the corresponding actual resource requirement can be further calculated in the decoding stage.

[0255] Step 502: The ratio of the parameter difference between the pre-filling stage and the decoding stage to the corresponding running parameters is determined as the corresponding constraint degree.

[0256] Based on the differences between each stage determined in step 501 above, the corresponding degree of constraint can be determined for the operating parameters in each stage.

[0257] Specifically, for the pre-filling stage, the difference between the operating parameters of the pre-filling stage and the corresponding actual resource requirements can be determined first, and the ratio between the difference and the corresponding operating parameters can be used as the constraint degree of the operating parameters in the pre-filling stage; for the decoding stage, the difference between the operating parameters of the decoding stage and the corresponding actual resource requirements can be determined first, and the ratio between the difference and the corresponding operating parameters can be used as the constraint degree of the operating parameters in the decoding stage.

[0258] In a specific example, taking the computing resources and computing requirements provided during the decoding phase as an example, for instance, if the computing resources for the decoding phase are 100 TFLOPs / s and the computing requirements are 110 TFLOPs / s, then the constraint level of the computing resources is 0.1.

[0259] Based on this, by calculating the degree of constraint of each operating parameter in the pre-filling stage and the decoding stage, the influence of each operating parameter on the overall response process of the target model can be subdivided into the pre-filling stage and the encoding stage, and the degree of constraint can be determined more accurately and precisely.

[0260] Therefore, it is possible to determine whether the operating parameters constitute a bottleneck based on the relationship between the degree of constraint and the constraint threshold.

[0261] For example, if the constraint level of the computing resources is 0.1, and the preset constraint threshold is 0.08, then because the constraint level of 0.1 is greater than the constraint threshold of 0.08, this computing resource constitutes a bottleneck for the target model to respond to task requests smoothly and in a timely manner. If the preset constraint threshold is 0.15, then because the constraint level of 0.1 is less than the constraint threshold of 0.15, this computing resource does not constitute a bottleneck for the target model to respond to task requests smoothly and in a timely manner.

[0262] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model bottleneck determination device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0263] like Figure 6 As shown, the model bottleneck determination device 600 in this embodiment includes:

[0264] The parameter acquisition module 601 is configured to, in response to receiving a task request, acquire multiple operating parameters of the target model and the target hardware, under the condition that the target hardware carries the target model, wherein the target model is used to respond to the task request;

[0265] The requirement acquisition module 602 is configured to acquire multiple actual resource requirements of the task request, each actual resource requirement representing a requirement for the corresponding operating parameters.

[0266] The constraint degree calculation module 603 is configured to determine the degree of constraint of each operating parameter on the corresponding actual resource requirements;

[0267] The bottleneck determination module 604 is configured to determine the operating parameters whose constraint degree is greater than the corresponding preset constraint threshold as bottleneck parameters, and to determine the location in the target hardware and / or the target model pointed to by the bottleneck parameter as the bottleneck location.

[0268] In some optional implementations, the parameter acquisition module 601 is further configured to:

[0269] Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request;

[0270] Obtain multiple actual hardware parameters of the target hardware;

[0271] The various architecture parameters, configuration parameters, and actual hardware parameters are structured and then used as the operating parameters.

[0272] Among them, obtaining multiple actual hardware parameters of the target hardware includes:

[0273] Obtain multiple standard hardware parameters of the target hardware;

[0274] The corresponding actual hardware parameters are determined based on the preset compensation factors corresponding to each standard hardware parameter. Each compensation factor represents the degree of difference between the corresponding standard hardware parameter and the corresponding actual hardware parameter.

[0275] In some optional implementations, the actual resource requirements include the actual computing power requirements; the process by which the target model responds to the task request includes a pre-filling stage and a decoding stage; the compensation factor corresponding to each standard hardware parameter includes a first efficiency compensation factor in the pre-filling stage and a second efficiency compensation factor in the decoding stage.

[0276] Accordingly, the demand acquisition module 602 is further configured as follows:

[0277] Determine the computational requirements of the task request in the pre-filling stage and the decoding stage respectively;

[0278] The computational requirements of the pre-filling stage are corrected according to the first efficiency compensation factor to obtain the first equivalent computational quantity, and the computational requirements of the decoding stage are corrected according to the second efficiency compensation factor to obtain the second equivalent computational quantity.

[0279] The first equivalent computational load and the second equivalent computational load are determined as the actual computing power requirement.

[0280] In some alternative implementations, actual resource requirements include actual memory requirements.

[0281] Accordingly, the demand acquisition module 602 is further configured as follows:

[0282] Obtain the static memory requirement and cache memory requirement of the target model, wherein the static memory requirement and the cache memory requirement have the same memory value in the pre-filling stage and the decoding stage;

[0283] The peak memory values ​​of the task requests in the pre-filling stage and the decoding stage are determined, and the larger peak memory value between the pre-filling stage and the decoding stage, the sum of the static memory requirement and the cache memory requirement is determined as the actual memory requirement.

[0284] In some alternative implementations, actual resource requirements include actual bandwidth requirements.

[0285] Accordingly, the demand acquisition module 602 is further configured as follows:

[0286] Determine the bandwidth requirements of the task request in the pre-filling stage and the decoding stage respectively, and determine the sum of the bandwidth requirements of the pre-filling stage and the decoding stage as the actual bandwidth requirement;

[0287] The bandwidth requirement of the pre-filling stage is the ratio between the total data transmission volume of the pre-filling stage and the latency budget of the pre-filling stage, and the bandwidth requirement of the decoding stage is the product of the data transmission volume per word and the number of words per second.

[0288] In some optional implementations, the constraint degree calculation module 603 is further configured to:

[0289] Determine the parameter difference between each operating parameter and the corresponding actual resource requirement in the pre-filling stage and the decoding stage, respectively;

[0290] The ratio of the parameter difference between the pre-filling stage and the decoding stage to the corresponding operating parameters is determined as the corresponding constraint level.

[0291] It should be noted that the implementation details and technical effects of each unit in the model bottleneck determination device provided in the embodiments of this disclosure can be referred to the descriptions of other embodiments in this disclosure, and will not be repeated here.

[0292] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 7 The computer system 700 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0293] like Figure 7 As shown, the computer system 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the computer system 700. The processing device 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0294] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows computer system 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 A computer system 700 with various electronic devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0295] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0296] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0297] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0298] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the following functions: Figure 2 The illustrated embodiments and their alternative implementations demonstrate a task management method.

[0299] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0300] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0301] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the unit itself; for example, a parameter acquisition module can also be described as "a module that, in response to receiving a task request, acquires multiple operating parameters of the target model and the target hardware, provided that the target hardware carries the target model."

[0302] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for determining a model bottleneck, comprising: In response to receiving a task request, under the condition that the target model is deployed on the target hardware, multiple operating parameters of the target model and the target hardware are obtained, wherein the target model is used to respond to the task request; The acquisition of multiple operating parameters of the target model and the target hardware includes: Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request; Obtain multiple actual hardware parameters of the target hardware; The various architecture parameters, configuration parameters, and actual hardware parameters are structured and then determined as the various operating parameters. The acquisition of multiple actual hardware parameters of the target hardware includes: Obtain multiple standard hardware parameters of the target hardware; The corresponding actual hardware parameters are determined based on the preset compensation factors corresponding to each standard hardware parameter. Each compensation factor represents the degree of difference between the corresponding standard hardware parameter and the corresponding actual hardware parameter. Obtain the multiple actual resource requirements of the task request, where each actual resource requirement represents the requirement for the corresponding operating parameters; Determine the degree of constraint that each operating parameter imposes on the corresponding actual resource requirements; The operating parameters whose constraint level is greater than the corresponding preset constraint threshold are identified as bottleneck parameters, and the location in the target hardware and / or the target model pointed to by the bottleneck parameter is identified as the bottleneck location.

2. The method according to claim 1, wherein, The actual resource requirements include actual computing power requirements; the process by which the target model responds to the task request includes a pre-filling stage and a decoding stage; the compensation factor corresponding to each standard hardware parameter includes a first efficiency compensation factor in the pre-filling stage and a second efficiency compensation factor in the decoding stage. as well as The process of obtaining the multiple actual resource requirements requested in the task request includes: Determine the computational requirements of the task request in the pre-filling stage and the decoding stage, respectively; The computational requirements of the pre-filling stage are corrected according to the first efficiency compensation factor to obtain the first equivalent computational quantity, and the computational requirements of the decoding stage are corrected according to the second efficiency compensation factor to obtain the second equivalent computational quantity. The first equivalent computational load and the second equivalent computational load are determined as the actual computing power requirement.

3. The method according to claim 2, wherein, The actual resource requirements include actual memory requirements; as well as The process of obtaining the multiple actual resource requirements requested in the task request includes: Obtain the static memory requirement and cache memory requirement of the target model, wherein the static memory requirement and the cache memory requirement have the same memory value in the pre-filling stage and the decoding stage; Determine the peak memory usage of the task request during the pre-filling and decoding phases, respectively. The larger of the stage memory peak values ​​from the pre-filling stage and the decoding stage, the sum of the static memory requirement and the cache memory requirement, is determined as the actual memory requirement.

4. The method according to claim 2, wherein, The actual resource requirements include actual bandwidth requirements; as well as The process of obtaining the multiple actual resource requirements requested in the task request includes: Determine the bandwidth requirements of the task request in the pre-filling stage and the decoding stage respectively, and determine the sum of the bandwidth requirements of the pre-filling stage and the decoding stage as the actual bandwidth requirement; The bandwidth requirement of the pre-filling stage is the ratio between the total data transmission volume of the pre-filling stage and the latency budget of the pre-filling stage, and the bandwidth requirement of the decoding stage is the product of the data transmission volume per word and the number of words per second.

5. The method according to claim 2, wherein, Determining the degree of constraint of each operating parameter on the corresponding actual resource requirements includes: Determine the parameter difference between each operating parameter and the corresponding actual resource requirement in the pre-filling stage and the decoding stage, respectively; The ratio of the parameter difference between the pre-filling stage and the decoding stage to the corresponding operating parameters is determined as the corresponding constraint level.

6. A model bottleneck determination device, comprising: The parameter acquisition module is configured to, in response to receiving a task request, acquire multiple operating parameters of the target model and the target hardware, under the condition that the target hardware carries the target model, wherein the target model is used to respond to the task request; The acquisition of multiple operating parameters of the target model and the target hardware includes: Obtain multiple architectural parameters of the target model and multiple configuration parameters that match the task request; Obtain multiple actual hardware parameters of the target hardware; The various architecture parameters, configuration parameters, and actual hardware parameters are structured and then determined as the various operating parameters. The acquisition of multiple actual hardware parameters of the target hardware includes: Obtain multiple standard hardware parameters of the target hardware; The corresponding actual hardware parameters are determined based on the preset compensation factors corresponding to each standard hardware parameter. Each compensation factor represents the degree of difference between the corresponding standard hardware parameter and the corresponding actual hardware parameter. The requirement acquisition module is configured to acquire multiple actual resource requirements of the task request, each actual resource requirement representing a requirement for the corresponding operating parameters. The constraint degree calculation module is configured to determine the degree of constraint of each running parameter on the corresponding actual resource requirements; The bottleneck determination module is configured to identify the operating parameters whose constraint degree is greater than the corresponding preset constraint threshold as bottleneck parameters, and to identify the location in the target hardware and / or the target model pointed to by the bottleneck parameter as the bottleneck location.

7. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by one or more processors, it implements the method as described in any one of claims 1 to 5.

9. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 5.

Images