Method, device, medium and product for evaluating service level of intelligent inference cluster

Abstract

The present disclosure provides a method and device for evaluating the service level of an intelligent inference cluster, a medium and a product, which are applied to the technical field of wisdom calculation center construction and evaluation. In the present disclosure, according to the type of the artificial intelligence model loaded on the artificial intelligence acceleration card in the intelligent inference cluster to be evaluated, the maximum number of card conversations supported by the artificial intelligence acceleration card is determined, and the type is used to represent the sensitivity of the artificial intelligence model to the time delay. According to the maximum number of card conversations and the business parameters of the intelligent inference cluster to be evaluated, the number of online users representing the service level of the intelligent inference cluster to be evaluated is calculated. Because the inference result of the intelligent inference cluster needs to be pushed to the customer, using the number of online users as an index for evaluating the service level of the intelligent inference cluster can be closer to the actual business scenario, making the evaluation more accurate. Therefore, the evaluation accuracy of the service level of the intelligent inference cluster is improved.

Description

Technical Field

[0001] This application belongs to the field of intelligent computing center construction and evaluation technology, specifically involving a method, device, medium and product for evaluating the business level of an intelligent reasoning cluster. Background Technology

[0002] Generally, artificial intelligence (AI) computing can be divided into two scenarios: intelligent training and intelligent inference. Intelligent training refers to the process of training an AI model. The training effectiveness of the AI ​​model can be reflected by calculating the computing power scale of the intelligent training cluster, thus evaluating the intelligent training cluster in the intelligent computing center. Intelligent inference refers to inputting data from real-world business scenarios into the trained AI model to deduce various results. The intelligent inference component is deployed in the intelligent inference cluster of the intelligent computing center, and its processing power reflects the level of the intelligent inference cluster within the center.

[0003] As artificial intelligence models are increasingly deployed in real-world business scenarios, the performance evaluation of intelligent inference clusters within intelligent computing centers becomes particularly crucial. Currently, the evaluation of the performance of intelligent inference clusters often employs the same methods as that used for intelligent training clusters. However, intelligent inference clusters and intelligent training clusters are geared towards different scenarios and have different evaluation metrics. This approach can lead to inaccurate assessments of the performance of intelligent inference clusters. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, medium and product for evaluating the business level of intelligent inference clusters, thereby improving the accuracy of evaluating the business level of intelligent inference clusters.

[0005] According to one aspect of this disclosure, a method for evaluating the business level of an intelligent inference cluster is provided, comprising:

[0006] Based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated, determine the maximum number of card sessions supported by the AI ​​accelerator card, where the type is used to characterize the sensitivity of the AI ​​model to latency.

[0007] Based on the maximum number of sessions per card and the business parameters of the intelligent inference cluster to be evaluated, calculate the number of online users used to characterize the business level of the intelligent inference cluster to be evaluated.

[0008] Optionally, determining the maximum number of sessions supported by the AI ​​accelerator card based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated includes:

[0009] Obtain the available memory space of the AI ​​accelerator card, where the available memory space is the maximum available space in the AI ​​accelerator card that can cache intermediate results generated by the session during inference;

[0010] Based on the available memory space and the first memory space, the number of first sessions is calculated, where the first memory space is the memory space required by each session during the inference process;

[0011] When the type characterizes the artificial intelligence model as insensitive to latency, the first number of sessions is taken as the maximum number of sessions for the card;

[0012] When the type characterizes the artificial intelligence model as being sensitive to latency, the second number of sessions is calculated based on the second memory space and the first memory space; the maximum number of sessions for the card is determined based on the first number of sessions and the second number of sessions; the second memory space characterizes the memory space that the artificial intelligence accelerator card allows the artificial intelligence model to use, and the second memory space is determined based on the latency parameter characterizing the response speed of the artificial intelligence model.

[0013] Optionally, obtaining the available memory space of the AI ​​accelerator card includes:

[0014] The available memory space is obtained by subtracting the memory space occupied by the operating system of the AI ​​accelerator card and the memory space occupied by the AI ​​model from the total memory space of the AI ​​accelerator card.

[0015] Optionally, determining the maximum number of sessions for the card based on the first number of sessions and the second number of sessions includes:

[0016] If the second session count is greater than or equal to zero, the minimum session count between the first session count and the second session count shall be taken as the maximum session count of the card;

[0017] If the number of the second sessions is less than zero, the second memory space is expanded by updating the latency parameter; the second memory space is updated to the expanded second memory space, and the process returns to the step of calculating the number of the second sessions based on the second memory space and the first memory space, until the number of the second sessions is greater than or equal to zero, and the minimum number of sessions between the first number of sessions and the second number of sessions is taken as the maximum number of sessions for the card.

[0018] Optionally, calculating the number of online users to characterize the service level of the intelligent inference cluster to be evaluated based on the maximum number of sessions of the card and the service parameters of the intelligent inference cluster to be evaluated includes:

[0019] The maximum number of sessions in the cluster is determined based on the number of AI acceleration cards in the intelligent inference cluster to be evaluated and the maximum number of sessions per card.

[0020] The average number of online users in the cluster is determined based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters.

[0021] The number of online users is determined based on the average number of online users in the cluster and the duration parameter in the business parameters.

[0022] Optionally, determining the average number of online users in the cluster based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters includes:

[0023] The number of online users during the peak business period is calculated by dividing the maximum number of sessions in the cluster by the resource utilization rate in the resource usage parameters.

[0024] The average number of online users in the cluster is calculated by dividing the number of online users during the peak business period by the peak concentration coefficient in the resource usage parameters.

[0025] Optionally, determining the number of online users based on the average number of online users in the cluster and the duration parameter in the service parameters includes:

[0026] The average total online time of online users is calculated by dividing the average number of online users in the cluster by the service duration of the intelligent inference cluster to be evaluated in the duration parameter.

[0027] The number of online users is calculated using the average total online time of the online users and the average online time of each online user in the duration parameter.

[0028] According to another aspect of this disclosure, an apparatus for evaluating the business level of an intelligent inference cluster is provided, comprising:

[0029] The determination module is used to determine the maximum number of sessions supported by the AI ​​acceleration card based on the type of AI model loaded on the AI ​​acceleration card in the intelligent inference cluster to be evaluated, wherein the type is used to characterize the sensitivity of the AI ​​model to latency.

[0030] The calculation module is used to calculate the number of online users that characterize the service level of the intelligent inference cluster to be evaluated, based on the maximum number of sessions of the card and the service parameters of the intelligent inference cluster to be evaluated.

[0031] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for evaluating the business level of the intelligent inference cluster.

[0032] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the above-described method for evaluating the business level of the intelligent inference cluster.

[0033] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes an evaluation method for implementing the above-described intelligent inference cluster business level.

[0034] As will be described in detail below, the method, apparatus, medium, and product for evaluating the service level of an intelligent inference cluster according to embodiments of this disclosure determine the maximum number of sessions supported by the AI ​​accelerator card based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated. The type is used to characterize the sensitivity of the AI ​​model to latency. Based on the maximum number of sessions and the service parameters of the intelligent inference cluster to be evaluated, the number of online users used to characterize the service level of the intelligent inference cluster is calculated. Different sensitivities of the AI ​​model to latency indicate different service carrying capacities of the intelligent inference cluster. Using latency sensitivity as one of the evaluation indicators makes the evaluation of service level closer to actual business scenarios. Furthermore, the number of online users, i.e., the user access volume that the intelligent inference cluster can support, can be calculated using this evaluation indicator characterizing the AI ​​model's latency sensitivity. Because the inference results of the intelligent inference cluster need to be pushed to customers, using the number of online users as an indicator for evaluating the service level of the intelligent inference cluster can more closely approximate actual business scenarios, making the evaluation more accurate. Therefore, the accuracy of the evaluation of the service level of the intelligent inference cluster is improved.

[0035] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0036] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0037] Figure 1 This is a flowchart illustrating a method for evaluating the service level of an intelligent inference cluster according to an embodiment of the present disclosure.

[0038] Figure 2 This is a flowchart illustrating a method for calculating the maximum number of sessions on a card according to an embodiment of the present disclosure.

[0039] Figure 3 This is a flowchart illustrating a method for calculating the number of online users according to an embodiment of the present disclosure.

[0040] Figure 4 This is another flowchart illustrating a method for evaluating the service level of an intelligent inference cluster according to an embodiment of the present disclosure.

[0041] Figure 5 This is a schematic diagram of the structure of an intelligent inference cluster business level assessment device according to an embodiment of the present disclosure.

[0042] Figure 6 This is a hardware block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0043] Figure 7 This is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solution of this application, the application scenario of this application will be described first below.

[0045] Generally, depending on the specific AI computing scenario, AI computing can be divided into intelligent training and intelligent inference. Intelligent training refers to the learning process, that is, the process of training an AI model. Because the intelligent training process does not require interaction with the client, the evaluation metric is relatively simple. The training effect of the AI ​​model can be reflected by the computing power scale, i.e., the number of computing cards, thus evaluating the intelligent training cluster in the intelligent computing center. Furthermore, the trained AI model can be used in intelligent inference scenarios.

[0046] Intelligent reasoning refers to applying trained artificial intelligence (AI) models to new data or scenarios, specifically data from real-world business situations, to make predictions or decisions. During the reasoning process, input data is typically passed to the AI ​​model, which then derives various corresponding results. Reasoning is the final stage in AI applications; it applies the trained AI model to real-world scenarios to solve practical problems. The intelligent reasoning component is deployed in the intelligent reasoning cluster of an intelligent computing center, and the level of this cluster can be reflected in its ability to process business data.

[0047] Currently, as AI models are increasingly deployed in real-world business scenarios, evaluating the performance of intelligent inference clusters within intelligent computing centers is crucial. Current assessments of the performance of intelligent inference clusters often employ the same methods as those used for intelligent training clusters. However, intelligent inference clusters and intelligent training clusters are designed for different scenarios, and therefore require different evaluation metrics. Since the results of intelligent inference can be directly delivered to customers, the performance of an intelligent inference cluster cannot be simply equated with the computing power of an intelligent training cluster. Using the same evaluation methods will lead to inaccurate assessments of the performance of intelligent inference clusters.

[0048] To address the aforementioned technical issues, this disclosure provides a method, apparatus, medium, and product for evaluating the service level of an intelligent inference cluster. In this disclosure, the maximum number of sessions supported by the AI ​​accelerator card is determined based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated. The type is used to characterize the AI ​​model's sensitivity to latency. Based on the maximum number of card sessions and the service parameters of the intelligent inference cluster to be evaluated, the number of online users used to characterize the service level of the intelligent inference cluster is calculated. Different sensitivities of AI models to latency indicate different service carrying capacities of the intelligent inference cluster. Using latency sensitivity as one of the evaluation indicators makes the service level evaluation more closely resemble actual business scenarios. Furthermore, the number of online users, i.e., the user access volume that the intelligent inference cluster can support, can be calculated using this evaluation indicator characterizing the AI ​​model's latency sensitivity. Because the inference results of the intelligent inference cluster need to be pushed to customers, using the number of online users as an indicator for evaluating the service level of the intelligent inference cluster can more closely approximate actual business scenarios, making the evaluation more accurate. Therefore, the accuracy of the service level evaluation of the intelligent inference cluster is improved.

[0049] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0050] Figure 1 This is a flowchart illustrating a method for evaluating the business level of an intelligent inference cluster, as provided in this disclosure. Figure 1 As shown, the method includes:

[0051] S101: Determine the maximum number of sessions supported by the AI ​​accelerator card based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated.

[0052] Specifically, the type of AI model is used to characterize its sensitivity to latency. Different methods are used to determine the maximum number of sessions per card depending on the type of AI model. The maximum number of sessions per card represents the maximum number of sessions the AI ​​acceleration card can support when that AI model is loaded.

[0053] Artificial intelligence models can be categorized into latency-sensitive and latency-insensitive types. Latency-sensitive models require a high response rate, meaning they must return results within a pre-defined latency parameter. Latency-insensitive models, on the other hand, do not require a high response rate; they simply need to return results without regard to the time taken.

[0054] When determining the maximum number of sessions for a card, if the AI ​​model is insensitive to latency, then there is no need to consider latency-related metrics; if the AI ​​model is latency-sensitive, then the impact of latency-related metrics on the determination of the maximum number of sessions needs to be considered.

[0055] The varying sensitivities of AI models to latency indicate different service carrying capacities of intelligent inference clusters. Using latency sensitivity as one of the evaluation metrics can make the assessment of service performance more closely reflect real-world business scenarios.

[0056] S102: Calculate the number of online users to characterize the business level of the intelligent inference cluster to be evaluated, based on the maximum number of sessions per card and the business parameters of the intelligent inference cluster to be evaluated.

[0057] Specifically, the business parameters include resource usage parameters and duration parameters. By combining the maximum number of sessions per AI acceleration card with the resource usage and duration parameters of the intelligent inference cluster to be evaluated, the number of online users of the intelligent inference cluster can be calculated, which represents the user access volume that the intelligent inference cluster can support. The number of online users is used to characterize the business performance of the intelligent inference cluster to be evaluated.

[0058] The inference results of the intelligent inference cluster need to be pushed to customers. Therefore, the number of online users is used as an indicator to evaluate the business level of the intelligent inference cluster. This quantitatively evaluates the business processing capacity of the intelligent inference cluster in the intelligent computing center, which can be closer to the actual business scenario and make the evaluation more accurate.

[0059] In this disclosure, the maximum number of sessions supported by the AI ​​accelerator card is determined based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated. The type is used to characterize the sensitivity of the AI ​​model to latency. Based on the maximum number of sessions and the business parameters of the intelligent inference cluster to be evaluated, the number of online users is calculated to characterize the business level of the intelligent inference cluster. Different sensitivities of AI models to latency indicate different business carrying capacities of the intelligent inference cluster. Using latency sensitivity as one of the evaluation indicators makes the assessment of business level closer to actual business scenarios. Furthermore, the number of online users, i.e., the user access volume that the intelligent inference cluster can support, can be calculated using this evaluation indicator characterizing the AI ​​model's latency sensitivity. Since the inference results of the intelligent inference cluster need to be pushed to customers, using the number of online users as an indicator to evaluate the business level of the intelligent inference cluster can more closely approximate actual business scenarios, making the evaluation more accurate. Therefore, the accuracy of the business level assessment of the intelligent inference cluster is improved.

[0060] Figure 2 A flowchart illustrating an exemplary method for determining the maximum number of sessions supported by an AI accelerator card, based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated, in one possible embodiment, is shown below. Figure 2 As shown, the method includes:

[0061] S201: Obtain the available memory space of the AI ​​accelerator card.

[0062] The available memory space is the maximum available space in the AI ​​accelerator card that can cache intermediate results generated during the inference process of the session.

[0063] For example, when inputting "What's the weather like today?" into a language-based AI model, the model will output the result word by word during intelligent reasoning. Each word output represents an intermediate result generated during the reasoning process. If the final output is "Today is sunny.", then "today" is cached as one of the intermediate results in the available memory space. All intermediate results before the final result is output are cached in the available memory space.

[0064] For example, the available memory space of the AI ​​accelerator card includes:

[0065] The available memory space is obtained by subtracting the memory space occupied by the operating system and the memory space occupied by the AI ​​accelerator card from the total memory space of the AI ​​accelerator card.

[0066] Specifically, the AI ​​accelerator card will contain an operating system for running AI models, such as Windows or Linux. The operating system itself will occupy a portion of the AI ​​accelerator card's memory. The AI ​​models loaded onto the AI ​​accelerator card will also occupy a portion of its memory. The amount of memory occupied by the loaded AI model is related to the number of its parameters.

[0067] Based on the total memory space of the AI ​​accelerator card, subtract the memory space occupied by the AI ​​accelerator card's operating system, and then subtract the memory space occupied by the loaded AI model. The remaining memory space is the actual usable memory space, which is used to cache intermediate results generated during the inference process of the session.

[0068] S202: Calculate the number of first sessions based on available memory space and first memory space.

[0069] The first memory space is the memory space occupied by each session during inference, and the size of the first memory space is related to the structure of the loaded artificial intelligence model.

[0070] Specifically, by dividing the available memory space obtained by S201 by the first memory space, the number of sessions that the AI ​​accelerator card can support can be obtained.

[0071] The number of first sessions can be calculated using the following formula:

[0072]

[0073] Among them, Session0 is the first session count, VR (Video Random Access Memory) is the total memory space of the AI ​​accelerator card, OS (Operating System) is the memory space occupied by the operating system of the AI ​​accelerator card, LM (Large-scale model) is the memory space occupied by the loaded AI model, and Cache is the first memory space.

[0074] S203: When the type representation AI model is not sensitive to latency, the number of the first session is used as the maximum number of sessions for the card.

[0075] Specifically, when the type of artificial intelligence model is latency-insensitive, when determining the maximum number of card sessions, there is no need to consider the latency parameter, and the first number of sessions obtained in S202 can be used as the maximum number of card sessions for the artificial intelligence acceleration card.

[0076] S204: When the type representation AI model is sensitive to latency, calculate the number of second sessions based on the second memory space and the first memory space; determine the maximum number of sessions for the card based on the number of first sessions and the number of second sessions.

[0077] The second memory space is the memory space that the AI ​​accelerator card allows the AI ​​model to use. Furthermore, the second memory space is determined based on the latency parameter that characterizes the response speed of the AI ​​model. In other words, the size of the second memory space is related to the latency parameter corresponding to the AI ​​model.

[0078] Specifically, latency parameters characterizing the response speed of an artificial intelligence model include the latency requirements of the artificial intelligence model, the bandwidth of the graphics processing unit (GPU) card accessing memory, and the bandwidth utilization of the GPU card's memory access bandwidth.

[0079] By multiplying the latency requirement, the GPU card's memory access bandwidth, and bandwidth utilization, the total memory space of the AI ​​accelerator card within the latency requirement can be calculated. Subtracting the memory space occupied by the loaded AI model, we can obtain the memory space that can be read from the AI ​​accelerator card within the latency requirement, which is the memory space that the AI ​​accelerator card allows the AI ​​model to use.

[0080] Then, based on the second memory space and the first memory space, the second number of sessions is calculated. The second number of sessions represents the number of sessions that the AI ​​acceleration card can support within the specified latency requirements.

[0081] The number of second sessions can be calculated using the following formula:

[0082]

[0083] Where Session1 is the number of second sessions, T (Time) is the latency requirement of the artificial intelligence model, B (Bandwidth) is the bandwidth of the GPU card accessing memory, r (rate) is the bandwidth utilization rate, LM is the memory space occupied by the loaded artificial intelligence model, and Cache is the first memory space.

[0084] For example, determining the maximum number of sessions for a card based on the first number of sessions and the second number of sessions includes:

[0085] If the number of the second session is greater than or equal to zero, the minimum number of sessions between the number of the first session and the number of the second session will be used as the maximum number of sessions for the card.

[0086] Specifically, when the number of the first session is greater than or equal to the number of the second session, it indicates that the loaded artificial intelligence model is more sensitive to the latency requirement, and the number of sessions supported by the artificial intelligence accelerator card has not reached the upper limit of the number of sessions that the artificial intelligence accelerator card can support. Therefore, the number of the second session is taken as the maximum number of sessions that the artificial intelligence accelerator card can support.

[0087] When the number of sessions in the first session is less than the number of sessions in the second session, it means that the loaded AI model is not sensitive to the latency requirement, and the AI ​​accelerator card is suitable for the loaded AI model to perform intelligent inference. However, due to the limitation of the AI ​​accelerator card's own memory space, the number of sessions in the first session is taken as the maximum number of sessions that can be supported.

[0088] If the number of second sessions is less than zero, the second memory space is expanded by updating the latency parameter. The second memory space is then updated to the expanded second memory space. The process then returns to the step of calculating the number of second sessions based on the second memory space and the first memory space, until the number of second sessions is greater than or equal to zero. The minimum number of sessions between the first and second sessions is then taken as the maximum number of sessions for the card.

[0089] Specifically, if the number of second sessions is less than zero, it means that the bandwidth and bandwidth utilization of the GPU card accessing memory of the AI ​​accelerator card cannot support the inference of the session. It is necessary to enable the AI ​​accelerator card to read more resource data within the latency requirements, which can be achieved by increasing the bandwidth and bandwidth utilization of the GPU card accessing memory.

[0090] Based on the updated latency parameters, the expanded second memory space is obtained, and the number of second sessions is recalculated until the number of second sessions is greater than or equal to zero. The recalculated number of second sessions is compared with the number of first sessions, and the minimum number of sessions between the two is determined as the maximum number of sessions on the card.

[0091] Figure 3 A flowchart illustrating an exemplary method for calculating the number of online users characterizing the service level of the intelligent inference cluster to be evaluated, based on the maximum number of card sessions and service parameters of the intelligent inference cluster to be evaluated, is provided in one possible embodiment. Figure 3 As shown, the method includes:

[0092] S301: Determine the maximum number of sessions in the cluster based on the number of AI accelerator cards and the maximum number of sessions per card in the intelligent inference cluster to be evaluated.

[0093] Specifically, as mentioned earlier, the maximum number of sessions per AI accelerator card was obtained using latency sensitivity as an evaluation metric. An AI inference cluster consists of multiple AI accelerator cards. By multiplying the number of AI accelerator cards by their maximum number of sessions, the maximum number of sessions the cluster can support for n online users during peak business hours can be calculated.

[0094] When the type representation of the AI ​​model loaded by the AI ​​accelerator card is sensitive to latency, the maximum number of sessions in the cluster can be calculated using the following intermediate formula:

[0095] AI×MIN(Session0, Session1)

[0096] Wherein, AI (Artificial Intelligence) is the number of AI accelerator cards in the intelligent inference cluster to be evaluated, and MIN (Session0, Session1) is the maximum number of sessions per AI accelerator card.

[0097] Assuming the type representation of the AI ​​model loaded by the AI ​​accelerator card is insensitive to latency, the maximum number of sessions in the cluster can be calculated using the following intermediate formula:

[0098] AI×Session0

[0099] Where AI represents the number of AI accelerator cards in the intelligent inference cluster to be evaluated, and Session0 represents the maximum number of sessions per AI accelerator card.

[0100] S302: Determine the average number of online users in the cluster based on the maximum number of sessions in the cluster and the resource usage parameters in the business parameters.

[0101] Specifically, the maximum number of sessions in the intelligent inference cluster can be obtained based on the average number of online users and the resource usage parameters of users during session inference. In this embodiment, the average number of online users per hour can be obtained through reverse derivation based on the maximum number of sessions and resource usage parameters. The resource usage parameters are obtained according to the actual business operations carried by the intelligent inference cluster.

[0102] For example, the average number of online users in the cluster is determined based on the maximum number of sessions in the cluster and the resource usage parameters in the business parameters, including:

[0103] The number of online users during peak business hours is calculated by dividing the maximum number of sessions in the cluster by the resource utilization rate in the resource usage parameters; the average number of online users in the cluster is calculated by dividing the number of online users during peak business hours by the peak concentration coefficient in the resource usage parameters.

[0104] Specifically, resource usage parameters include resource utilization rate and peak concentration factor. Resource utilization rate indicates that during peak business periods, some users may be online but not sending inference requests for sessions; in this case, they can be considered not consuming inference resources. Peak concentration factor indicates the ratio of the number of online users to the average number of online users during peak business periods.

[0105] When the type representation of the AI ​​model loaded by the AI ​​accelerator card is sensitive to latency, the average number of online users can be obtained according to the following formula:

[0106]

[0107] Where AI represents the number of AI accelerator cards in the intelligent inference cluster to be evaluated, MIN(Session0, Session1) represents the maximum number of sessions per AI accelerator card, UR(User Rate) represents the resource utilization rate of user-requested sessions, and PC(Peak Coefficient) represents the peak concentration coefficient.

[0108] Assuming the type representation of the AI ​​model loaded by the AI ​​accelerator card is insensitive to latency, the average number of online users can be obtained using the following formula:

[0109]

[0110] Where AI represents the number of AI accelerator cards in the intelligent inference cluster to be evaluated, Session0 represents the maximum number of sessions per AI accelerator card, UR represents the resource utilization rate of user-requested sessions, and PC represents the peak concentration factor.

[0111] In this embodiment, the average number of online users can be calculated in hours, i.e., PC is the daily peak business concentration factor.

[0112] S303: Determine the number of online users based on the average number of online users in the cluster and the duration parameter in the business parameters.

[0113] Specifically, the average number of online users in the intelligent inference cluster can be obtained based on the number of online users and the duration of online interaction during the session inference process. In this embodiment, the number of online users, i.e., the number of online users n mentioned in S301, can be obtained through reverse derivation based on the average number of online users and the duration parameter. The duration parameter is obtained according to the actual business carried by the intelligent inference cluster.

[0114] For example, the number of online users is determined based on the average number of online users in the cluster and the duration parameter in the business parameters, including:

[0115] The average total online time of online users is calculated by dividing the average number of online users in the cluster by the service duration of the intelligent inference cluster to be evaluated in the duration parameter; the number of online users is calculated by using the average total online time of online users and the average online time of each online user in the duration parameter.

[0116] Specifically, the duration parameters include the service duration of the intelligent inference cluster to be evaluated and the average online duration of each online user.

[0117] When the type representation of the AI ​​model loaded by the AI ​​accelerator card is sensitive to latency, the number of online users can be obtained using the following formula:

[0118]

[0119] Wherein, UC (User Count) is the number of online users, AI is the number of AI accelerator cards in the intelligent inference cluster to be evaluated, MIN (Session0, Session1) is the maximum number of sessions per AI accelerator card, UR is the resource utilization rate of user-requested sessions, PC is the peak concentration factor, DT (Day Time) is the service duration of the intelligent inference cluster to be evaluated, and LT (Line Time) is the average online time for each online user.

[0120] Assuming the type representation of the AI ​​model loaded by the AI ​​accelerator card is insensitive to latency, the number of online users can be obtained using the following formula:

[0121]

[0122] Wherein, UC is the number of online users, AI is the number of AI accelerator cards in the intelligent inference cluster to be evaluated, Session0 is the maximum number of sessions per AI accelerator card, UR is the resource utilization rate of user-requested sessions, PC is the peak concentration factor, DT is the service duration of the intelligent inference cluster to be evaluated, and LT is the average online time for each online user.

[0123] In this embodiment, DT can obtain data on the daily service duration of the intelligent inference cluster to be evaluated, and LT can obtain the average daily online duration of each online user.

[0124] This embodiment is based on the actual user access scenario of the intelligent inference cluster in the intelligent computing center. It considers the impact of latency parameters on service level and conducts service level assessments for latency-insensitive and latency-sensitive artificial intelligence models respectively. Moreover, it is closer to the actual business scenario than simply using computing power scale to assess the service level of the intelligent inference cluster, thus improving the accuracy of the assessment of the service level of the intelligent inference cluster.

[0125] Figure 4 Another flowchart for a method of evaluating the business level of an intelligent inference cluster provided in this disclosure. (See attached flowchart.) Figure 4 As shown, the method includes:

[0126] S401: Evaluate the number of first sessions for the AI ​​accelerator card.

[0127] In this embodiment, the type of artificial intelligence model loaded in the artificial intelligence accelerator card is used as an example to illustrate the type that is sensitive to latency.

[0128] The available memory space of the AI ​​accelerator card is obtained by subtracting the memory space occupied by the operating system and the memory space occupied by the AI ​​model from the total memory space of the AI ​​accelerator card.

[0129] The number of sessions for the AI ​​accelerator card can be obtained by dividing the available memory space by the first memory space. The first memory space is the memory space occupied by each session during inference.

[0130] S402: Evaluate the number of second sessions of the AI ​​accelerator card under preset latency parameters.

[0131] In cases where the AI ​​model is latency-sensitive, by multiplying the latency requirement, the GPU card's memory access bandwidth, and the bandwidth utilization in the latency parameters, we can calculate the total memory space of the AI ​​accelerator card within the latency requirement. Subtracting the memory space occupied by the loaded AI model, we can obtain the memory space that can be read from the AI ​​accelerator card within the latency requirement, which is the second memory space that the AI ​​accelerator card allows the AI ​​model to use.

[0132] Then, the number of sessions for the AI ​​accelerator card is obtained by dividing the second memory space by the first memory space.

[0133] S403: Calculate the number of online users that characterize the business level of the intelligent inference cluster.

[0134] The minimum number of sessions between the first and second session counts is taken as the maximum number of sessions per card. Based on the maximum number of sessions per card, the number of AI acceleration cards in the intelligent inference cluster, resource usage parameters, and time parameters, the number of online users can be calculated. The number of online users is used to characterize the business performance of the intelligent inference cluster.

[0135] Figure 5 This is a schematic diagram of a device for evaluating the business level of an intelligent inference cluster, provided in this disclosure, used to execute the aforementioned method for evaluating the business level of the intelligent inference cluster. Figure 5 As shown, the device 500 includes a determination module 510 and a calculation determination module 520.

[0136] The determination module 510 is used to determine the maximum number of card sessions supported by the artificial intelligence acceleration card based on the type of artificial intelligence model loaded on the artificial intelligence acceleration card in the intelligent inference cluster to be evaluated, wherein the type is used to characterize the sensitivity of the artificial intelligence model to latency.

[0137] The calculation module 520 is used to calculate the number of online users that characterize the service level of the intelligent inference cluster to be evaluated, based on the maximum number of sessions of the card and the service parameters of the intelligent inference cluster to be evaluated.

[0138] Optionally, the determining module is used to:

[0139] Obtain the available memory space of the AI ​​accelerator card, where the available memory space is the maximum available space in the AI ​​accelerator card that can cache intermediate results generated by the session during inference;

[0140] Based on the available memory space and the first memory space, the number of first sessions is calculated, where the first memory space is the memory space required by each session during the inference process;

[0141] When the type characterizes the artificial intelligence model as insensitive to latency, the first number of sessions is taken as the maximum number of sessions for the card;

[0142] When the type characterizes the artificial intelligence model as being sensitive to latency, the second number of sessions is calculated based on the second memory space and the first memory space; the maximum number of sessions for the card is determined based on the first number of sessions and the second number of sessions; the second memory space characterizes the memory space that the artificial intelligence accelerator card allows the artificial intelligence model to use, and the second memory space is determined based on the latency parameter characterizing the response speed of the artificial intelligence model.

[0143] Optionally, the determining module is used to:

[0144] The available memory space is obtained by subtracting the memory space occupied by the operating system of the AI ​​accelerator card and the memory space occupied by the AI ​​model from the total memory space of the AI ​​accelerator card.

[0145] Optionally, the determining module is used to:

[0146] If the second session count is greater than or equal to zero, the minimum session count between the first session count and the second session count shall be taken as the maximum session count of the card;

[0147] If the number of the second sessions is less than zero, the second memory space is expanded by updating the latency parameter; the second memory space is updated to the expanded second memory space, and the process returns to the step of calculating the number of the second sessions based on the second memory space and the first memory space, until the number of the second sessions is greater than or equal to zero, and the minimum number of sessions between the first number of sessions and the second number of sessions is taken as the maximum number of sessions for the card.

[0148] Optionally, the computing module is used for:

[0149] The maximum number of sessions in the cluster is determined based on the number of AI acceleration cards in the intelligent inference cluster to be evaluated and the maximum number of sessions per card.

[0150] The average number of online users in the cluster is determined based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters.

[0151] The number of online users is determined based on the average number of online users in the cluster and the duration parameter in the business parameters.

[0152] Optionally, the computing module is used for:

[0153] The number of online users during the peak business period is calculated by dividing the maximum number of sessions in the cluster by the resource utilization rate in the resource usage parameters.

[0154] The average number of online users in the cluster is calculated by dividing the number of online users during the peak business period by the peak concentration coefficient in the resource usage parameters.

[0155] Optionally, the computing module is used for:

[0156] The average total online time of online users is calculated by dividing the average number of online users in the cluster by the service duration of the intelligent inference cluster to be evaluated in the duration parameter.

[0157] The number of online users is calculated using the average total online time of the online users and the average online time of each online user in the duration parameter.

[0158] This application also provides an electronic device for performing the above-described method for evaluating the service level of an intelligent inference cluster. Please refer to... Figure 6 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected via the bus 602. The memory 601 stores a computer program that can run on the processor 600. When the processor 600 runs the computer program, it executes the business level evaluation method of the intelligent inference cluster provided in any of the foregoing embodiments of this application.

[0159] The memory 601 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 603 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0160] Bus 602 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs. After receiving an execution instruction, the processor 600 executes the program. The service level evaluation method of the intelligent inference cluster disclosed in any of the foregoing embodiments of this application can be applied to the processor 600, or implemented by the processor 600.

[0161] The processor 600 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 600 or by instructions in software form. The processor 600 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 601. Processor 600 reads the information in memory 601 and, in conjunction with its hardware, completes the steps of the above method.

[0162] The electronic device provided in this application embodiment and the business level evaluation method of the intelligent inference cluster provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0163] This application also provides a computer-readable storage medium corresponding to the business level assessment method of the intelligent inference cluster provided in the foregoing embodiments. The computer-readable storage medium shown can be an optical disc, on which a computer program is stored. When the computer program is run by a processor, it executes the business level assessment method of the intelligent inference cluster provided in any of the foregoing embodiments.

[0164] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0165] The computer-readable storage medium provided in the above embodiments of this application and the business level evaluation method of the intelligent inference cluster provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0166] This application also provides a computer program product 700, such as... Figure 7 As shown. This computer program product carries computer program 701, and the instructions included in the program code can be used to execute the steps of the business level assessment method for the intelligent inference cluster described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0167] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0168] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0169] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0170] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0171] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0172] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0173] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for evaluating the business level of an intelligent inference cluster, characterized in that, include: Based on the type of AI model loaded on the AI ​​accelerator card in the intelligent inference cluster to be evaluated, determine the maximum number of card sessions supported by the AI ​​accelerator card, where the type is used to characterize the sensitivity of the AI ​​model to latency. Based on the maximum number of sessions for the card and the business parameters of the intelligent inference cluster to be evaluated, calculate the number of online users used to characterize the business level of the intelligent inference cluster to be evaluated; The step of determining the maximum number of sessions supported by the AI ​​acceleration card based on the type of AI model loaded on the AI ​​acceleration card in the intelligent inference cluster to be evaluated includes: Obtain the available memory space of the AI ​​accelerator card, where the available memory space is the maximum available space in the AI ​​accelerator card that can cache intermediate results generated by the session during inference; Based on the available memory space and the first memory space, the number of first sessions is calculated, where the first memory space is the memory space required by each session during the inference process; When the type characterizes the artificial intelligence model as insensitive to latency, the first number of sessions is taken as the maximum number of sessions for the card; When the type characterizes the artificial intelligence model as being sensitive to latency, the second number of sessions is calculated based on the second memory space and the first memory space; the maximum number of sessions for the card is determined based on the first number of sessions and the second number of sessions; the second memory space characterizes the memory space that the artificial intelligence accelerator card allows the artificial intelligence model to use, and the second memory space is determined based on the latency parameter characterizing the response speed of the artificial intelligence model; The step of calculating the number of online users to characterize the service level of the intelligent inference cluster under evaluation, based on the maximum number of sessions per card and the service parameters of the intelligent inference cluster to be evaluated, includes: The maximum number of sessions in the cluster is determined based on the number of AI acceleration cards in the intelligent inference cluster to be evaluated and the maximum number of sessions per card. The average number of online users in the cluster is determined based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters. The number of online users is determined based on the average number of online users in the cluster and the duration parameter in the business parameters.

2. The method according to claim 1, characterized in that, The step of obtaining the available memory space of the artificial intelligence accelerator card includes: The available memory space is obtained by subtracting the memory space occupied by the operating system of the AI ​​accelerator card and the memory space occupied by the AI ​​model from the total memory space of the AI ​​accelerator card.

3. The method according to claim 1, characterized in that, Determining the maximum number of sessions for the card based on the first number of sessions and the second number of sessions includes: If the second session count is greater than or equal to zero, the minimum session count between the first session count and the second session count shall be taken as the maximum session count of the card; If the number of the second sessions is less than zero, the second memory space is expanded by updating the latency parameter; the second memory space is updated to the expanded second memory space, and the process returns to the step of calculating the number of the second sessions based on the second memory space and the first memory space, until the number of the second sessions is greater than or equal to zero, and the minimum number of sessions between the first number of sessions and the second number of sessions is taken as the maximum number of sessions for the card.

4. The method according to claim 1, characterized in that, The step of determining the average number of online users in the cluster based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters includes: The number of online users during the peak business period is calculated by dividing the maximum number of sessions in the cluster by the resource utilization rate in the resource usage parameters. The average number of online users in the cluster is calculated by dividing the number of online users during the peak business period by the peak concentration coefficient in the resource usage parameters.

5. The method according to claim 1, characterized in that, Determining the number of online users based on the average number of online users in the cluster and the duration parameter in the service parameters includes: The average total online time of online users is calculated by dividing the average number of online users in the cluster by the service duration of the intelligent inference cluster to be evaluated in the duration parameter. The number of online users is calculated using the average total online time of the online users and the average online time of each online user in the duration parameter.

6. A device for evaluating the business level of an intelligent inference cluster, characterized in that, include: The determination module is used to determine the maximum number of sessions supported by the AI ​​acceleration card based on the type of AI model loaded on the AI ​​acceleration card in the intelligent inference cluster to be evaluated, wherein the type is used to characterize the sensitivity of the AI ​​model to latency. The calculation module is used to calculate the number of online users that characterize the service level of the intelligent inference cluster to be evaluated, based on the maximum number of sessions of the card and the service parameters of the intelligent inference cluster to be evaluated. The determining module is used for: Obtain the available memory space of the AI ​​accelerator card, where the available memory space is the maximum available space in the AI ​​accelerator card that can cache intermediate results generated by the session during inference; Based on the available memory space and the first memory space, the number of first sessions is calculated, where the first memory space is the memory space required by each session during the inference process; When the type characterizes the artificial intelligence model as insensitive to latency, the first number of sessions is taken as the maximum number of sessions for the card; When the type characterizes the artificial intelligence model as being sensitive to latency, the second number of sessions is calculated based on the second memory space and the first memory space; the maximum number of sessions for the card is determined based on the first number of sessions and the second number of sessions; the second memory space characterizes the memory space that the artificial intelligence accelerator card allows the artificial intelligence model to use, and the second memory space is determined based on the latency parameter characterizing the response speed of the artificial intelligence model; The calculation module is used for: The maximum number of sessions in the cluster is determined based on the number of AI acceleration cards in the intelligent inference cluster to be evaluated and the maximum number of sessions per card. The average number of online users in the cluster is determined based on the maximum number of sessions in the cluster and the resource usage parameters in the service parameters. The number of online users is determined based on the average number of online users in the cluster and the duration parameter in the business parameters.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs any one of claims 1-5.

Images