Inference service management method, equipment, medium and computer program product

Abstract

The invention relates to the technical field of computers, and discloses an inference service management method and device, a medium and a computer program product, and the method comprises the steps: receiving a language model dialogue, and forwarding the dialogue to a corresponding inference service copy through a load balancer; when the language model reasoning service runs, carrying out statistics on throughput data of the reasoning service copy in a set period by utilizing a reasoning service scaling component; when the throughput data exceeds a set throughput threshold value, sending a capacity expansion notification signal to a reasoning service resource scheduler through a reasoning service expansion component, so that the reasoning service resource scheduler carries out reasoning service capacity expansion to obtain a capacity expansion copy; and utilizing a load balancer to configure the address of the expansion copy. In this way, the throughput in the language model service serves as the basis for service capacity expansion and load balancing selection, capacity expansion of the language model reasoning service is triggered when the throughput exceeds the threshold value, then resources are optimally distributed and used, the resource utilization rate can be increased, and user experience can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a reasoning service management method, device, medium and computer program product. Background Art

[0002] The reasoning service of the language model requires more graphics processing unit (GPU) resources, but a single host machine can usually only deploy one service, which may not be able to meet the conversation requests of a large number of users in high-concurrency scenarios on the Internet. At the same time, data centers generally have a pile of GPU server resources of different specifications. Based on the principle of asset reuse, language model services need to be run on GPU servers of different specifications. In related technical solutions, multiple service copies of the same application have the same processor memory specifications, and they have the same weights in elastic scaling and load balancing strategy management. However, load balancing strategies such as polling and minimum number of connections are forwarded based on the same load capacity provided by different services, which are not suitable for the language model service scenario; elastic scaling strategies are mostly based on simple resource utilization indicators, and usually do not take into account the performance indicators unique to language model reasoning services, and cannot achieve the expansion of language model reasoning services. Summary of the invention

[0003] The purpose of the present invention is to provide an inference service management method, device, medium and computer program product, which are applicable to language model service scenarios, can achieve the expansion of language model inference services, and then optimize the allocation of resources, improve resource utilization and improve user experience.

[0004] In order to solve the above technical problems, the present invention provides an inference service management method for an inference service platform including a load balancer, an inference service scaling component and an inference service resource scheduler, the method comprising: receiving a language model conversation, and forwarding the language model conversation to a corresponding inference service replica using the load balancer; When the language model inference service is running, the inference service scaling component is used to count the throughput data of the inference service replica within a set period; When the statistical throughput data exceeds the set throughput threshold, an expansion notification signal is sent to the reasoning service resource scheduler through the reasoning service scaling component, so that the reasoning service resource scheduler performs reasoning service expansion and obtains an expanded copy; The load balancer is used to configure the address of the expanded replica.

[0005] In a first aspect, in the above-mentioned inference service management method provided by the present invention, receiving a language model dialogue, and forwarding the language model dialogue to a corresponding inference service copy by using the load balancer, comprises: Receiving a first language model dialog carrying a target dialog identifier; Determining a corresponding reasoning service copy according to the target conversation identifier; The first language model dialog is forwarded to the determined inference service replica using the load balancer.

[0006] On the other hand, in the above-mentioned inference service management method provided by the present invention, before receiving the first language model dialogue carrying the target dialogue identifier, it also includes: Establishing an association between the conversation identifier and the inference service copy; Determining a corresponding reasoning service copy according to the target conversation identifier includes: According to the established association relationship between the dialog identifier and the inference service copy, in combination with the target dialog identifier, the inference service copy associated with the target dialog identifier is determined.

[0007] On the other hand, in the above-mentioned reasoning service management method provided by the present invention, establishing an association relationship between a conversation identifier and a reasoning service copy includes: When a language model conversation is first created, a corresponding conversation identifier is randomly generated; Utilize the load balancer to query the throughput real-time data of the inference service replicas, and select the inference service replica with the minimum throughput; The relationship between the randomly generated conversation identifier and the address of the selected inference service copy is recorded to establish an association relationship between the conversation identifier and the inference service copy.

[0008] On the other hand, the above-mentioned reasoning service management method provided by the present invention further includes: If the throughput of the determined inference service replica has reached the maximum value or the inference service replica does not exist, the load balancer is used to forward the first language model dialogue to other inference service replicas, and the association relationship between the dialogue identifier and the inference service replica is updated.

[0009] On the other hand, in the above-mentioned inference service management method provided by the present invention, receiving a language model dialogue, and forwarding the language model dialogue to a corresponding inference service copy by using the load balancer, comprises: receiving a second language model dialogue without a dialogue identifier; Selecting, using the load balancer, an inference service replica having a minimum ratio between real-time throughput and maximum throughput; The second language model dialog is forwarded to the selected inference service replica.

[0010] On the other hand, the above-mentioned reasoning service management method provided by the present invention further includes: When the statistical throughput data is zero, sending a capacity reduction notification signal to the reasoning service resource scheduler through the reasoning service scaling component, so that the reasoning service resource scheduler performs capacity reduction of the reasoning service and obtains a reduced copy; The load balancer is used to remove the address of the reduced-capacity replica.

[0011] On the other hand, in the above-mentioned reasoning service management method provided by the present invention, before receiving the language model dialogue, it also includes: When there are nodes of different specifications in the cluster, perform throughput performance tests on the language model service and collect statistics on the maximum throughput performance data of the language model service on nodes of different specifications. Based on the statistical maximum throughput performance data, different throughput thresholds are defined for different inference service copies of the same language model service.

[0012] On the other hand, in the above-mentioned reasoning service management method provided by the present invention, before receiving the language model dialogue, it also includes: The inference service resource scheduler is used to select resource specifications that meet set conditions to deploy the language model inference service according to cluster resource usage.

[0013] On the other hand, in the above-mentioned inference service management method provided by the present invention, the inference service resource scheduler is used to select resource specifications that meet the set conditions to deploy the language model inference service according to the cluster resource usage, including: Using the inference service resource scheduler to select nodes in the cluster that meet the processor memory requirements according to the configured processor and memory parameters; Obtain a resource specification list according to the inference service name, and select a resource specification that provides the maximum throughput from the resource specification list; Determine whether the selected nodes meet the selected resource specifications; If the selected nodes meet the selected resource specifications, an idle node among the selected nodes is selected to deploy the language model inference service; If the selected nodes do not meet the selected resource specifications, the corresponding nodes are selected according to the order of throughput size to deploy the language model inference service.

[0014] On the other hand, in the above-mentioned inference service management method provided by the present invention, after deploying the language model inference service, it also includes: Recording the address of the inference service replica, and configuring the address of the inference service replica in the load balancer; Forwarding the language model dialogue to a corresponding inference service replica using the load balancer includes: The load balancer is used to forward the language model dialogue to the corresponding inference service replica according to the address of the configured inference service replica.

[0015] On the other hand, in the above-mentioned reasoning service management method provided by the present invention, the throughput data of the reasoning service replica within a set period is counted by using the reasoning service scaling component, including: Calculate the total throughput of each language model conversation of each user in the inference service replica within a set time period using the inference service scaling component; The throughput sum and the set duration are used to calculate the average throughput of each language model dialogue for each user within the set duration.

[0016] In order to solve the above technical problems, the present invention further provides an inference service management device, the device comprising: Memory for storing computer programs; A processor is used to implement the steps of the above-mentioned reasoning service management method when executing the computer program.

[0017] In order to solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned reasoning service management method are implemented.

[0018] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above reasoning service management method when executed by a processor.

[0019] It can be seen from the above technical scheme that an inference service management method provided by the present invention is used for an inference service platform including a load balancer, an inference service scaling component and an inference service resource scheduler. The method includes: receiving a language model dialogue, and forwarding the language model dialogue to a corresponding inference service replica using a load balancer; when the language model inference service is running, using the inference service scaling component to count the throughput data of the inference service replica within a set period; when the counted throughput data exceeds the set throughput threshold, sending an expansion notification signal to the inference service resource scheduler through the inference service scaling component, so that the inference service resource scheduler expands the inference service and obtains an expanded replica; and using the load balancer to configure the address of the expanded replica.

[0020] The beneficial effect of the present invention is that the above-mentioned reasoning service management method provided by the present invention uses a load balancer to forward the language model dialogue to the corresponding reasoning service copy, uses the reasoning service scaling component to count the throughput data of the reasoning service copy within a set period, and when the data exceeds the set throughput threshold, sends an expansion notification signal to the reasoning service resource scheduler through the reasoning service scaling component, and the reasoning service resource scheduler expands the reasoning service to obtain the expanded copy, and uses the load balancer to configure the address of the expanded copy. This method is suitable for language model service scenarios, and can optimize the reasoning service scaling mechanism and load balancing strategy configuration in the language model service scenario, and use the throughput in the language model service as the basis for service expansion and load balancing selection. When the throughput exceeds the threshold, the expansion of the language model reasoning service is triggered, thereby optimizing the allocation and use of resources, improving resource utilization and improving user experience.

[0021] In addition, the present invention also provides corresponding reasoning service management equipment, non-volatile storage media and computer program products for the reasoning service management method, which have the same or corresponding technical features as the above-mentioned reasoning service management method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flowchart of the reasoning service management method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the architecture corresponding to the inference service management method provided in an embodiment of the present invention; Figure 3 A signaling interaction diagram corresponding to the reasoning service management method provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an inference service management device provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of an inference service management device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] With the development of artificial intelligence technology, language models play an increasingly important role in the field of natural language processing, and can handle complex language tasks such as text generation, translation, and conversation. Language model reasoning services usually require more graphics processing unit (GPU) resources. A single host can generally only deploy one language model service. In high-concurrency Internet scenarios, it is not enough to meet the conversation requests of a large number of users. At the same time, data centers usually have a pile of GPU server resources of different specifications. Based on the principle of asset reuse, language model services need to be run on GPU servers of different specifications. Therefore, how to effectively manage the resources required for reasoning services, expand the language model reasoning service in real time, and define a load balancing strategy suitable for the language model reasoning service has become a key issue in improving resource utilization and improving user experience.

[0025] In relevant technical solutions, central processing unit (CPU) and memory are usually used as resource specifications. Even if they run on servers of different specifications, multiple copies of the same application have the same CPU and memory specifications, and will provide the same maximum load capacity by default. They have the same weight in elastic scaling and load balancing strategy management. However, for language model services, their performance indicators are strongly dependent on the GPU computing power and video memory size of the running nodes, that is, the GPU model and number of the language model service running nodes. When different copies of the inference service run on different types of nodes, there are usually different throughput limits, that is, the hardware GPU resources determine the maximum load capacity of the inference service. However, related load balancing strategies such as polling and minimum number of connections are forwarded based on different services providing the same load capacity, which is not applicable to the language model service scenario.

[0026] In addition, with the development of containerization technology, the automatic increase and decrease of service copies is realized based on the elastic scaling mechanism, that is, the number of service instances is dynamically adjusted according to the actual load to adapt to changing business needs. However, elastic scaling strategies are mostly based on simple resource utilization indicators, such as CPU memory utilization, queries per second (QPS), etc., and usually do not take into account the performance indicators unique to language model inference services, such as throughput (token / s), language model services running on nodes of different specifications with different load capacities, etc. Simple CPU, memory, QPS, etc. cannot accurately represent the status of the inference service.

[0027] In addition, language model services usually use key-value cache (KVCache) technology to reuse the generated token key-value pair information to improve the reasoning speed. This requires that multiple rounds of conversations on the same topic use the same language model service copy to maximize the reuse of the key-value cache. The relevant load balancing strategy provides a certain session retention capability, such as load balancing strategies based on source IP and session retention. However, in the language model service scenario, a client usually has conversations on multiple topics. The source IP and session cannot distinguish between conversations on different topics, and there is a problem of session expiration, which is not suitable for the language model service scenario.

[0028] In order to solve the above problems, the present invention provides an inference service management method, which can be applicable to language model inference service scenarios and optimizes the scaling strategy and load balancing strategy configuration in the language model service scenario.

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A flowchart of the inference service management method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method is used for an inference service platform including a load balancer, an inference service scaling component and an inference service resource scheduler, including: S101, receiving a language model dialogue, and using a load balancer to forward the language model dialogue to a corresponding inference service replica.

[0031] It should be noted that a language model refers to a model that learns from a large amount of text to master many language features such as language structure, vocabulary usage, sentence composition rules, etc. In the present invention, the language model can select a large language model (LLM) or other types of language models, which are not limited here. The language model inference service is an online inference service based on language model deployment, which supports multi-copy deployment. The language model inference service can be provided to Internet users in the form of multiple rounds of dialogue. For each round of dialogue, the language model service generates a long sequence, that is, a streaming output of multiple tokens. For the end user, the output of the inference service is output in the form of each Chinese character or word.

[0032] Figure 2 The schematic diagram of the architecture corresponding to the inference service management method provided in the embodiment of the present invention. Figure 2 As shown, the user dialogue request can be passed to the network service (such as Web service) through the proxy service (such as Nginx proxy), and then passed to the load balancer by the network service. The load balancer is a device used to distribute network traffic or workload between computing resources. When executing step S101, the load balancer can receive the language model dialogue, and then forward the language model dialogue to the corresponding inference service replica.

[0033] S102: When the language model inference service is running, the inference service scaling component is used to count the throughput data of the inference service replica within a set period.

[0034] It should be noted that the inference service scaling component is a tool for dynamically adjusting inference service resources. The inference service scaling component of the present invention can count the throughput data of the inference service replica within a set period (i.e., the number of tokens generated per second) to further realize the dynamic adjustment of computing resources. The setting period can be determined according to actual conditions and is not limited here. Throughput is a fine-grained performance indicator in the language model service. The present invention uses the throughput indicator as the basis for subsequent service expansion.

[0035] When executing step S102, when the language model inference service is running, the inference service scaling component can be used to count the throughput data of multiple inference service copies within a set period.

[0036] S103. When the statistical throughput data exceeds the set throughput threshold, an expansion notification signal is sent to the inference service resource scheduler through the inference service scaling component, so that the inference service resource scheduler expands the inference service and obtains an expanded copy.

[0037] It should be noted that the inference service resource scheduler is a device for managing and allocating various resources required for inference services. The inference service resource scheduler of the present invention can realize the expansion of inference services and obtain expansion copies to further process inference tasks.

[0038] Before executing step S103, the present invention may predefine a set throughput threshold. Different set throughput thresholds are defined for different replicas of the same language model inference service, and different replicas provide differentiated load capacities.

[0039] When executing step S103, it is determined whether the throughput data counted by the inference service scaling component exceeds the corresponding pre-defined throughput threshold; if so, it means that the processing speed of the inference service copy has reached a certain threshold, and an expansion notification signal is sent to the inference service resource scheduler through the inference service scaling component, so that the inference service resource scheduler expands the inference service and obtains an expanded copy.

[0040] S104. Use the load balancer to configure the address of the expanded replica.

[0041] like Figure 2 As shown, after the inference service resource scheduler obtains the expanded replica, the load balancer can be used to configure the address of the expanded replica, which can be assigned to different node specifications, such as the first node, the second node, the third node, the fourth node, and so on.

[0042] In the above-mentioned reasoning service management method provided by the embodiment of the present invention, a load balancer is used to forward the language model dialogue to the corresponding reasoning service replica, and the reasoning service scaling component is used to count the throughput data of the reasoning service replica within a set period. When the data exceeds the set throughput threshold, the reasoning service scaling component sends an expansion notification signal to the reasoning service resource scheduler, and the reasoning service resource scheduler expands the reasoning service to obtain the expanded replica, and the address of the expanded replica is configured by the load balancer. In this way, the reasoning service scaling mechanism and load balancing strategy configuration in the language model service scenario can be optimized, and the throughput in the language model service can be used as the basis for service expansion and load balancing selection. When the throughput exceeds the threshold, the expansion of the language model reasoning service is triggered, thereby optimizing the allocation and use of resources, improving resource utilization and improving user experience.

[0043] Furthermore, in a specific implementation, in the above-mentioned inference service management method provided in an embodiment of the present invention, step S101 receives a language model dialogue, and uses a load balancer to forward the language model dialogue to a corresponding inference service replica, which may specifically include: receiving a first language model dialogue carrying a target dialogue identifier; determining a corresponding inference service replica according to the target dialogue identifier; and using a load balancer to forward the first language model dialogue to the determined inference service replica.

[0044] In practice, the present invention introduces a language model conversation identifier (Identity, ID) for load balancing strategy selection, that is, the conversation identifier is used as a consideration for service elastic scaling and load balancing strategy. Different language model conversations have different conversation identifiers.

[0045] Figure 3 The signaling interaction diagram corresponding to the inference service management method provided in the embodiment of the present invention is shown in FIG. Figure 3As shown, the corresponding inference service replica can be determined through the language model conversation identifier, and the load balancer can forward the received language model conversation to the determined inference service replica, thereby improving resource utilization and application operation efficiency.

[0046] Furthermore, in a specific implementation, in the above-mentioned inference service management method provided by the embodiment of the present invention, before receiving the first language model dialogue carrying the target dialogue identifier, it may also include: establishing an association relationship between the dialogue identifier and the inference service replica. That is, the language model dialogue identifier is bound to a certain inference service replica for reusing the key-value cache information in the historical dialogue to establish an association relationship between the dialogue identifier and the inference service replica, and the load balancer may save the association relationship between the dialogue identifier and the inference service replica.

[0047] Correspondingly, determining the corresponding inference service copy according to the target conversation identifier in the above steps may specifically include: determining the inference service copy associated with the target conversation identifier in combination with the target conversation identifier according to the established association relationship between the conversation identifier and the inference service copy.

[0048] In implementation, after receiving the first language model dialog carrying the target dialog identifier, the target dialog identifier is parsed, and the inference service copy bound to the target dialog identifier can be determined by using the established binding relationship between the dialog identifier and the inference service copy.

[0049] Furthermore, in specific implementation, the above steps establish an association relationship between the conversation identifier and the inference service copy, which may specifically include: when the language model conversation is first created, randomly generating a corresponding conversation identifier; using the load balancer to query the real-time throughput data of the inference service copy, and selecting the inference service copy with the smallest throughput; recording the relationship between the randomly generated conversation identifier and the address of the selected inference service copy to establish an association relationship between the conversation identifier and the inference service copy.

[0050] In implementation, when the corresponding language model dialogue is first created, the association relationship between the dialogue identifier and the inference service copy can be established. Specifically, the corresponding dialogue identifier is randomly generated first. At this time, the load balancer does not have the inference service copy information associated with the dialogue identifier. Then, the load balancer is used to query the real-time throughput data of each inference service copy and select the inference service copy with the smallest throughput. Finally, the load balancer records the association relationship between the randomly generated dialogue identifier and the address of the selected inference service copy. When the dialogue is requested again, the dialogue can be forwarded to the inference service copy with historical dialogue information (i.e., previously used) first, which can improve the inference speed.

[0051] Furthermore, in a specific implementation, the above-mentioned inference service management method provided in an embodiment of the present invention may also include: if the throughput of the determined inference service replica has reached the maximum value or the inference service replica does not exist, using a load balancer to forward the first language model dialogue to other inference service replicas, and updating the association between the dialogue identifier and the inference service replica.

[0052] In implementation, when the throughput of the inference service replica has reached the maximum value, or the inference service replica itself does not exist, the load balancer can be used to forward the first language model dialogue to other inference service replicas, and update the association between the dialogue identifier and the inference service replica.

[0053] In addition, in the specific implementation, in the above-mentioned inference service management method provided in the embodiment of the present invention, step S101 receives the language model dialogue, and uses the load balancer to forward the language model dialogue to the corresponding inference service replica, which can also specifically include: receiving a second language model dialogue that does not carry a dialogue identifier; using the load balancer to select the inference service replica with the smallest ratio between the real-time throughput and the maximum throughput; forwarding the second language model dialogue to the selected inference service replica.

[0054] In implementation, Figure 3 As shown, after receiving the second language model dialogue that does not carry the dialogue identifier, the load balancer can be used to select the inference service replica with the smallest ratio between the real-time throughput and the maximum throughput, and forward the second language model dialogue to the selected inference service replica. That is, when there are multiple language model services of inference service replicas in the cluster, and the received language model dialogue does not carry the dialogue identifier, the received dialogue can be preferentially forwarded to the idle inference service replica with a larger throughput.

[0055] The present invention can use the following formula to calculate the real-time throughput and the ratio between the real-time throughput and the maximum throughput: ; in, represents the j-th conversation throughput of user i, t represents the set duration, represents the real-time throughput, Indicates the maximum throughput of the language model on nodes of different specifications. Indicates the ratio between real-time throughput and maximum throughput.

[0056] Furthermore, in a specific implementation, the above-mentioned inference service management method provided in an embodiment of the present invention may also include: when the statistical throughput data is zero, sending a reduction notification signal to the inference service resource scheduler through the inference service scaling component, so that the inference service resource scheduler performs inference service reduction and obtains a reduced copy; and using the load balancer to move out the address of the reduced copy.

[0057] In implementation, Figure 3 As shown, when the throughput data of the set period statistics is zero, the inference service scaling component can send a scaling notification signal to the inference service resource scheduler, so that the inference service resource scheduler can scale down the inference service and obtain a scaled-down copy, thereby ensuring that the inference service can run efficiently under different load conditions.

[0058] It should be pointed out that the scaling-down mechanism generally performs a scaling-down operation when the traffic is lower than a certain threshold, but the local key-value cache needs to be used to optimize the reasoning performance when the language model is running. The same conversation can only achieve the best performance by using the same reasoning service copy. Therefore, the scaling-down operation of the language model reasoning service needs to be triggered when there is no traffic. The present invention can select the setting period as 30 minutes. When no request is received within 30 minutes, the reasoning service copy is scaled down, but the reasoning service needs to retain at least one reasoning service copy.

[0059] Furthermore, in a specific implementation, in the above-mentioned inference service management method provided in an embodiment of the present invention, before receiving a language model dialogue, it may also include: when there are nodes of various specifications in a cluster, performing a throughput performance test on the language model service, and counting the maximum throughput performance data of the language model service at nodes of different specifications; based on the counted maximum throughput performance data, defining different set throughput thresholds for different inference service copies of the same language model service.

[0060] In implementation, when there are GPU nodes of various specifications in the artificial intelligence cluster, the present invention can perform throughput performance testing on the language model service to be deployed, and count the maximum throughput performance data of the language model service on GPU nodes of different specifications, wherein different set throughput thresholds are defined for different inference service copies of the same language model service, thereby optimizing the allocation and use of GPU resources.

[0061] It should be pointed out that in the language model service scenario, based on the principle of data center asset reuse, different service copies of the same language model service are allowed to run on GPU nodes of different specifications. The maximum throughput of the language model service on GPU nodes of different specifications is used as the elastic scaling and load balancing indicator. That is, the same service has different expansion indicators and request forwarding indicators when it runs on nodes of different specifications.

[0062] The following is an example of testing the throughput of the language model service to be deployed with different specifications and different numbers of accelerator cards. Each specification can meet the minimum resource requirements of the inference task. For example, the throughput on 8 A100 cards is , the throughput on 8 H100 cards is , the throughput on 4 A100 cards is .

[0063] The obtained performance test data is configured to the inference service platform, and the resource specifications used to run the language model inference service are defined, for example, several tested specifications are defined: 8 H100s, 8 A100s, and 4 A100s. The inference service resource scheduler can select appropriate resource specifications to deploy the language model inference service according to the cluster resource usage. That is, the present invention can select resource specifications that meet the set conditions to deploy the language model inference service according to the cluster resource usage through the inference service resource scheduler to ensure the stability of the inference operation.

[0064] The present invention can use the data in Table 1 for resource scheduling and access policy management.

[0065] Table 1 Performance test data

[0066] In the specific implementation, the above steps use the inference service resource scheduler to select resource specifications that meet the set conditions to deploy the language model inference service according to the cluster resource usage, which can specifically include: using the inference service resource scheduler to filter nodes that meet the processor memory requirements in the cluster according to the configured processor and memory parameters; obtaining a resource specification list according to the inference service name, and selecting the resource specification that provides the maximum throughput from the resource specification list; judging whether the filtered nodes meet the selected resource specifications; if the filtered nodes meet the selected resource specifications, selecting idle nodes among the filtered nodes to deploy the language model inference service; if the filtered nodes do not meet the selected resource specifications, selecting corresponding nodes according to the throughput size to deploy the language model inference service.

[0067] It should be noted that when deploying a language model service on the inference service platform, it is necessary to set the CPU and memory parameters, but the number and model of GPU cards are not configured. The inference service resource scheduler automatically selects the appropriate number of GPU cards. The configuration of CPU and memory is used to prevent the host node from being abnormal due to memory abuse and CPU abuse in the inference service. The inference service platform performs scheduling according to the cluster resources and the inference service resource specifications. The scheduling process may include: first, using the inference service scheduler to filter nodes that meet the CPU memory requirements in the cluster according to the configured CPU and memory parameters. Then, according to the inference service name, the available resource specifications are obtained, and the resource specifications that can provide the maximum throughput are selected for resource scheduling, that is, the appropriate resource specifications are automatically selected for scheduling based on the maximum throughput of the language model service at nodes of different specifications. Based on this strategy, the number of scaling times of the inference service can be reduced, and more service accesses can be directed to the same inference service copy. In extreme cases, using RadixAttention (a method for optimizing language model inference performance) can reuse the key-value cache to cache and reduce redundant calculations, thereby improving the performance of the inference service.

[0068] When the idle resources of the cluster cannot meet the selected resource specifications, the resource specifications that provide less throughput are selected for scheduling. Taking LLM-mode-2 in Table 1 as an example, when The resource specification selection order of the present invention is 8 H100s, 8 A100s, and 4 A100s, that is, the corresponding nodes are selected in the order of throughput from large to small. If the resource specification with the smallest throughput is used for scheduling and still no node can be selected, the task scheduling fails and needs to wait for other tasks to release resources and schedule again.

[0069] Furthermore, in a specific implementation, in the above-mentioned inference service management method provided in an embodiment of the present invention, after deploying the language model inference service, it may also include: recording the address of the inference service copy, and configuring the address of the inference service copy in the load balancer.

[0070] Accordingly, forwarding the language model dialogue to the corresponding inference service replica using the load balancer may specifically include: forwarding the language model dialogue to the corresponding inference service replica according to the configured address of the inference service replica using the load balancer.

[0071] In implementation, when the inference service is deployed, the IP address of the inference service replica can be recorded. , Maximum Throughput , and configure the replica IP address in the platform's built-in load balancer. The platform's built-in load balancing service automatically configures forwarding, forwards external requests to the node where the inference service is located, and provides inference services to the outside world.

[0072] Furthermore, in a specific implementation, in the above-mentioned inference service management method provided in an embodiment of the present invention, step S102 uses the inference service scaling component to count the throughput data of the inference service replica within a set period, which may specifically include: using the inference service scaling component to calculate the sum of the throughput of each language model dialogue of each user in the inference service replica within the set duration; using the sum of the throughput and the set duration, calculating the average throughput of each language model dialogue of each user within the set duration.

[0073] In implementation, since each inference service will provide multiple rounds of dialogue services for different users, the throughput of each dialogue of each user is usually calculated to obtain the processing speed of the token of each dialogue. The statistical unit is token / s. When performing throughput statistics on the copy, it is necessary to summarize the throughput of each dialogue. The present invention can count the total throughput of accessing the inference service copy within a set time length (such as 1 minute), calculate the average throughput within the time period, and trigger the expansion of the inference service when the average reaches the set threshold. The present invention can specifically trigger the expansion when the real-time throughput reaches 90% of the maximum throughput of the inference service copy, and shrink the inference service copy when there is no request within a certain period.

[0074] Take LLM-mode-2 in Table 1 deployed on 8-card H100 as an example: calculate the average throughput of each user and each conversation within the set duration (such as 1 minute), and trigger the expansion operation when it is greater than 90% of the maximum throughput of the replica. The reason why 90% is chosen here is that the amount of GPU resources occupied by a single request is uncertain due to the difference in length between user requests and model responses, as well as the randomness of the number of tokens generated by the model in the prompt and generate stages, so 10% of the idle capacity needs to be reserved. The specific formula is as follows: ; in, represents the j-th conversation throughput of user i, and t represents the set duration, which can be set to 60s.

[0075] It should be added that the present invention can also be used for inference services running across GPUs, neural processing units (NPUs) and other types of acceleration cards, that is, some copies run on GPU nodes and some copies run on other types of acceleration card nodes, which will not be elaborated here.

[0076] In the above embodiments, the reasoning service management method is described in detail, and the present invention also provides corresponding embodiments of the reasoning service management device and the reasoning service management equipment. It should be noted that the present invention describes the embodiments of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0077] Figure 4 A schematic diagram of the structure of the inference service management device provided by an embodiment of the present invention. Based on the perspective of functional modules, the device of this embodiment includes: A replica forwarding module 10, configured to receive a language model dialogue and forward the language model dialogue to a corresponding inference service replica; The data statistics module 11 is used to collect the throughput data of the inference service replica within a set period when the language model inference service is running; The replica expansion module 12 is used to expand the inference service capacity and obtain an expanded replica when the statistical throughput data exceeds the set throughput threshold; The address configuration module 13 is used to configure the address of the expansion replica.

[0078] In the above-mentioned inference service management device provided by the embodiment of the present invention, the language model dialogue can be forwarded to the corresponding inference service copy through the interaction of the above-mentioned four modules, and the throughput data of the inference service copy within the set period can be counted. When the data exceeds the set throughput threshold, the inference service is expanded, the expanded copy is obtained, and the address of the expanded copy is configured. In this way, the inference service scaling mechanism and load balancing strategy configuration in the language model service scenario can be optimized, and the throughput in the language model service can be used as the basis for service expansion and load balancing selection. When the throughput exceeds the threshold, the expansion of the language model inference service is triggered, thereby optimizing the allocation and use of resources, improving resource utilization and improving user experience.

[0079] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. And it has the same beneficial effects as the above-mentioned reasoning service management method.

[0080] Figure 5 This is a schematic diagram of the structure of the inference service management device provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 5 As shown, the reasoning service management device includes: A memory 20, used for storing computer programs; The processor 21 is used to implement the steps of the reasoning service management method mentioned in the above embodiment when executing the computer program.

[0081] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0082] The memory 20 may include one or more non-volatile storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the reasoning service management method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the above-mentioned reasoning service management method, etc.

[0083] In some embodiments, the reasoning service management device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26. Those skilled in the art will appreciate that Figure 5 The structure shown in does not constitute a limitation on the inference service management device, and may include more or fewer components than shown in the figure. The inference service management device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: the inference service management method, the effect is the same as above.

[0084] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0085] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code. The non-volatile storage medium provided by the present invention can implement the above-mentioned reasoning service management method, and the effect is the same as above.

[0086] Finally, the present invention also provides an embodiment corresponding to a computer program product. The computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps recorded in the above-mentioned inference service management method embodiment are implemented. The computer program product provided by the present invention can implement the above-mentioned inference service management method, and the effect is the same as above.

[0087] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0088] The above is a detailed introduction to the reasoning service management method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for managing an inference service, characterized in that: For an inference service platform including a load balancer, an inference service scaling component and an inference service resource scheduler, the method comprises: receiving a language model conversation, and forwarding the language model conversation to a corresponding inference service replica using the load balancer; When the language model inference service is running, the inference service scaling component is used to count the throughput data of the inference service replica within a set period; When the statistical throughput data exceeds the set throughput threshold, an expansion notification signal is sent to the reasoning service resource scheduler through the reasoning service scaling component, so that the reasoning service resource scheduler performs reasoning service expansion and obtains an expanded copy; The load balancer is used to configure the address of the expanded replica.

2. The inference service management method according to claim 1, characterized in that: Receiving a language model conversation, and forwarding the language model conversation to a corresponding inference service replica using the load balancer, comprising: Receiving a first language model dialog carrying a target dialog identifier; Determine a corresponding reasoning service copy according to the target conversation identifier; The first language model dialog is forwarded to the determined inference service replica using the load balancer.

3. The inference service management method according to claim 2, characterized in that: Before receiving the first language model dialogue carrying the target dialogue identifier, the method further includes: Establishing an association between the conversation identifier and the inference service copy; Determining a corresponding reasoning service copy according to the target conversation identifier includes: According to the established association relationship between the dialog identifier and the inference service copy, in combination with the target dialog identifier, the inference service copy associated with the target dialog identifier is determined.

4. The inference service management method according to claim 3, characterized in that: Establish an association between the conversation identifier and the inference service copy, including: When a language model conversation is first created, a corresponding conversation identifier is randomly generated; Utilize the load balancer to query the throughput real-time data of the inference service replicas, and select the inference service replica with the minimum throughput; The relationship between the randomly generated conversation identifier and the address of the selected inference service copy is recorded to establish an association relationship between the conversation identifier and the inference service copy.

5. The inference service management method according to claim 3, characterized in that: Also includes: If the throughput of the determined inference service replica has reached the maximum value or the inference service replica does not exist, the load balancer is used to forward the first language model dialogue to other inference service replicas, and the association relationship between the dialogue identifier and the inference service replica is updated.

6. The inference service management method according to claim 1, characterized in that: Receiving a language model conversation, and forwarding the language model conversation to a corresponding inference service replica using the load balancer, comprising: receiving a second language model dialogue without a dialogue identifier; Selecting, using the load balancer, an inference service replica having a minimum ratio between real-time throughput and maximum throughput; The second language model dialog is forwarded to the selected inference service replica.

7. The inference service management method according to claim 1, characterized in that: Also includes: When the statistical throughput data is zero, sending a capacity reduction notification signal to the reasoning service resource scheduler through the reasoning service scaling component, so that the reasoning service resource scheduler performs capacity reduction of the reasoning service and obtains a reduced copy; The load balancer is used to remove the address of the reduced-capacity replica.

8. The inference service management method according to claim 1, characterized in that: Before receiving the language model dialogue, it also includes: When there are nodes of different specifications in the cluster, perform throughput performance tests on the language model service and collect statistics on the maximum throughput performance data of the language model service on nodes of different specifications. Based on the statistical maximum throughput performance data, different throughput thresholds are defined for different inference service copies of the same language model service.

9. The inference service management method according to claim 1, characterized in that: Before receiving the language model dialogue, it also includes: The inference service resource scheduler is used to select resource specifications that meet set conditions to deploy the language model inference service according to cluster resource usage.

10. The inference service management method according to claim 9, characterized in that: The inference service resource scheduler is used to select resource specifications that meet set conditions to deploy the language model inference service according to cluster resource usage, including: Using the inference service resource scheduler to select nodes in the cluster that meet the processor memory requirements according to the configured processor and memory parameters; Obtain a resource specification list according to the inference service name, and select a resource specification that provides the maximum throughput from the resource specification list; Determine whether the selected nodes meet the selected resource specifications; If the selected nodes meet the selected resource specifications, an idle node among the selected nodes is selected to deploy the language model inference service; If the selected nodes do not meet the selected resource specifications, the corresponding nodes are selected according to the order of throughput size to deploy the language model inference service.

11. The inference service management method according to claim 10, characterized in that: After deploying the language model inference service, the following is also included: Recording the address of the inference service replica, and configuring the address of the inference service replica in the load balancer; Forwarding the language model dialogue to a corresponding inference service replica using the load balancer includes: The load balancer is used to forward the language model dialogue to the corresponding inference service replica according to the address of the configured inference service replica.

12. The inference service management method according to claim 1, characterized in that: The inference service scaling component is used to collect statistics on the throughput data of the inference service replica within a set period, including: Calculate the total throughput of each language model conversation of each user in the inference service replica within a set time period using the inference service scaling component; The throughput sum and the set duration are used to calculate the average throughput of each language model dialogue for each user within the set duration.

13. An inference service management device, characterized in that: The device comprises: Memory for storing computer programs; A processor, configured to implement the steps of the inference service management method according to any one of claims 1 to 12 when executing the computer program.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the inference service management method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the reasoning service management method according to any one of claims 1 to 12 are implemented.

Images