Load balancing scheduling method and device for model reasoning and electronic equipment

Abstract

The invention discloses a load balancing scheduling method and device for model reasoning and electronic equipment. The method comprises the following steps: receiving a current request of a user; estimating the processing time of the current request according to the parameters of the historical request, the processing time of the historical request and the parameters of the current request; judging whether idle service instances exist or not according to the service instance state list; wherein the service instance state list comprises service instance numbers and states and priorities thereof; if no idle service instance exists, the current request and the corresponding parameters are placed in a request queue; and if the idle service instance exists, distributing the request selected from the request queue according to the first index or the current request to the service instance selected according to the second index, and updating the state of the service instance in the service instance state list. According to the method, load balancing is carried out on the request by recording and managing the processing state of the back-end service instance, and the method is more suitable for model reasoning service.

Description

Technical Field

[0001] The present invention relates to the technical field of load balancing scheduling for model reasoning, and in particular to a load balancing scheduling method, device and electronic equipment for model reasoning. Background Art

[0002] Load balancing is a technology for distributing computing tasks or network traffic to optimize resource usage, improve response speed, and ensure system availability. It distributes requests or data flows evenly to multiple servers or resources to avoid overloading a single device while ensuring the overall performance and stability of the system.

[0003] At present, there are mainly the following load balancing algorithms: Round Robin, Least Connections, Source IP Hash, Weighted Round Robin, and Weighted Least Connections. These load balancing algorithms only distribute user requests to different backend services according to certain rules and strategies, without considering the characteristics of model reasoning services (generative language, generative video, and video understanding, etc.): the processing time required is relatively long (minutes, or even more than 10 minutes); most of these models are user-exclusive and can only provide services to one user at a time; due to different user request parameters, the processing time of different requests varies greatly. Therefore, the use of the above load balancing algorithms can easily cause pressure on a single backend service request and long waiting times for users, which cannot meet the model reasoning scenario. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0005] A first aspect of the present invention provides a load balancing scheduling method for model reasoning, comprising: Receive the user's current request; Estimate the processing time of the current request based on the parameters and processing time of historical requests and the parameters of the current request; According to the service instance status list, determine whether there is an idle service instance; wherein the service instance status list includes the service instance number, status and priority; If there is no idle service instance, the current request and its corresponding parameters are placed in the request queue; If there is an idle service instance, the request selected from the request queue according to the first indicator or the current request will be assigned to the service instance selected according to the second indicator, and the status of the service instance will be updated in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

[0006] Preferably, the first indicator also includes: user priority, user status, number of user historical requests and / or length of time the user waits in line.

[0007] Preferably, the method further comprises: The remaining processing time is evaluated based on the estimated processing time and the used processing time of the request being processed by the service instance. Before the service instance processing is completed, the next request is assigned to the waiting queue of the service instance in advance.

[0008] Preferably, the method further comprises: maintaining the survival status of the service instance through a heartbeat check, and if a service instance with a status of "processing" is found to be offline, placing the request being processed into a request queue.

[0009] Preferably, the request includes a synchronous blocking request and an asynchronous non-blocking request; the synchronous blocking request means that the client waits after the request is issued until the service instance completes processing and returns the result; the asynchronous non-blocking request means that the client immediately returns the processing estimated time, waiting estimated time and overall response estimated time after the request is issued, and pushes the processing result to the client after the service instance completes processing.

[0010] Preferably, the method further comprises: if there is no idle service instance, returning a prompt of rejecting the service.

[0011] Preferably, the method further comprises: Set the length of the request queue. If the length is reached, a denial of service prompt is returned. If the length is not reached, the current request and its corresponding parameters are placed in the request queue.

[0012] A second aspect of the present invention provides a load balancing scheduling device for model reasoning, comprising: A request receiving module, used to receive the current request of the user; A processing time estimation module is used to estimate the processing time of the current request based on the parameters of historical requests and the processing time of historical requests and the parameters of the current request; A judgment module, used to judge whether there is an idle service instance according to the service instance status list; wherein the service instance status list includes the service instance number, status and priority; A first allocation module, configured to place the current request and its corresponding parameters in a request queue if there is no idle service instance; The second allocation module is used to allocate the request selected from the request queue according to the first indicator or the current request to the service instance selected according to the second indicator if there is an idle service instance, and update the status of the service instance in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

[0013] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the load balancing scheduling method for model reasoning as described in the first aspect.

[0014] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the load balancing scheduling method for model reasoning as described in the first aspect.

[0015] The beneficial effect of the present invention is that the load balancing scheduling method provided by the present invention not only distributes the user's request to different backend services according to certain rules and strategies, but also load balances the request by recording and managing the processing status of the backend service instance, which is more suitable for the characteristics of the model reasoning service. Therefore, the technical solution provided by the present invention is more suitable for load balancing scheduling of model reasoning, and when the backend service resources are insufficient, a more fair and higher resource utilization load balancing strategy can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the flow of the load balancing scheduling method for model reasoning according to the present invention; Figure 2 This is a functional structure diagram of the load balancing scheduling device for model reasoning described in the present invention. DETAILED DESCRIPTION

[0017] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0019] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0020] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0021] The display screen is used to display the user interface of each application.

[0022] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0023] Embodiment 1 like Figure 1 As shown, an embodiment of the present invention provides a load balancing scheduling method for model reasoning, including: S101, receiving a current request from a user; S102, estimating the processing time of the current request based on the parameters of the historical requests and the processing time of the historical requests and the parameters of the current request; S103, judging whether there is an idle service instance according to the service instance status list; wherein the service instance status list includes the service instance number, status and priority; S104, if there is no idle service instance, the current request and its corresponding parameters are placed in the request queue; S105, if there is an idle service instance, the request selected from the request queue according to the first indicator or the current request will be allocated to the service instance selected according to the second indicator, and the status of the service instance will be updated in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

[0024] Among them, S104 and S105 are in no particular order.

[0025] The service instance status list is shown in Table 1.

[0026] Table 1 Service instance status list Service instance number state Priority 1 Processing, no queued requests 1 2 Processing, there are queued requests 1 3 idle 2 4 Offline 3 As can be seen from Table 1, the status of a service instance can include "Processing", "Idle" and "Offline". "Processing" means that the service instance is working and processing requests. "Idle" means that the service instance is idle and is not processing any requests. "Offline" means that the service instance is disconnected and cannot process requests. The number of a service instance is unique, that is, each service instance has a different number. The priority of a service instance can be set based on factors such as its resources and processing power. The priority of the service instance with the best resources and the strongest processing power can be set to the highest, represented by the number "1". As the priority decreases, it is represented by increasing numbers.

[0027] In the embodiment of the present invention, since the request is assigned to the service instance, when the service instance processes the request, the status of the service instance will be updated in the service instance status list, that is, the status of the service instance will be recorded, so the information of the processed requests (historical requests) will be recorded, such as the parameters of the historical requests and the processing time of the historical requests, etc., as shown in Table 2.

[0028] Table 2 Parameters of historical requests and processing time of historical requests parameter Processing time {"prompt":"Running deer","video_time":"3 seconds"} 620 seconds {"prompt":"Flying excavator","video_time":"5 seconds"} 915 seconds In a preferred embodiment of the present invention, the following method can be used to estimate the processing time of the current request based on the parameters of historical requests and the processing time of historical requests and the parameters of the current request: for example, a model can be trained based on the parameters of historical requests and the processing time of historical requests, and then the trained model and the parameters of the current request are used to predict the processing time of the current request; the processing rules can also be manually summarized and the rules can be encoded to achieve the prediction of the processing time.

[0029] In the embodiment of the present invention, whether there is an idle service instance can be determined according to the "status" in the service instance status list. When the "status" of a service instance in the list is "idle", it means that there is an idle service instance. Otherwise, there is no idle service instance.

[0030] In one embodiment of the present invention, as the number of received requests increases, when all service instances are in the "processing" state, the newly received request can enter the request queue and wait for processing. In the specific application process, it is also possible to determine whether a new request should enter the request queue by setting the length of the request queue. For example, when the new request reaches the set length of the request queue, a prompt of denying service can be returned; when the new request does not reach the set length of the request queue, the current request and its corresponding parameters can be placed in the request queue.

[0031] In another embodiment of the present invention, as the number of received requests continues to increase, when all service instances are in the "processing" state, that is, when there are no idle service instances, a prompt of rejecting the service can be directly returned without entering the request queue to wait.

[0032] In an embodiment of the present invention, when there is an idle service instance, the request selected from the request queue according to the first indicator or the current request can be allocated to the service instance selected according to the second indicator. Wherein, if there is an idle service instance after receiving the current request, the current request is directly allocated to the service instance. If no new request is received, a request can be selected from the request queue and allocated to the service instance. If multiple service instances are idle, they can be selected according to priority, and the service instance with a high priority is preferentially selected. Wherein, when selecting a request from the request queue, it can be selected according to the first indicator, and the first indicator includes the estimated processing time of the request, the user priority, the user's status, the number of historical requests of the user and / or the length of time the user waits in line. The selection can be made based on a comprehensive consideration of several indicators, or it can be made based on one indicator. In the actual application process, a corresponding selection strategy can be formulated according to the actual situation.

[0033] In an embodiment of the present invention, a service instance may be selected according to a second indicator. The second indicator may include a priority of the service instance. The priority of the service instance may be determined according to the following method: for example, the original priority of the service instance and the estimated remaining processing time may be weighted, and the weighted value may be re-prioritized; or the priority may be arranged only according to the estimated remaining processing time of the service instance.

[0034] In the embodiment of the present invention, the status of the service instance is updated in the service instance status list so as to allocate and match a more reasonable service instance for subsequent requests.

[0035] In a preferred embodiment of the present invention, the remaining processing time can be evaluated based on the estimated processing time and the used processing time of the request being processed by the service instance, and the next request can be allocated to the waiting queue of the service instance in advance before the service instance processing is completed. In this way, before the service instance processing is completed, the request can be queued in the waiting queue of the service instance in advance, thereby improving the utilization rate of resources.

[0036] In a preferred embodiment of the present invention, the method may further include: maintaining the survival status of the service instance through heartbeat checks, and if a service instance with a status of "processing" is found to be offline, placing the request being processed into a request queue.

[0037] In an embodiment of the present invention, the request may include a synchronous blocking request and an asynchronous non-blocking request; the synchronous blocking request is that the client waits after the request is issued until the service instance completes processing and returns the result; the asynchronous non-blocking request is that the client immediately returns the processing estimated time, waiting estimated time and overall response estimated time after the request is issued, and pushes the processing result to the client after the service instance is processed.

[0038] Embodiment 2 like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, an embodiment of the present invention also provides a load balancing scheduling device for model reasoning, including: The request receiving module 201 is used to receive the current request of the user; A processing time estimation module 202, used to estimate the processing time of the current request based on the parameters of the historical requests and the processing time of the historical requests and the parameters of the current request; The judgment module 203 is used to judge whether there is an idle service instance according to the service instance status list; wherein the service instance status list includes the service instance number, its status and priority; A first allocation module 204, configured to place the current request and its corresponding parameters in a request queue if there is no idle service instance; The second allocation module 205 is used to allocate the request selected from the request queue according to the first indicator or the current request to the service instance selected according to the second indicator if there is an idle service instance, and update the status of the service instance in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

[0039] Furthermore, in the request allocation module, the first indicator also includes: user priority, user status, number of user historical requests and / or length of time the user waits in line.

[0040] Furthermore, the device also includes an advance queuing module, which is used to evaluate the remaining processing time based on the estimated processing time and the used processing time of the request being processed by the service instance, and allocate the next request to the waiting queue of the service instance in advance before the service instance processing is completed.

[0041] Furthermore, the device also includes a service instance maintenance module, which is used to maintain the survival status of the service instance through heartbeat checks. If a service instance with a status of "processing" is found to be offline, the request being processed is placed in a request queue.

[0042] Furthermore, the request includes a synchronous blocking request and an asynchronous non-blocking request; the synchronous blocking request means that the client waits until the service instance completes processing and returns the result after the request is issued; the asynchronous non-blocking request means that the client immediately returns the processing estimated time, waiting estimated time and overall response estimated time after the request is issued, and pushes the processing result to the client after the service instance completes processing.

[0043] Furthermore, the device also includes a denial of service module, which is used to return a denial of service prompt if there is no idle service instance.

[0044] Furthermore, the device also includes a request queue length setting module, which is used to set the length of the request queue. If the set length is reached, a prompt of denial of service is returned; if the set length is not reached, the current request and its corresponding parameters are placed in the request queue.

[0045] The device can be implemented through the load balancing scheduling method for model reasoning provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0046] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the load balancing scheduling method for model reasoning as described in the first embodiment.

[0047] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the load balancing scheduling method for model reasoning as described in Example 1.

[0048] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A load balancing scheduling method for model reasoning, characterized in that: include: Receive the user's current request; Estimate the processing time of the current request based on the parameters and processing time of historical requests and the parameters of the current request; According to the service instance status list, determine whether there is an idle service instance; wherein the service instance status list includes the service instance number, status and priority; If there is no idle service instance, the current request and its corresponding parameters are placed in the request queue; If there is an idle service instance, the request selected from the request queue according to the first indicator or the current request will be assigned to the service instance selected according to the second indicator, and the status of the service instance will be updated in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

2. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The first indicator also includes: user priority, user status, number of user historical requests and / or length of time the user waits in line.

3. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The method further comprises: The remaining processing time is evaluated based on the estimated processing time and the used processing time of the request being processed by the service instance. Before the service instance processing is completed, the next request is assigned to the waiting queue of the service instance in advance.

4. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The method further includes: maintaining the survival status of the service instance through a heartbeat check, and if it is found that the service instance with a status of "processing" is offline, placing the request being processed into a request queue.

5. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The requests include synchronous blocking requests and asynchronous non-blocking requests; the synchronous blocking request means that the client waits until the service instance completes processing and returns the result after the request is issued; the asynchronous non-blocking request means that the client immediately returns the processing estimated time, waiting estimated time and overall response estimated time after the request is issued, and pushes the processing result to the client after the service instance completes processing.

6. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The method further comprises: if there is no idle service instance, returning a prompt of rejecting the service.

7. The load balancing scheduling method for model reasoning according to claim 1, characterized in that: The method further comprises: Set the length of the request queue. If the length is reached, a denial of service prompt is returned. If the length is not reached, the current request and its corresponding parameters are placed in the request queue.

8. A load balancing scheduling device for model reasoning, characterized in that: include: A request receiving module, used to receive the current request of the user; A processing time estimation module is used to estimate the processing time of the current request based on the parameters of historical requests and the processing time of historical requests and the parameters of the current request; A judgment module, used to judge whether there is an idle service instance according to the service instance status list; wherein the service instance status list includes the service instance number, status and priority; A first allocation module, configured to place the current request and its corresponding parameters in a request queue if there is no idle service instance; The second allocation module is used to allocate the request selected from the request queue according to the first indicator or the current request to the service instance selected according to the second indicator if there is an idle service instance, and update the status of the service instance in the service instance status list; wherein the first indicator includes the estimated processing time of the requests stored in the request queue, and the second indicator includes the priority of the service instance.

9. A memory, characterized in that: Multiple instructions are stored, and the instructions are used to implement the load balancing scheduling method for model reasoning as described in any one of claims 1-7.

10. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, the memory stores multiple instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the load balancing scheduling method for model reasoning as described in any one of claims 1-7.

Images