Service scheduling method, electronic device, storage medium and computer program

Abstract

The present application discloses a service scheduling method, electronic equipment, storage medium, and computer program, relating to the field of data processing technology. A formula can be used to calculate the business type of user request information based on the request source or number of response tokens and response time obtained by analyzing the user request information. A corresponding target scheduling rule is determined based on the determined business type and the request model obtained by analyzing the user request information. A targeted scheduling strategy can then be executed using the target scheduling rule. Therefore, the technical problem of how to perform balanced scheduling according to business type during the use of Ollama can be solved, achieving the technical effect of executing targeted scheduling strategies according to business types to achieve specialized optimization processing for different business types.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a service scheduling method, electronic equipment, storage medium, and computer program. Background Art

[0002] The trend toward large language models as services and platforms is growing. Ollama is an open-source Large Language Model (LLM) service platform, accessible via an Application Programming Interface (API). It has minimal hardware and software requirements, making deployment and maintenance simple. Ollama is widely used in the industry as a foundational service for large language model applications and is suitable for lightweight service scenarios.

[0003] However, the convenience and lightweight nature of Ollama come with its low performance, lack of targeted configuration, lack of dedicated resource optimization for different types of business requests, lack of a load balancing mechanism, and limited support for high-concurrency requests. Therefore, how to balance scheduling by business type when using Ollama is an urgent issue that needs to be addressed. Summary of the Invention

[0004] The present application provides a service scheduling method, electronic device, storage medium and computer program to at least solve the problem in the related art of how to perform balanced scheduling according to business type when using Ollama.

[0005] This application provides a service scheduling method, including:

[0006] Performing request parsing on the acquired user request information to obtain the request fields corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request fields include at least the request source and the request model;

[0007] Determining whether the request source includes the business type of the user's request information, and if the request source does not include the business type of the user's request information, calculating the balance value based on the number of response tokens and the response time to obtain a target balance value;

[0008] Performing business classification processing on the user request information according to the target balance value to obtain the business type of the user request information;

[0009] Determine the target scheduling rules corresponding to the user request information based on the business type and request model, and schedule the corresponding target service according to the target scheduling rules, and execute the user request information through the target service.

[0010] The present application also provides a service scheduling device, including:

[0011] a parsing unit configured to parse the acquired user request information to obtain a request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request field includes at least a request source and a request model;

[0012] A first determining unit, configured to determine whether the request source includes a business type of the user request information;

[0013] A calculation unit, configured to calculate a balance value based on the number of response tokens and the response time to obtain a target balance value when the request source does not include the business type of the user request information;

[0014] A classification unit, configured to perform business classification processing on the user request information according to the target balance value to obtain the business type of the user request information;

[0015] The second determining unit is used to determine the target scheduling rule corresponding to the user request information according to the service type and the request model;

[0016] The scheduling unit is used to schedule the corresponding target service according to the target scheduling rules and execute the user request information through the target service.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned service scheduling methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned service scheduling methods are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned service scheduling methods when executed by a processor.

[0020] Through the service scheduling method, electronic device, storage medium and computer program of the present application, the business type of the user request information can be calculated using a formula based on the request source or the number of response words and the response time obtained by analyzing the user request information, and the corresponding target scheduling rules can be determined based on the determined business type and the request model obtained by analyzing the user request information. Targeted scheduling strategies can then be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types during the use of Ollama can be solved, and targeted scheduling strategies can be executed according to business types to achieve the technical effect of specialized optimization processing of different business types. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a service scheduling method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of another service scheduling method provided in an embodiment of the present application;

[0024] Figure 3 A processing flow chart of a service scheduling system provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a request parsing process provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of a service scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] Figure 1A flow chart of a service scheduling method provided in an embodiment of the present application is provided, and the method is described in detail in conjunction with the execution flow of the service scheduling method.

[0031] like Figure 1 As shown, the service scheduling method includes:

[0032] Step 101 , performing request parsing processing on the acquired user request information to obtain the request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request field at least includes the request source and the request model.

[0033] In an embodiment of the present application, user request information is the request input by the user that needs to be processed and the related information of this request, including but not limited to: the request that the user needs to process (for example: asking questions, searching, etc.), the related information of this request (for example: the model instance that the user needs to use, the type characteristics of this request, etc.).

[0034] The request field contains the key content of the user request information obtained after parsing the user request information, including but not limited to: request identifier (ID), request source, request model, and request prompt. The request ID is globally unique and independently identifies the user request information. The request source identifies the business source that sent the user request information. Different business sources often handle different services with different characteristics. Therefore, the request source can be used to simply describe the business characteristics of the user request information. The request model is used to determine the LLM that the user request information intends to invoke. The request prompt indicates the input provided to the model instance, namely, the request that the user needs to process.

[0035] Furthermore, the number of response tokens and response time for user request information can be pre-collected through the API call monitoring module. Response tokens represent the smallest unit of text (e.g., a word, phrase, or punctuation mark) in natural language processing and API responses. The number of response tokens refers to the number of tokens contained in the API response. For example, in a chatbot or text analysis service, the API will return a certain number of tokens as processing results. This number of tokens is the response token count and can be used to assess the efficiency and quality of text processing.

[0036] Response time refers to the time interval between when the API receives a user request and when it sends a response.

[0037] Step 102: Determine whether the request source includes the business type of the user request information. If the request source does not include the business type of the user request information, calculate the balance value based on the number of response words and the response time to obtain a target balance value.

[0038] In the embodiments of the present application, the service type of a user's request can be directly determined using the type information in the request source. However, not every request source includes this type information. Therefore, when determining the service type of a user's request, the request source is first checked to see if it is associated with a preset service type (e.g., the first service type or the second service type). For example, if the request source is an online customer service system and the type information included indicates the first service type, the preset service type is used directly.

[0039] When the request source is not associated with a preset business type, it is necessary to calculate the balance value of the user request information in order to determine the business type of the user request information.

[0040] It should be noted that the business types include at least a first business type and a second business type. The first business type is a high-frequency, low-response word type (such as single-round question and answer, user question and answer, and auxiliary programming), which requires a fast response (low response time), and has low graphics processing unit (GPU) resource usage but frequently applies for release; the second business type is a low-frequency, high-response word type (such as long text generation, image generation), which requires continuous occupation of a large amount of GPU resources and has a higher tolerance for response time.

[0041] The target balance value is a comprehensive indicator that reflects the balance between the data volume (number of response tokens) and processing speed (response time) of the API response without business type information.

[0042] Step 103: Perform business classification processing on the user request information according to the target balance value to obtain the business type of the user request information.

[0043] In an embodiment of the present application, business classification is performed based on the target balance value. The specific classification method is, for example, classification based on the size of the target balance value.

[0044] Specifically, it includes but is not limited to: when the target balance value is less than the preset balance value threshold, the business type of the user requested information is determined to be the first business type; when the target balance value is greater than the preset balance value threshold, the business type of the user requested information is determined to be the second business type; when the target balance value is equal to the preset balance value threshold, the business type of the user requested information is determined to be the preset business type.

[0045] Among them, the preset balance value threshold is a custom-set value, for example: 0, etc., which means that if the target balance value is greater than 0, the user request information is classified as the second business type, the target balance value is equal to 0, it is determined as the preset business type, and the target balance value is less than 0, the user request information is classified as the second business type.

[0046] Step 104 : determining a target scheduling rule corresponding to the user request information according to the service type and the request model, scheduling the corresponding target service according to the target scheduling rule, and executing the user request information through the target service.

[0047] In an embodiment of the present application, when determining the target scheduling rules, the corresponding target service can be determined according to the business type, and the model instance for processing user request information in the target service can be determined according to the request model. The target scheduling rules include at least two types, namely the first scheduling rule and the second scheduling rule. Once the target scheduling rules are determined, the corresponding target service is scheduled according to the target scheduling rules.

[0048] The target service is determined by the target scheduling rule and is used to process the corresponding user request information.

[0049] Different business types correspond to different target scheduling rules. For example, the first business type corresponds to the first scheduling rule, and the second business type corresponds to the second scheduling rule. The first scheduling rule includes, but is not limited to, rules for processing user request information of the first business type, and scheduling rules for capacity expansion or contraction required during the processing of user request information of the first business type. For example, during the processing of user request information of the first business type, capacity expansion or contraction operations must be performed within the same target service.

[0050] The content of the second scheduling rule includes but is not limited to: processing rules for user request information of the second business type, and scheduling rules that require expansion or reduction in capacity during the process of processing user request information of the second business type. For example, during the process of processing user request information of the second business type, expansion or reduction operations can be performed in different target services, etc.

[0051] It should be noted that the first scheduling rule corresponds to a single target service, and the second scheduling rule corresponds to multiple target services.

[0052] Furthermore, in order to facilitate understanding of the implementation process of this application, this application provides a flowchart of another service scheduling method, such as Figure 2 As shown, different scheduling logics are implemented according to the different business types of user requested information.

[0053] Through the service scheduling method of the present application, the business type of the user request information can be calculated using a formula based on the request source or the number of response words and the response time obtained by analyzing the user request information, and the corresponding target scheduling rules can be determined based on the determined business type and the request model obtained by analyzing the user request information. Targeted scheduling strategies can then be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types in the process of using Ollama can be solved, so that targeted scheduling strategies can be executed according to business types to achieve the technical effect of specialized optimization processing of different business types.

[0054] In an achievable embodiment of the present application, the present application proposes a service scheduling system, such as Figure 3 As shown, Figure 3 This is a process flow chart for the service scheduling system provided in this application. This application combines multiple Ollama services into an Ollama cluster, achieving unified access and scheduling, and providing services externally as a whole. Externally, this eliminates the need for customized requests, while internally, it achieves balanced scheduling and high availability based on service type. The internal scheduling logic is invisible to external users.

[0055] Furthermore, in the process of executing the service scheduling method of the present application, the service scheduling system includes: a request scheduling module, an API call monitoring module, a GPU card load monitoring module and an Ollama service interaction module.

[0056] The request scheduling module (including the request parsing module) is the unified entry point for user request information. That is, users no longer send user request information directly to the Ollama service, but instead send user request information to the request scheduling module. The functions of the request scheduling module include but are not limited to: parsing user request information, scheduling user request information to the corresponding Ollama service and corresponding model instance based on API call monitoring and graphics processing unit (GPU) card load monitoring, etc., to achieve load balancing scheduling while meeting business requirements.

[0057] API call monitoring module (i.e., call monitoring module): monitors the frequency of API calls (for example, the target call frequency of user request information, historical call frequency, etc.), response time, and the number of response tokens (the number of response words).

[0058] Monitor API call frequency by business type. API call frequency is a key indicator for determining whether to scale down model instances. Track the request reception and response return times for each request ID. Using the response return and request reception times, you can calculate the response time for user requests. The number of response tokens can be obtained from the response.

[0059] GPU card load monitoring module: monitors the GPU card's Video Random Access Memory (VRAM) usage and GPU usage (for example, target service load information, etc.).

[0060] Ollama service interaction module: responsible for starting and uninstalling models (including but not limited to: creating model instances in the target service, deleting model instances in the target service, etc.).

[0061] The main function of the Ollama service interaction module is to operate multiple Ollama services (including the target service), directly interact with multiple Ollama services, and is responsible for creating model instances (expansion) and destroying model instances (reduction) on multiple Ollama services.

[0062] Furthermore, it monitors the availability of the Ollama service. If it detects that the Ollama service has crashed and is unavailable, it proactively reports this to the request dispatch module. The request dispatch module then reschedules the user request to a new model instance to ensure successful execution of the current request, and then removes the Ollama service. Upon detecting that the Ollama service has returned to normal, it proactively reports this to the request dispatch module, resuming dispatch of user requests to that service.

[0063] In an achievable embodiment of the present application, regarding the parsing process of user request information, the present application embodiment provides a flowchart of request parsing, such as Figure 4 Shown, including:

[0064] Step 401: parse the user request information through a preset request parsing module to obtain a request field corresponding to the user request information.

[0065] In an embodiment of the present application, the preset request parsing module is a pre-designed software component that can be used to analyze and process user request information. It contains multiple parsing rules and algorithms for extracting useful data from requests. The preset request parsing module can be integrated into the request scheduling module to parse user request information.

[0066] The preset request parsing module can receive user requests through a predefined protocol adaptation layer, parse user request information according to protocol specifications, and identify and verify the integrity and legitimacy of key fields.

[0067] During the parsing process, the preset request parsing module can also adopt a multi-level exception handling mechanism: if key fields are missing (for example, the request source is not carried), an error code is returned and the process is interrupted; if the field format is illegal (for example, the request ID is repeated), a unique new ID is generated and an alarm event is recorded.

[0068] Step 402: extract and process the user request information through a preset call monitoring module to obtain the initial word quantity and initial response time of the user request information.

[0069] In the embodiments of this application, the preset call monitoring module is a pre-designed software component used to monitor and record data related to API calls. The initial number of tokens refers to the total number of tokens contained in the response content actually generated by the model instance in the target service after processing the user request information. The initial response time refers to the time it takes from receiving the user request information to fully generating the response result (the result of the target service processing the user request information).

[0070] Among them, the preset call monitoring module (API call monitoring module) captures and records the performance indicators related to request execution in user request information in real time, obtains the initial number of words and initial response time of the user request through data extraction and processing, and provides original data support for subsequent dynamic scheduling and scaling decisions.

[0071] Step 403, obtain the number of historical words and the historical response time; wherein, the number of historical initial words is the number of response words of the request information obtained at the previous adjacent moment when the user requested the information, and the historical response time is the response time of the request information obtained at the previous adjacent moment when the user requested the information.

[0072] In the embodiments of this application, the initial number of tokens and initial response time are unstable and fluctuate in actual use, causing repeated jumps in the service type classification results, leading to frequent scheduling policy changes and severe performance degradation. Therefore, it is necessary to smooth the initial number of tokens and initial response time to stabilize them. This smoothing process requires the participation of historical number of tokens and historical response times.

[0073] When obtaining the number of historical words and the historical response time, they can be directly obtained from the time series database. The time series database can be integrated into the request scheduling module, and the request scheduling module can save and regularly update the number of historical words and the historical response time.

[0074] Step 404 : performing data smoothing processing on the initial word unit number and the initial response time using a preset smoothing algorithm according to the historical word unit number and the historical response time to obtain the response word unit number and the response time.

[0075] In an embodiment of the present application, the preset smoothing algorithm is a user-defined algorithm, such as an exponential moving average (EMA) smoothing algorithm. Specifically, the present application does not impose any limitation on the preset smoothing algorithm.

[0076] Specifically, the calculation of the number of response words and the response time can be achieved through but not limited to formulas (1) and (2):

[0077] Formula (1)

[0078] Formula (2)

[0079] in, and After smoothing The value at that moment (i.e., the number of response tokens and response time). and After smoothing The value at that moment (that is, the number of historical tokens and historical response time). and yes The original (before smoothing) values ​​at time (i.e., the initial number of tokens and initial response time), It is the preset smoothing factor. The larger the smoothing factor, the smoother the value change.

[0080] By parsing user request information and obtaining the smoothed response word count and response time, high-precision indicator processing can be achieved in dynamic load scenarios, providing a reliable data foundation for business classification and resource scheduling.

[0081] In an achievable embodiment of the present application, in the process of determining whether the request source includes the business type of the user requested information, the following method can also be used but is not limited to: when the request source includes the business type of the user requested information, obtain the business type in the request source.

[0082] In an embodiment of the present application, if the request source includes the business type of the user request information, the target scheduling rules can be determined directly based on this business type. By coordinating the preset business classification with dynamic calculation, flexible management of business types can be achieved - ensuring both efficient scheduling of known business types and being compatible with adaptive identification of unknown businesses.

[0083] In an achievable embodiment of the present application, the balance value calculation based on the number of response words and the response time can also be implemented in but not limited to the following manner: the balance value calculation is performed based on a preset quantity watershed, a preset time watershed, a preset balance coefficient, the number of response words and the response time to obtain a target balance value; wherein the preset quantity watershed is at least used to represent the quantity threshold corresponding to the number of response words, the preset time watershed is at least used to represent the time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the number of response words and the response time.

[0084] In the embodiments of this application, the preset quantity watershed is a set threshold that represents a limit on the number of response tokens (i.e., the number of tokens in a response, which is typically related to the size or complexity of the response). When the number of response tokens exceeds this threshold, it indicates that the number of tokens is affecting the performance of the API.

[0085] The preset time watershed is another set threshold that represents a limit on the response time (i.e., the time it takes for the API to process a request and return a response). When the response time exceeds this threshold, it indicates that the API is responding slowly.

[0086] The target balance value can be calculated using, but not limited to, formula (3):

[0087] Formula (3)

[0088] in, is the target balance value, is the preset balance coefficient, is the preset quantity watershed, For the preset time watershed, In response to the number of tokens, is the response time.

[0089] By calculating the balance value based on the number of response words and response time, the system achieves a dynamic balance between business demand and resource supply while ensuring the scientific nature of classification.

[0090] In an achievable embodiment of the present application, the business classification processing of user request information based on the target balance value can also be implemented in but not limited to the following manner: when the target balance value is less than the preset balance value threshold, the business type of the user request information is determined to be the first business type; when the target balance value is greater than the preset balance value threshold, the business type of the user request information is determined to be the second business type; wherein the number of response words of the user request information of the second business type is greater than the number of response words of the user request information of the first business type, and the response time of the user request information of the second business type is less than the response time of the user request information of the first business type; when the target balance value is equal to the preset balance value threshold, the business type of the user request information is determined to be the preset business type; wherein the preset business type is the first business type or the second business type.

[0091] In an embodiment of the present application, the preset balance value threshold is a custom-set value, for example: 0, etc., indicating that if the target balance value is greater than 0, the user request information is classified as the second business type, the target balance value equal to 0 is determined as the preset business type, and the target balance value is less than 0, the user request information is classified as the second business type.

[0092] If the calculated target balance value is less than the preset balance value threshold, the service type of the user request information is determined to be the first service type, which means that the processing of the user request information is relatively simple, the number of response tokens is small, and the response time is short.

[0093] If the target balance value is greater than the preset balance value threshold, the service type of the user request information is determined to be the second service type, which means that the processing of the user request information is more complex, the number of response words is larger, but the response time is relatively longer.

[0094] If the target balance value is exactly equal to the preset balance value threshold, the service type of the user's requested information is determined to be the preset service type. The preset service type can be the first service type or the second service type, which is customized according to actual conditions.

[0095] Different types of services are distinguished based on target balance values ​​(which represent the performance characteristics of user requests). This allows for targeted resource allocation optimization, improved processing efficiency, and ensures that user requests of different service types are appropriately handled. For example, for requests from the second service type, more computing resources may be allocated to ensure a fast response, while for requests from the first service type, a different optimization strategy may be employed.

[0096] In an achievable embodiment of the present application, the process of determining the target scheduling rule corresponding to the user request information can also be implemented in the following manner, but is not limited to: when the business type is determined to be the first business type, determine a single service corresponding to the request model, and use the single service as the target service; determine the first scheduling rule corresponding to the user request information based on the target service, the first business type and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in a single service; when the business type is determined to be the second business type, determine multiple services corresponding to the request model, and use multiple services as target services; determine the second scheduling rule corresponding to the user request information based on the target service, the second business type and the request model; wherein the second scheduling rule is at least used to ensure that the user request information is executed in multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

[0097] In an embodiment of the present application, when it is determined that the business type of the user request information is the first business type, a single service corresponding to the request model will be identified, and the service can handle requests of this type. The determined single service will be used as the target service, that is, the service requested by the user will be executed.

[0098] The first scheduling rule is formulated based on the target service, the first business type, and the request model. It aims to ensure that the user request information can be smoothly executed in the selected single service, including but not limited to: resource allocation, priority setting, load balancing and other strategies.

[0099] When it is determined that the business type of the user request information is the second business type, multiple services corresponding to the request model will be identified. The multiple services combined can process the request more efficiently, and the determined multiple services will be used as target services, that is, the service set that executes the user request.

[0100] The second scheduling rule is formulated based on the target service, the second business type, and the request model. It aims to ensure that user request information can be coordinated and executed across multiple services, including but not limited to: inter-service communication, parallel processing, failover, and other strategies.

[0101] In an achievable embodiment of the present application, if the GPU card load monitoring module monitors that the load of the GPU card where the current Ollama service (target service) is located is too high, it is necessary to change the business type of the user request information being processed to the second business type to allocate more computing resources to ensure a quick response. Specifically, but not limited to, the following method can also be adopted: when the business type is determined to be the first business type, the load information of the target service is obtained, and when it is determined based on the load information that the load of the target service is greater than a preset load threshold, a target load record is generated based on the load information; a plurality of historical load records are obtained; wherein the plurality of historical load records are load records generated at multiple adjacent moments before the target load record; when it is determined based on the historical load records and the target load records that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first business type is updated to the second business type.

[0102] In an embodiment of the present application, the preset load threshold is a custom-set threshold, for example: 80% utilization rate, 70% utilization rate, etc., and the preset continuous threshold is also a custom-set threshold, for example: 4 times, 5 times, etc. Specifically, this application does not impose any restrictions on the preset load threshold and the preset continuous threshold.

[0103] Regarding the implementation process of this application, it can be carried out through the following example: If the GPU card load monitoring module monitors that the GPU card load where the current Ollama service (target service) is located is too high, for example: VRAM occupancy and GPU utilization rate are continuously higher than 80%, then the type of user request information at this time will be forcibly upgraded to the second business type, and scheduled according to the second business type mode.

[0104] By paying attention to the load information of the target service, the business type can be dynamically adjusted to adapt to the actual load situation of the target service, thereby ensuring the stability and efficiency of the target service. While ensuring the efficient scheduling of user request information of the first business type, it effectively prevents the risk of resource overload, reduces the service error rate caused by overload, and improves resource utilization.

[0105] In an achievable embodiment of the present application, when user request information continues to increase, further scheduling processing is required through the target service, that is, the target service is expanded to meet the situation of the continued increase in user request information. At the same time, scheduling has different logic for different business types. Therefore, when the user request information of the first business type continues to increase, the way the target service expands the capacity is different from the way the target service expands the capacity when the user request information of the second business type continues to increase.

[0106] When the user request information of the first business type continues to increase, the target service may be expanded in the following manner but is not limited to: when the business type is determined to be the first business type, the response time is calculated and processed with a ratio of the preset time watershed to obtain a first time ratio; when the first time ratio is greater than a first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

[0107] In an embodiment of the present application, the first preset ratio threshold is a custom-set threshold, for example: the response time reaches 75% of the preset time watershed, the response time reaches 80% of the preset time watershed, etc. Specifically, regarding the first preset ratio threshold, this application does not impose any restrictions.

[0108] For user request information of the first business type, it tends to run in the same Ollama service (target service). When the processing of first business type requests continues to increase and the response time increases to 75% of the preset time watershed, multi-model processing needs to be enabled.

[0109] It should be noted that the Ollama service generally does not allow multiple instances of the same model to run in the same service. Therefore, the Ollama service interaction module can create metadata for a new model instance (the first model instance), using a different model ID from the initial model instance but the same model file. This tricks the Ollama service into thinking it is running a different model instance. After the first model instance successfully runs, it reports the model ID to the request scheduling module. The request scheduling module then maintains a mapping between model names and model IDs, effectively matching model names to multiple model instances running in the Ollama service.

[0110] This means creating a first model instance within the target service node (the Ollama service node currently processing the request). This first model instance uses the same model file as the initial model instance (the currently running instance), but is assigned a different model ID (e.g., the initial model instance ID is "llama2-7b#1" and the first model instance ID is "llama2-7b#2") to bypass Ollama service restrictions. The creation of the first model instance is performed through the Ollama service interaction module and includes, but is not limited to, the following steps:

[0111] Generate a model ID for the first model instance, ensuring that it does not conflict with the initial model instance; call to load the model file and start the first model instance process; update the model name-ID mapping table and add the first model instance to the available queue.

[0112] By dynamically adjusting resource allocation based on the length of response time and creating additional model instances to handle requests, the performance and response speed of service scheduling can be optimized. While ensuring low latency for the first type of business, elastic resource supply can be achieved, the request discard rate in burst traffic scenarios can be reduced, and service quality can be improved.

[0113] When the user request information of the second business type continues to increase, the target service may be expanded in the following ways, but is not limited to: when the business type is determined to be the second business type, the response time is calculated and processed with a ratio of the preset time watershed to obtain a second time ratio; when the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

[0114] In an embodiment of the present application, the second preset ratio threshold is a custom-set threshold, for example: the response time reaches 3 times the preset time watershed, the response time reaches 3.5 times the preset time watershed, etc. Specifically, regarding the second preset ratio threshold, this application does not impose any restrictions.

[0115] For the second type of user requests, they tend to be distributed across different Ollama services (targets). When the number of requests for the first type continues to increase and the response time exceeds three times the preset time threshold, the Ollama service interaction module will launch a new model instance of the same service on another Ollama service.

[0116] By dynamically adjusting resource allocation based on the length of response time, model instances can be flexibly created in different services according to the response time of different business types to optimize resource utilization and system performance. While ensuring the resource exclusivity of the second business type, elastic expansion and intelligent load balancing across services can be achieved, significantly improving service stability and user experience in high-load scenarios.

[0117] In an achievable embodiment of the present application, after creating the first model instance or the second model instance, the user request information needs to be assigned to the newly created model instance for processing. Regarding the assignment of user request information, the following method can be used but is not limited to: obtaining the number of multiple running requests corresponding to each model instance in the target service, and performing pairwise comparison processing on the multiple running request numbers to obtain a number comparison result; when it is determined that the business type is the first business type, the user request information is assigned to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance; when it is determined that the business type is the second business type, the user request information is assigned to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

[0118] In an embodiment of the present application, the request scheduling module monitors each model instance in the target service and records the number of running requests each is processing. For example, the number of requests received and the number of requests responded and returned by each model running in the Ollama service can be calculated by subtracting the number of requests responded and returned by the same model instance from the number of requests received by the same model instance.

[0119] The user request information can be allocated to the model instance with fewer running requests. Since the number of running requests in the newly created first model instance or second model instance is 0, the user request information can be accurately allocated to the first model instance or the second model instance.

[0120] By allocating user request information to model instances with fewer running requests, system resources can be effectively utilized, avoiding overloading of some model instances while other model instances are idle, allowing user requests to be responded to quickly, improving user experience, making business processing more flexible, and adjusting the allocation strategy of model instances according to different business needs.

[0121] In an achievable embodiment of the present application, when scheduling user request information, scaling down processing will also be performed. Specifically, the following method may be used but is not limited to: obtaining the target call frequency and historical call frequency of the user request information; wherein the historical call frequency is the call frequency of the request information obtained at the previous adjacent moment of the user request information; performing a ratio calculation based on the target call frequency and the historical call frequency to obtain a frequency ratio, and scaling down the target service when the frequency ratio is less than a preset frequency ratio threshold.

[0122] In an embodiment of the present application, the preset frequency ratio threshold is a custom-set threshold, for example: the target call frequency is 50% of the historical call frequency, the target call frequency is 45% of the historical call frequency, etc. Specifically, this application does not impose any restrictions on the preset frequency ratio threshold.

[0123] When it is detected that the call frequency of user request information of a certain business type decreases, it is necessary to reduce the model instance (scale down). Every time it is detected that the call frequency is half of the previous historical call frequency, one model instance is reduced.

[0124] By scaling down based on the target call frequency of user request information, service resources can be dynamically adjusted to adapt to changes in the call frequency of user request information, which helps to improve resource utilization, reduce operating costs, and ensure service stability and responsiveness.

[0125] In an achievable embodiment of the present application, the scaling down process can also be implemented in the following manner but is not limited to: when the business type is determined to be the first business type, any model instance in the target service is deleted; when the business type is determined to be the second business type, in the target service, the model instance in the service with the highest load is deleted.

[0126] In an embodiment of the present application, when the user request information belongs to the first business type, the scaling-down strategy in the first scheduling rule will be executed. At this time, a model instance in the target service will be randomly selected for deletion without considering the current load situation, number of running requests or other performance indicators of the model instance, and it will be directly removed from the target service.

[0127] When a user request falls under the second service type, the scaling-down strategy in the second scheduling rule is executed. This strategy evaluates the load of each model instance in multiple target services and determines the model instance with the highest load for deletion. Load can be measured using metrics such as the number of running requests, CPU usage, and memory usage.

[0128] For user request information of the first business type, any model instance on the same Ollama service (target service) can be reduced. For user request information of the second business type, the model instance on the Ollama service with the highest GPU load is reduced.

[0129] It should be noted that after scaling up or down, a minimum wait time must be met before the next scaling operation can be performed. This prevents frequent jitter in scaling up or down, or drastic scaling up or down in a short period of time. Ultimately, the number of model instances and the request load are dynamically balanced.

[0130] Through differentiated scaling-down strategies, we can maximize resource utilization efficiency while ensuring service levels, which helps to dynamically adjust resources according to actual business needs and ensure service stability and efficiency.

[0131] In an achievable embodiment of the present application, the status of the target service can also be monitored and abnormal target services can be eliminated in a timely manner. Specifically, the following methods are included but not limited to: obtaining the status information of the target service; when it is determined that the target service has a fault based on the status information, rescheduling the user request information and marking the target service as an unavailable service.

[0132] In summary, this application can achieve the following technical effects:

[0133] 1. This application can use a formula to calculate the business type of the user request information based on the request source or the number of response words and the response time obtained by analyzing the user request information, and determine the corresponding target scheduling rule based on the determined business type and the request model obtained by analyzing the user request information. Targeted scheduling strategies can then be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types in the process of using Ollama can be solved, and targeted scheduling strategies can be executed according to business types to achieve the technical effect of specialized optimization processing of different business types.

[0134] 2. This application solves the problem of poor service performance of a single Ollama by clustering Ollama and connecting four external processing modules, while minimizing the intrusion into Ollama.

[0135] 3. This application distinguishes between two types of services and uses a formula to calculate the requested service type based on configuration or the number of response tokens and response time, thus implementing a targeted scheduling strategy based on the type. This allows for specialized optimization of different service types.

[0136] 4. This application periodically monitors the response time and call frequency of requests, determines the timing of expanding and shrinking the model instance, and maintains a dynamic balance between the model instance and the request load, thereby achieving a dynamic balance between maintaining service quality and resource consumption.

[0137] 5. This application monitors the status of the Ollama service, promptly removes abnormal target services, and promptly adds the recovered target services to the scheduling targets, thereby achieving high availability of large model services.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0139] The embodiment of the present application also provides a device for service scheduling. Figure 5 A schematic diagram of the structure of a service scheduling device provided for application, such as Figure 5 Shown, including:

[0140] The parsing unit 51 is configured to parse the acquired user request information to obtain a request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request field includes at least a request source and a request model;

[0141] A first determining unit 52 is configured to determine whether the request source includes the business type of the user request information;

[0142] A calculation unit 53 is configured to calculate a balance value based on the number of response tokens and the response time to obtain a target balance value when the request source does not include the service type of the user request information;

[0143] A classification unit 54 is configured to perform service classification processing on the user request information according to the target balance value to obtain the service type of the user request information;

[0144] The second determining unit 55 is used to determine the target scheduling rule corresponding to the user request information according to the service type and the request model;

[0145] The scheduling unit 56 is used to schedule the corresponding target service according to the target scheduling rule, and execute the user request information through the target service.

[0146] In one embodiment of the present application, the parsing unit 51 is further configured to:

[0147] The user request information is parsed by a preset request parsing module to obtain the request field corresponding to the user request information;

[0148] The preset call monitoring module is used to extract and process the user request information to obtain the initial word quantity and initial response time of the user request information;

[0149] Obtain the number of historical tokens and historical response time; wherein the number of historical initial tokens is the number of response tokens of the request information obtained immediately before the user requested the information, and the historical response time is the response time of the request information obtained immediately before the user requested the information;

[0150] According to the historical word unit number and historical response time, the initial word unit number and initial response time are smoothed by a preset smoothing algorithm to obtain the response word unit number and response time.

[0151] In an embodiment of the present application, the first determining unit 52 is further configured to, when the request source includes the service type of the user request information, obtain the service type in the request source.

[0152] In one embodiment of the present application, the calculation unit 53 is further configured to:

[0153] Calculate the balance value based on the preset quantity watershed, the preset time watershed, the preset balance coefficient, the number of response words, and the response time to obtain the target balance value;

[0154] Among them, the preset quantity watershed is at least used to represent the quantity threshold corresponding to the number of response words, the preset time watershed is at least used to represent the time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the number of response words and the response time.

[0155] In one embodiment of the present application, the classification unit 54 is further configured to:

[0156] When the target balance value is less than the preset balance value threshold, determining the service type of the user requested information as the first service type;

[0157] If the target balance value is greater than a preset balance value threshold, determining the service type of the user request information as a second service type; wherein the number of response tokens of the user request information of the second service type is greater than the number of response tokens of the user request information of the first service type, and the response time of the user request information of the second service type is less than the response time of the user request information of the first service type;

[0158] When the target balance value is equal to the preset balance value threshold, the service type of the user requested information is determined to be a preset service type; wherein the preset service type is the first service type or the second service type.

[0159] In one embodiment of the present application, the second determining unit 55 is further configured to:

[0160] When the business type is determined to be the first business type, determining a single service corresponding to the request model, and using the single service as a target service;

[0161] Determining a first scheduling rule corresponding to the user request information based on the target service, the first service type, and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in a single service;

[0162] In the case where the business type is determined to be the second business type, determining multiple services corresponding to the request model, and using the multiple services as target services;

[0163] Determine the second scheduling rule corresponding to the user request information based on the target service, the second business type and the request model; wherein the second scheduling rule is at least used to ensure that the user request information is executed in multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

[0164] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0165] When the service type is determined to be the first service type, obtaining load information of the target service, and when it is determined based on the load information that the load of the target service is greater than a preset load threshold, generating a target load record based on the load information;

[0166] Acquire multiple historical load records; wherein the multiple historical load records are load records generated at multiple adjacent moments before the target load record;

[0167] When it is determined according to the historical load record and the target load record that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first service type is updated to the second service type.

[0168] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0169] When the service type is determined to be the first service type, a ratio calculation is performed on the response time and a preset time watershed to obtain a first time ratio;

[0170] When the first time ratio is greater than the first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

[0171] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0172] When the service type is determined to be the second service type, a ratio calculation is performed on the response time and a preset time watershed to obtain a second time ratio;

[0173] When the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

[0174] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0175] Obtain the number of multiple running requests corresponding to each model instance in the target service, and perform pairwise comparison processing on the multiple running request numbers to obtain a number comparison result;

[0176] In the case where the business type is determined to be the first business type, the user request information is allocated to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance;

[0177] When it is determined that the service type is the second service type, the user request information is allocated to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

[0178] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0179] Obtain the target call frequency and historical call frequency of the user's request information; wherein the historical call frequency is the call frequency of the request information obtained at the previous adjacent time point of the user's request information;

[0180] A frequency ratio is obtained by calculating the ratio of the target call frequency to the historical call frequency. If the frequency ratio is less than a preset frequency ratio threshold, the target service is scaled down.

[0181] In one embodiment of the present application, the scheduling unit 56 is further configured to:

[0182] When the business type is determined to be the first business type, any model instance in the target service is deleted;

[0183] When it is determined that the business type is the second business type, in the target service, the model instance in the service with the highest load is deleted.

[0184] For the description of the features in the embodiment corresponding to the service scheduling device, please refer to the relevant description of the embodiment corresponding to the service scheduling method, and no further details will be given here.

[0185] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned service scheduling method embodiments.

[0186] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned service scheduling method embodiments when running.

[0187] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0188] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned service scheduling method embodiments are implemented.

[0189] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned service scheduling method embodiments are implemented.

[0190] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] The above is a detailed introduction to a service scheduling method, electronic device, storage medium and computer program provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A service scheduling method, characterized in that: include: Performing request parsing on the acquired user request information to obtain a request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request field includes at least a request source and a request model; determining whether the request source includes the business type of the user request information, and if the request source does not include the business type of the user request information, calculating a balance value based on the number of response tokens and the response time to obtain a target balance value; When the target balance value is less than a preset balance value threshold, determining the service type of the user request information as a first service type; If the target balance value is greater than the preset balance value threshold, determining the service type of the user request information as a second service type; wherein the number of response tokens of the user request information of the second service type is greater than the number of response tokens of the user request information of the first service type, and the response time of the user request information of the second service type is less than the response time of the user request information of the first service type; When the target balance value is equal to the preset balance value threshold, determining the service type of the user request information as a preset service type; wherein the preset service type is the first service type or the second service type; Determining a target scheduling rule corresponding to the user request information according to the service type and the request model, and scheduling a corresponding target service according to the target scheduling rule, and executing the user request information through the target service; wherein the target service is an Ollama service determined by the target scheduling rule and used to process the corresponding user request information; The determining of the target scheduling rule corresponding to the user request information according to the service type and the request model includes: In the case where it is determined that the business type is the first business type, determining a single service corresponding to the request model, and using the single service as the target service; Determining a first scheduling rule corresponding to the user request information based on the target service, the first business type, and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in the single service; In the case where it is determined that the business type is the second business type, determining a plurality of services corresponding to the request model, and using the plurality of services as the target services; Determine the second scheduling rule corresponding to the user request information based on the target service, the second business type and the request model; wherein, the second scheduling rule is at least used to ensure that the user request information is executed in the multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

2. The service scheduling method according to claim 1, characterized in that: The request parsing process is performed on the acquired user request information to obtain the request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information. Performing information parsing on the user request information by a preset request parsing module to obtain the request field corresponding to the user request information; Performing data extraction and processing on the user request information by calling a preset monitoring module to obtain the initial word quantity and initial response time of the user request information; Obtaining a historical number of tokens and a historical response time; wherein the historical number of tokens is the number of response tokens of the request information obtained at the immediately preceding moment of the user request information, and the historical response time is the response time of the request information obtained at the immediately preceding moment of the user request information; The initial word unit number and the initial response time are smoothed by a preset smoothing algorithm according to the historical word unit number and the historical response time to obtain the response word unit number and the response time.

3. The service scheduling method according to claim 1, characterized in that: After determining whether the request source includes the business type of the user request information, the method further includes: In a case where the request source includes the service type of the user request information, the service type in the request source is acquired.

4. The service scheduling method according to claim 1, characterized in that: The balance value calculation process is performed based on the number of response words and the response time to obtain the target balance value, including: Calculating a balance value based on a preset quantity watershed, a preset time watershed, a preset balance coefficient, the number of response words, and the response time to obtain the target balance value; Among them, the preset quantity watershed is at least used to represent the quantity threshold corresponding to the number of response words, the preset time watershed is at least used to represent the time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the number of response words and the response time.

5. The service scheduling method according to claim 1, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: If it is determined that the service type is the first service type, acquiring load information of the target service, and if it is determined according to the load information that the load of the target service is greater than a preset load threshold, generating a target load record according to the load information; Acquire multiple historical load records; wherein the multiple historical load records are load records generated at multiple adjacent moments before the target load record; When it is determined based on the historical load record and the target load record that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first business type is updated to the second business type.

6. The service scheduling method according to claim 1, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: When it is determined that the service type is the first service type, performing a ratio calculation on the response time and a preset time watershed to obtain a first time ratio; When the first time ratio is greater than a first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

7. The service scheduling method according to claim 6, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: When it is determined that the service type is the second service type, performing a ratio calculation on the response time and the preset time watershed to obtain a second time ratio; When the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

8. The service scheduling method according to claim 7, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: Obtaining multiple numbers of running requests corresponding to each model instance in the target service, and performing pairwise comparison processing on the multiple numbers of running requests to obtain a number comparison result; In a case where it is determined that the service type is the first service type, allocating the user request information to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance; In the case where it is determined that the business type is the second business type, the user request information is allocated to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

9. The service scheduling method according to claim 7, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: Obtaining a target call frequency and a historical call frequency of the user request information; wherein the historical call frequency is a call frequency of the request information obtained at an adjacent time immediately before the user request information; A ratio calculation is performed based on the target call frequency and the historical call frequency to obtain a frequency ratio, and when the frequency ratio is less than a preset frequency ratio threshold, the target service is scaled down.

10. The service scheduling method according to claim 9, characterized in that: When the frequency ratio is less than a preset frequency ratio threshold, scaling down the target service includes: In the case where it is determined that the business type is the first business type, deleting any model instance in the target service; When it is determined that the business type is the second business type, in the target service, the model instance in the service with the highest load is deleted.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the service scheduling method according to any one of claims 1 to 10 when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the service scheduling method according to any one of claims 1 to 10.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the service scheduling method according to any one of claims 1 to 10 are implemented.

Images