Service scheduling method, electronic equipment, storage medium and computer program

Abstract

The invention discloses a service scheduling method, electronic equipment, a storage medium and a computer program, and relates to the technical field of data processing, the service type of user request information can be calculated by utilizing a formula according to a request source or response lexical element quantity and response time obtained by analyzing the user request information; according to the method and the device, the service type is determined, the corresponding target scheduling rule is determined according to the determined service type and the request model obtained by analyzing the user request information, and then the targeted scheduling strategy can be executed through the target scheduling rule, so that the technical problem of how to carry out balanced scheduling according to the service type in the Ollan use process can be solved, and the user experience is improved. The technical effect of executing a targeted scheduling strategy according to the service type so as to realize special optimization processing on different service types is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a service scheduling method, electronic equipment, storage medium and computer program. Background Art

[0002] The trend is to turn large language models into services and platforms. Ollama is an open source Large Language Model (LLM) service platform that supports access via the Application Programming Interface (API). It does not require high software and hardware environment dependencies, and is relatively simple to deploy and maintain. Ollama is mostly used in the industry as a base service for large language model applications and is suitable for lightweight service scenarios.

[0003] However, the problem brought by Ollama's convenience and lightness is its low performance, lack of targeted configuration, no special resource optimization for different types of business requests, no load balancing mechanism, and limited support for high-concurrency requests. Therefore, how to balance scheduling according to business types in the process of using Ollama is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a service scheduling method, an electronic device, a storage medium and a computer program, so as to at least solve the problem in the related art of how to perform balanced scheduling according to business types in the process of using Ollama.

[0005] The present application provides a service scheduling method, including: Performing request parsing processing on the acquired user request information to obtain the request field corresponding to the user request information, as well as the number of response tokens and the response time of the user request information; wherein the request field at least includes the request source and the request model; Determine whether the request source includes the business type of the user request information, and if the request source does not include the business type of the user request information, calculate the balance value according to the number of response words and the response time to obtain a target balance value; Performing business classification processing on the user request information according to the target balance value to obtain the business type of the user request information; The target scheduling rules corresponding to the user request information are determined according to the business type and the request model, and the corresponding target service is scheduled according to the target scheduling rules, and the user request information is executed through the target service.

[0006] The present application also provides a service scheduling device, including: The parsing unit is used to perform request parsing processing on the acquired user request information to obtain the request field corresponding to the user request information, as well as the number of response words and the response time of the user request information; wherein the request field at least includes the request source and the request model; A first determining unit, used to determine whether the request source includes the business type of the user request information; A calculation unit, configured to calculate a balance value according to the number of response words and the response time to obtain a target balance value when the request source does not include the business type of the user request information; A classification unit, used to perform business classification processing on the user request information according to the target balance value to obtain the business type of the user request information; A second determining unit, configured to determine a target scheduling rule corresponding to the user request information according to the service type and the request model; The scheduling unit is used to schedule the corresponding target service according to the target scheduling rule, and execute the user request information through the target service.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned service scheduling methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned service scheduling methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned service scheduling methods when the computer program is executed by a processor.

[0010] Through the service scheduling method, electronic device, storage medium and computer program of the present application, the business type of the user request information can be calculated using a formula based on the request source or the number of response words and the response time obtained by analyzing the user request information, and the corresponding target scheduling rule can be determined based on the determined business type and the request model obtained by analyzing the user request information. Then, targeted scheduling strategies can be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types in the process of using Ollama can be solved, so as to achieve the technical effect of executing targeted scheduling strategies according to business types to achieve specialized optimization processing of different business types. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart of a service scheduling method provided in an embodiment of the present application; Figure 2 A flowchart of another service scheduling method provided in an embodiment of the present application; Figure 3 A processing flow chart of a service scheduling system provided in an embodiment of the present application; Figure 4 A schematic diagram of a request parsing process provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a service scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0016] Figure 1 A flow chart of a service scheduling method provided for an embodiment of the present application is provided, and the method is described in detail in conjunction with the execution flow of the service scheduling method.

[0017] like Figure 1 As shown, the service scheduling method includes: Step 101, perform request parsing processing on the acquired user request information to obtain the request field corresponding to the user request information, as well as the number of response words and the response time of the user request information; wherein the request field at least includes the request source and the request model.

[0018] In an embodiment of the present application, user request information is the request input by the user that needs to be processed and related information of this request, including but not limited to: the request that the user needs to process (for example: asking questions, searching, etc.), related information of this request (for example: the model instance that the user needs to use, the type characteristics of this request, etc.).

[0019] The request field is the key content of the user request information obtained after parsing the user request information, including but not limited to: request identifier (ID), request source, request model and request prompt words, etc. Among them, the request ID is the globally unique data used to independently identify this user request information, and the request source is used to determine the business source that sends the user request information. Different business sources often handle different businesses, and different businesses have different characteristics. Therefore, the business characteristics of the user request information can be simply described through the request source. The request model is used to determine the LLM that the user request information expects to call, and the request prompt word indicates the input part provided to the model instance, that is, the request that the user needs to process.

[0020] Furthermore, the number of response tokens and the response time of the user's request information can be obtained in advance through the API call monitoring module. The response token represents the smallest unit in the text of natural language processing and API response (for example, a word, phrase or punctuation mark). The number of response tokens refers to the number of tokens contained in the API response. For example, in a chatbot or text analysis service, the API will return a certain number of tokens as the processing result. The number of these tokens is the number of response tokens, which can be used to evaluate the efficiency and quality of text processing.

[0021] Response time refers to the time interval from when the API receives user request information to when it sends a response.

[0022] Step 102, determining whether the request source includes the business type of the user request information, and if the request source does not include the business type of the user request information, performing balance value calculation processing based on the number of response words and the response time to obtain a target balance value.

[0023] In the embodiment of the present application, the service type of the user request information can be directly determined by the type information in the request source, but not every request source of the user request information includes the type information. Therefore, when determining the service type of the user request information, first check whether the request source has been associated with a preset service type (such as the first service type or the second service type). For example, if the request source is an online customer service system, and the type information included therein is displayed as the first service type, the preset service type is directly used.

[0024] When the preset business type is not associated with the request source, it is necessary to calculate the balance value of the user request information in order to determine the business type of the user request information.

[0025] It should be noted that the business types include at least a first business type and a second business type, wherein the first business type is a high frequency, low response word number (such as single-round question and answer, user question and answer, assisted programming), requires a fast response (low response time), and has low graphics processing unit (GPU) resource occupancy but frequently applies for release; the second business type is a low frequency, high response word number (such as long text generation, image generation), requires continuous occupation of a large amount of GPU resources, and has a higher tolerance for response time.

[0026] The target balance value is a comprehensive indicator that reflects the balance between the data volume (number of response tokens) and processing speed (response time) of the API response without business type information.

[0027] Step 103: Perform business classification processing on the user request information according to the target balance value to obtain the business type of the user request information.

[0028] In the embodiment of the present application, the business classification process is performed by the target balance value. The specific classification method is, for example, performing classification process according to the size of the target balance value.

[0029] Specifically, it includes but is not limited to: when the target balance value is less than the preset balance value threshold, the business type of the user requested information is determined to be the first business type; when the target balance value is greater than the preset balance value threshold, the business type of the user requested information is determined to be the second business type; when the target balance value is equal to the preset balance value threshold, the business type of the user requested information is determined to be the preset business type.

[0030] Among them, the preset balance value threshold is a custom-set value, for example: 0, etc., which means that if the target balance value is greater than 0, the user request information is classified as the second business type, the target balance value is equal to 0 and is determined as the preset business type, and the target balance value is less than 0 and the user request information is classified as the second business type.

[0031] Step 104, determining the target scheduling rule corresponding to the user request information according to the business type and the request model, and scheduling the corresponding target service according to the target scheduling rule, and executing the user request information through the target service.

[0032] In an embodiment of the present application, when determining the target scheduling rules, the corresponding target service can be determined according to the business type, and the model instance for processing user request information in the target service can be determined according to the request model. The target scheduling rules include at least two types, namely, a first scheduling rule and a second scheduling rule. Once the target scheduling rules are determined, the corresponding target service is scheduled according to the target scheduling rules.

[0033] The target service is an Ollama service determined by the target scheduling rule and used to process the corresponding user request information.

[0034] Different business types correspond to different target scheduling rules, for example, the first business type corresponds to the first scheduling rule, and the second business type corresponds to the second scheduling rule. The content of the first scheduling rule includes but is not limited to: processing rules for user request information of the first business type, and scheduling rules for capacity expansion or contraction in the process of processing user request information of the first business type, for example, in the process of processing user request information of the first business type, the capacity expansion or contraction operation needs to be performed in the same target service, etc.

[0035] The content of the second scheduling rule includes but is not limited to: processing rules for user request information of the second business type, and scheduling rules for expanding or shrinking capacity in the process of processing user request information of the second business type. For example, in the process of processing user request information of the second business type, the expansion or reduction operation can be performed in different target services, etc.

[0036] It should be noted that the first scheduling rule corresponds to a single target service, and the second scheduling rule corresponds to multiple target services.

[0037] Further, in order to facilitate understanding of the implementation process of the present application, the present application provides a flowchart of another service scheduling method, such as Figure 2 As shown, different scheduling logics are implemented according to different business types of user requested information.

[0038] Through the service scheduling method of the present application, the business type of the user request information can be calculated using a formula based on the request source or the number of response words and the response time obtained by analyzing the user request information, and the corresponding target scheduling rule can be determined based on the determined business type and the request model obtained by analyzing the user request information. Then, targeted scheduling strategies can be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types in the process of using Ollama can be solved, so as to achieve the technical effect of executing targeted scheduling strategies according to business types to achieve specialized optimization processing of different business types.

[0039] In an achievable embodiment of the present application, the present application proposes a service scheduling system, such as Figure 3 As shown, Figure 3 A processing flow chart of a service scheduling system provided for this application. This application combines multiple Ollama services into an Ollama cluster to achieve unified entry and scheduling, and provide services to the outside world as a whole. No customized requests are required for external implementation, and balanced scheduling and high availability are achieved internally according to business types. The internal scheduling logic is unaware of external use.

[0040] Furthermore, in the process of executing the service scheduling method of the present application, the service scheduling system includes: a request scheduling module, an API call monitoring module, a GPU card load monitoring module and an Ollama service interaction module.

[0041] Request scheduling module (including: request parsing module): is the unified entrance for user request information, that is, users no longer send user request information directly to the Ollama service, but send user request information to the request scheduling module. The functions of the request scheduling module include but are not limited to: parsing user request information, scheduling user request information to the corresponding Ollama service and the corresponding model instance based on the monitoring of API calls and the monitoring of the graphics processing unit (GPU) card load, etc., to achieve load balancing scheduling while meeting business requirements.

[0042] API call monitoring module (i.e. call monitoring module): monitors the frequency of API calls (e.g. target call frequency of user request information, historical call frequency, etc.), response time and number of response tokens (number of response words).

[0043] Monitor the API call frequency according to the business type. The API call frequency is the criterion for determining whether the model instance should be scaled down. Track the request receiving time and response return time corresponding to each request ID. The response return time and request receiving time can be used to calculate the response time of the user's request information. The number of response tokens for the request can be obtained from the response.

[0044] GPU card load monitoring module: monitors the GPU card's video random access memory (VRAM) occupancy and GPU usage (for example, target service load information, etc.).

[0045] Ollama service interaction module: responsible for starting and uninstalling models (including but not limited to: creating model instances in the target service, deleting model instances in the target service, etc.).

[0046] The main function of the Ollama service interaction module is to operate multiple Ollama services (including the target service), directly interact with multiple Ollama services, and is responsible for creating model instances (expanding capacity) and destroying model instances (reducing capacity) on multiple Ollama services.

[0047] Furthermore, it is responsible for monitoring the availability of the Ollama service. If it is detected that the Ollama service crashes and is unavailable, it will actively report to the request scheduling module. The request scheduling module will reschedule the user request information to the new model instance to ensure that the current request can be executed successfully, and then remove the Ollama service. When it is detected that the Ollama service has returned to normal, it will also actively report to the request scheduling module and resume scheduling the user request information to the service.

[0048] In an achievable embodiment of the present application, regarding the parsing process of user request information, the present application embodiment provides a flowchart of request parsing, such as Figure 4 As shown, including: Step 401: parse the user request information through a preset request parsing module to obtain a request field corresponding to the user request information.

[0049] In an embodiment of the present application, the preset request parsing module is a pre-designed software component that can be used to analyze and process user request information, and includes multiple parsing rules and algorithms for extracting useful data from requests. The preset request parsing module can be integrated into the request scheduling module to parse user request information.

[0050] The preset request parsing module can receive user requests through a predefined protocol adaptation layer, parse user request information according to protocol specifications, and identify and verify the integrity and legitimacy of key fields.

[0051] During the parsing process, the preset request parsing module can also adopt a multi-level exception handling mechanism: if key fields are missing (such as: the request source is not carried), an error code is returned and the process is interrupted; if the field format is illegal (such as: the request ID is repeated), a unique new ID is generated and an alarm event is recorded.

[0052] Step 402: extract data from the user request information by calling a preset monitoring module to obtain the initial word quantity and initial response time of the user request information.

[0053] In an embodiment of the present application, the preset call monitoring module is a pre-designed software component for monitoring and recording the relevant data of API calls. The initial word number refers to the total number of words contained in the response content actually generated by the model instance in the target service after processing the user request information. The initial response time refers to the time consumed from receiving the user request information to completely generating the response result (the result of the target service processing the user request information).

[0054] Among them, the preset call monitoring module (API call monitoring module) captures and records the performance indicators related to request execution in the user request information in real time, obtains the initial number of words and initial response time of the user request through data extraction and processing, and provides original data support for subsequent dynamic scheduling and expansion and contraction decisions.

[0055] Step 403, obtain the number of historical word units and the historical response time; wherein the number of historical initial word units is the number of response word units of the request information obtained at the previous adjacent moment when the user requested the information, and the historical response time is the response time of the request information obtained at the previous adjacent moment when the user requested the information.

[0056] In the embodiment of the present application, since the initial word number and initial response time are unstable in actual use and are fluctuating values, the business type classification results will be repeatedly changed, resulting in frequent scheduling strategy changes and serious performance degradation. Therefore, it is necessary to smooth the initial word number and initial response time to stabilize the initial word number and initial response time, and the process of smoothing the initial word number and initial response time requires the participation of the historical word number and historical response time.

[0057] When obtaining the historical word number and historical response time, they can be directly obtained from the time series database. The time series database can be integrated into the request scheduling module. The request scheduling module can save and regularly update the historical word number and historical response time.

[0058] Step 404 , performing data smoothing processing on the initial word unit number and the initial response time by using a preset smoothing algorithm according to the historical word unit number and the historical response time, to obtain the response word unit number and the response time.

[0059] In an embodiment of the present application, the preset smoothing algorithm is a custom-set algorithm, such as an exponential moving average (EMA) smoothing algorithm, etc. Specifically, the present application does not impose any limitation on the preset smoothing algorithm.

[0060] Specifically, the calculation of the number of response words and the response time can be achieved by, but not limited to, formulas (1) and (2): Formula (1) Formula (2) in, and After smoothing The value at that moment (i.e., the number of response tokens and the response time). and After smoothing The value at that moment (that is, the number of historical tokens and historical response time). and yes The original (before smoothing) values ​​at time (i.e., the initial number of tokens and initial response time), It is the preset smoothing factor. The larger the smoothing factor is, the smoother the value changes.

[0061] By parsing user request information and obtaining the smoothed response word count and response time, high-precision indicator processing can be achieved in dynamic load scenarios, providing a reliable data basis for business classification and resource scheduling.

[0062] In an achievable embodiment of the present application, in the process of determining whether the request source includes the business type of the user request information, the following method may also be used but is not limited to: when the request source includes the business type of the user request information, the business type in the request source is obtained.

[0063] In an embodiment of the present application, if the request source includes the business type of the user request information, the target scheduling rules can be determined directly based on this business type. By coordinating the preset business classification with dynamic calculation, flexible management of business types can be achieved - ensuring both efficient scheduling of known business types and being compatible with adaptive identification of unknown businesses.

[0064] In an achievable embodiment of the present application, the balance value calculation based on the response word quantity and the response time can also be implemented in but not limited to the following manner: balance value calculation is performed based on a preset quantity watershed, a preset time watershed, a preset balance coefficient, the response word quantity and the response time to obtain a target balance value; wherein the preset quantity watershed is at least used to represent a quantity threshold corresponding to the response word quantity, the preset time watershed is at least used to represent a time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the response word quantity and the response time.

[0065] In the embodiment of the present application, the preset quantity watershed is a set threshold value used to indicate a limit of the number of response tokens (i.e., the number of tokens in a response, which is usually related to the size or complexity of the response). When the number of response tokens exceeds this threshold value, it indicates that the number of tokens affects the performance of the API.

[0066] The preset time watershed is another set threshold that represents a limit on the response time (i.e., the time it takes for the API to process a request and return a response). When the response time exceeds this threshold, it indicates that the API is responding slowly.

[0067] The target balance value can be calculated by, but not limited to, formula (3): Formula (3) in, is the target balance value, is the preset balance coefficient, is the preset quantity watershed, To preset the time watershed, In response to the number of tokens, For the response time.

[0068] By calculating the balance value based on the number of response words and response time, the system achieves a dynamic balance between business demand and resource supply while ensuring the scientific nature of classification.

[0069] In an achievable embodiment of the present application, the business classification processing of user request information according to the target balance value can also be implemented in but not limited to the following manner: when the target balance value is less than the preset balance value threshold, the business type of the user request information is determined to be the first business type; when the target balance value is greater than the preset balance value threshold, the business type of the user request information is determined to be the second business type; wherein the number of response words of the user request information of the second business type is greater than the number of response words of the user request information of the first business type, and the response time of the user request information of the second business type is less than the response time of the user request information of the first business type; when the target balance value is equal to the preset balance value threshold, the business type of the user request information is determined to be the preset business type; wherein the preset business type is the first business type or the second business type.

[0070] In an embodiment of the present application, the preset balance value threshold is a custom-set value, for example: 0, etc., indicating that if the target balance value is greater than 0, the user request information is classified as the second business type, the target balance value equal to 0 is determined as the preset business type, and the target balance value is less than 0. The user request information is classified as the second business type.

[0071] If the calculated target balance value is less than the preset balance value threshold, the service type of the user request information is determined as the first service type, which means that the processing of the user request information is relatively simple, the number of response words is small, and the response time is short.

[0072] If the target balance value is greater than the preset balance value threshold, the service type of the user request information is determined as the second service type, which means that the processing of the user request information is more complicated, the number of response words is larger, but the response time is relatively longer.

[0073] If the target balance value is exactly equal to the preset balance value threshold, the service type of the user's requested information is determined as the preset service type. The preset service type can be the first service type or the second service type, which is customized according to actual conditions.

[0074] Different business types are distinguished according to the target balance value (which can represent the performance characteristics of user request information), so that resource allocation can be optimized in a targeted manner, processing efficiency can be improved, and user request information of different business types can be properly processed. For example, for requests of the second business type, more computing resources will be allocated to ensure a fast response, while for requests of the first business type, different optimization strategies will be adopted.

[0075] In an achievable embodiment of the present application, the process of determining the target scheduling rule corresponding to the user request information can also be implemented in but not limited to the following manner: when the business type is determined to be the first business type, determine a single service corresponding to the request model, and use the single service as the target service; determine a first scheduling rule corresponding to the user request information based on the target service, the first business type, and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in a single service; when the business type is determined to be the second business type, determine multiple services corresponding to the request model, and use multiple services as target services; determine a second scheduling rule corresponding to the user request information based on the target service, the second business type, and the request model; wherein the second scheduling rule is at least used to ensure that the user request information is executed in multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

[0076] In an embodiment of the present application, when it is determined that the business type of the user request information is the first business type, a single service corresponding to the request model will be identified, and the service can handle requests of this type. The determined single service will be used as the target service, that is, the service that executes the user request.

[0077] The first scheduling rule is a first scheduling rule formulated based on the target service, the first business type and the request model, and is intended to ensure that the user request information can be smoothly executed in the selected single service, including but not limited to: resource allocation, priority setting, load balancing and other strategies.

[0078] When it is determined that the business type of the user request information is the second business type, multiple services corresponding to the request model will be identified. The multiple services combined can process the request more efficiently. The determined multiple services are used as target services, that is, the service set that executes the user request.

[0079] The second scheduling rule is a second scheduling rule formulated based on the target service, the second business type and the request model, and is intended to ensure that user request information can be coordinated and executed in multiple services, including but not limited to: communication between services, parallel processing, failover and other strategies.

[0080] In an achievable embodiment of the present application, if the GPU card load monitoring module monitors that the GPU card load where the current Ollama service (target service) is located is too high, it is necessary to change the business type of the user request information being processed to a second business type to allocate more computing resources to ensure a quick response. Specifically, but not limited to the following method can also be used: when the business type is determined to be the first business type, the load information of the target service is obtained, and when it is determined according to the load information that the load of the target service is greater than a preset load threshold, a target load record is generated according to the load information; a plurality of historical load records are obtained; wherein the plurality of historical load records are load records generated at a plurality of previous adjacent moments of the target load record; when it is determined according to the historical load records and the target load records that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first business type is updated to the second business type.

[0081] In an embodiment of the present application, the preset load threshold is a custom-set threshold, for example: 80% utilization rate, 70% utilization rate, etc., and the preset continuous threshold is also a custom-set threshold, for example: 4 times, 5 times, etc. Specifically, with respect to the preset load threshold and the preset continuous threshold, the present application does not impose any restrictions.

[0082] Regarding the implementation process of the present application, it can be carried out through the following example: If the GPU card load monitoring module monitors that the GPU card load where the current Ollama service (target service) is located is too high, for example: VRAM occupancy and GPU utilization rate continue to be higher than 80%, then the type of request for user information requested at this time will be forcibly upgraded to the second business type, and scheduled according to the second business type mode.

[0083] By paying attention to the load information of the target service, the business type can be dynamically adjusted to adapt to the actual load situation of the target service, thereby ensuring the stability and efficiency of the target service. While ensuring the efficient scheduling of user request information of the first business type, it can effectively prevent the risk of resource overload, reduce the service error rate caused by overload, and improve resource utilization.

[0084] In an achievable embodiment of the present application, when user request information continues to increase, further scheduling processing is required through the target service, that is, the target service is expanded to meet the situation of the continuous increase in user request information. At the same time, the scheduling has different logic for different business types. Therefore, when the user request information of the first business type continues to increase, the way in which the target service expands the capacity is different from the way in which the target service expands the capacity when the user request information of the second business type continues to increase.

[0085] When user request information of the first business type continues to increase, the target service may be expanded in the following manner but is not limited to: when the business type is determined to be the first business type, the response time is calculated and processed with a ratio of a preset time watershed to obtain a first time ratio; when the first time ratio is greater than a first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

[0086] In an embodiment of the present application, the first preset ratio threshold is a custom-set threshold, for example: the response time reaches 75% of the preset time watershed, the response time reaches 80% of the preset time watershed, etc. Specifically, regarding the first preset ratio threshold, the present application does not impose any restrictions.

[0087] For user request information of the first business type, it tends to run in the same Ollama service (target service). When the processing of first business type requests continues to increase and the response time increases to 75% of the preset time watershed, multi-model processing needs to be enabled.

[0088] It should be noted that the Ollama service usually does not allow multiple identical model instances to run in the same service. Therefore, the metadata information of a new model instance (the first model instance) can be created through the Ollama service interaction module, using a different model id from the initial model instance, but using the same model file. Let the Ollama service think that different model instances are running. After the first model instance runs successfully, the model id is fed back to the request scheduling module. The request scheduling module adds the correspondence between the maintenance model name and the model id, that is, the request scheduling module corresponds the model name to the multiple model instances running in the Ollama service.

[0089] That is, the first model instance is created in the target service node (i.e., the Ollama service node currently processing the request). The first model instance uses the same model file as the initial model instance (the currently running instance), but is assigned a different model ID (e.g., the initial model instance ID is "llama2-7b#1", and the first model instance is "llama2-7b#2") to bypass the restrictions of the Ollama service. The creation of the first model instance is performed through the Ollama service interaction module, including but not limited to the following steps: Generate a model ID for the first model instance to ensure that it does not conflict with the initial model instance; call to load the model file and start the first model instance process; update the model name-ID mapping table and add the first model instance to the available queue.

[0090] By dynamically adjusting resource allocation according to the length of response time and creating additional model instances to process requests, the performance and response speed of service scheduling can be optimized. While ensuring low latency for the first type of business, elastic resource supply can be achieved, the request discard rate in burst traffic scenarios can be reduced, and service quality can be improved.

[0091] When the user request information of the second business type continues to increase, the target service may be expanded in the following ways but are not limited to: when the business type is determined to be the second business type, the response time is ratioed to a preset time watershed to obtain a second time ratio; when the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

[0092] In an embodiment of the present application, the second preset ratio threshold is a custom-set threshold, for example: the response time reaches 3 times of the preset time watershed, the response time reaches 3.5 times of the preset time watershed, etc. Specifically, regarding the second preset ratio threshold, the present application does not impose any restrictions.

[0093] For the user request information of the second business type, it tends to be distributed to different Ollama services (targets) for operation. When the processing of the first business type request continues to increase and the response time increases to 3 times the preset time watershed, through the Ollama service interaction module, a model instance of the same service is started on another Ollama service.

[0094] By dynamically adjusting resource allocation according to the length of response time, model instances can be flexibly created in different services according to the response time of different business types to optimize resource utilization and system performance. While ensuring the resource exclusivity of the second business type, elastic expansion and intelligent load balancing across services can be achieved, significantly improving service stability and user experience in high-load scenarios.

[0095] In an achievable embodiment of the present application, after creating the first model instance or the second model instance, the user request information needs to be assigned to the newly created model instance for processing. Regarding the assignment of the user request information, the following method can be used but is not limited to: obtaining multiple numbers of running requests corresponding to each model instance in the target service, and performing pairwise comparison processing on the multiple numbers of running requests to obtain a number comparison result; when it is determined that the business type is the first business type, the user request information is assigned to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance; when it is determined that the business type is the second business type, the user request information is assigned to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

[0096] In an embodiment of the present application, the request scheduling module monitors each model instance in the target service and records the number of running requests being processed by each of them. For example, the number of requests received by the model running in each Ollama service and the number of requests responded and returned. The number of running requests can be obtained by subtracting the number of requests responded and returned by the same model instance from the number of requests received by the same model instance.

[0097] The user request information can be allocated to the model instance with a smaller number of running requests. Since the number of running requests in the newly created first model instance or second model instance is 0, the user request information can be accurately allocated to the first model instance or the second model instance.

[0098] By allocating user request information to model instances with fewer running requests, system resources can be used effectively, avoiding overloading of some model instances while other model instances are idle, allowing user requests to be responded to quickly, improving user experience, making business processing more flexible, and adjusting the allocation strategy of model instances according to different business needs.

[0099] In an achievable embodiment of the present application, when scheduling user request information, downscaling processing will also be performed. Specifically, the following method may be used but is not limited to: obtaining the target call frequency and historical call frequency of the user request information; wherein the historical call frequency is the call frequency of the request information obtained at the previous adjacent moment of the user request information; performing a ratio calculation based on the target call frequency and the historical call frequency to obtain a frequency ratio, and when the frequency ratio is less than a preset frequency ratio threshold, performing downscaling processing on the target service.

[0100] In an embodiment of the present application, the preset frequency ratio threshold is a custom-set threshold, for example: the target call frequency is 50% of the historical call frequency, the target call frequency is 45% of the historical call frequency, etc. Specifically, the present application does not impose any restrictions on the preset frequency ratio threshold.

[0101] When it is detected that the call frequency of user request information of a certain business type decreases, it is necessary to perform model instance reduction (scaling) operation. Every time it is detected that the call frequency is halved compared to the previous historical call frequency, one model instance is reduced.

[0102] By scaling down based on the target call frequency of user request information, service resources can be dynamically adjusted to adapt to changes in the call frequency of user request information, which helps to improve resource utilization, reduce operating costs, and ensure service stability and responsiveness.

[0103] In an achievable embodiment of the present application, the scaling down process can also be implemented in the following manner but is not limited to: when the business type is determined to be the first business type, any model instance in the target service is deleted; when the business type is determined to be the second business type, in the target service, the model instance in the service with the highest load is deleted.

[0104] In an embodiment of the present application, when the user request information belongs to the first business type, the scaling-down strategy in the first scheduling rule will be executed. At this time, a model instance in the target service will be randomly selected for deletion. The model instance will be directly removed from the target service without considering the current load situation, number of running requests or other performance indicators.

[0105] When the user request information belongs to the second business type, the reduction strategy in the second scheduling rule will be executed. At this time, the load of each model instance in multiple target services will be evaluated, and the model instance in the target service with the highest load will be deleted. The load can be measured by indicators such as the number of running requests, CPU usage, and memory usage.

[0106] For user request information of the first service type, any model instance on the same Ollama service (target service) can be reduced. For user request information of the second service type, the model instance on the Ollama service with the highest GPU card load is reduced.

[0107] It should be noted that after expansion or reduction, the minimum waiting time must be met before the next expansion / reduction operation can be performed again. Avoid frequent jitters in expansion / reduction or drastic expansion / reduction in a short period of time. Finally, the number of model instances and request load are dynamically balanced.

[0108] Through differentiated scaling-down strategies, we can maximize resource utilization efficiency while ensuring service levels, help dynamically adjust resources according to actual business needs, and ensure service stability and efficiency.

[0109] In an achievable embodiment of the present application, the status of the target service can also be monitored and abnormal target services can be eliminated in a timely manner. Specifically, the following methods are included but are not limited to: obtaining status information of the target service; when it is determined that the target service has a fault based on the status information, rescheduling the user request information and marking the target service as an unavailable service.

[0110] In summary, this application can achieve the following technical effects: 1. The present application can use a formula to calculate the business type of the user request information based on the request source or the number of response words and the response time obtained by analyzing the user request information, and determine the corresponding target scheduling rule based on the determined business type and the request model obtained by analyzing the user request information. Then, targeted scheduling strategies can be executed through the target scheduling rules. Therefore, the technical problem of how to perform balanced scheduling according to business types in the process of using Ollama can be solved, so as to achieve the technical effect of executing targeted scheduling strategies according to business types to achieve specialized optimization processing of different business types.

[0111] 2. This application solves the problem of low performance of a single Ollama service by clustering Ollama and connecting 4 external processing modules with less intrusion to Ollama.

[0112] 3. This application distinguishes two types of services, calculates the requested service type using a formula based on the configuration or the number of response words and the response time, and implements targeted scheduling strategies based on the type, thereby achieving specialized optimization processing for different service types.

[0113] 4. This application periodically monitors the response time and call frequency of requests, determines the timing of expanding and shrinking the model instance, and maintains a dynamic balance between the model instance and the request load, thereby achieving a dynamic balance between maintaining service quality and resource consumption.

[0114] 5. This application monitors the status of the Ollama service, removes abnormal target services in a timely manner, and adds the restored target services to the scheduling targets in a timely manner, thereby achieving high availability of large model services.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0116] The embodiment of the present application also provides a service scheduling device, Figure 5 A schematic diagram of the structure of a service scheduling device provided for application, such as Figure 5 As shown, including: The parsing unit 51 is used to perform request parsing processing on the acquired user request information to obtain the request field corresponding to the user request information, and the number of response word elements and the response time of the user request information; wherein the request field at least includes the request source and the request model; A first determining unit 52, configured to determine whether the request source includes a business type of user request information; A calculation unit 53, configured to calculate a balance value according to the number of response words and the response time to obtain a target balance value when the request source does not include the service type of the user request information; A classification unit 54, configured to perform service classification processing on the user request information according to the target balance value to obtain the service type of the user request information; The second determining unit 55 is used to determine the target scheduling rule corresponding to the user request information according to the service type and the request model; The scheduling unit 56 is used to schedule the corresponding target service according to the target scheduling rule, and execute the user request information through the target service.

[0117] In one embodiment of the present application, the parsing unit 51 is further configured to: The user request information is parsed by a preset request parsing module to obtain the request field corresponding to the user request information; The preset call monitoring module is used to extract and process the user request information to obtain the initial word quantity and initial response time of the user request information; Obtain the number of historical tokens and historical response time; wherein the number of historical initial tokens is the number of response tokens of the request information obtained at the previous adjacent time when the user requested the information, and the historical response time is the response time of the request information obtained at the previous adjacent time when the user requested the information; According to the historical word unit number and historical response time, the initial word unit number and initial response time are smoothed by a preset smoothing algorithm to obtain the response word unit number and response time.

[0118] In an embodiment of the present application, the first determining unit 52 is further configured to, when the request source includes the service type of the user request information, obtain the service type in the request source.

[0119] In one embodiment of the present application, the calculation unit 53 is further configured to: The balance value is calculated and processed according to the preset quantity watershed, the preset time watershed, the preset balance coefficient, the number of response words and the response time to obtain the target balance value; Among them, the preset quantity watershed is at least used to represent the quantity threshold corresponding to the number of response words, the preset time watershed is at least used to represent the time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the number of response words and the response time.

[0120] In one embodiment of the present application, the classification unit 54 is further used for: When the target balance value is less than a preset balance value threshold, determining the service type of the user request information as a first service type; When the target balance value is greater than the preset balance value threshold, the business type of the user request information is determined to be a second business type; wherein the number of response words of the user request information of the second business type is greater than the number of response words of the user request information of the first business type, and the response time of the user request information of the second business type is less than the response time of the user request information of the first business type; When the target balance value is equal to the preset balance value threshold, the service type of the user requested information is determined as the preset service type; wherein the preset service type is the first service type or the second service type.

[0121] In one embodiment of the present application, the second determining unit 55 is further configured to: In the case where the business type is determined to be the first business type, a single service corresponding to the request model is determined, and the single service is used as a target service; Determine a first scheduling rule corresponding to the user request information based on the target service, the first business type, and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in a single service; In the case where the business type is determined to be the second business type, multiple services corresponding to the request model are determined, and the multiple services are used as target services; Determine the second scheduling rule corresponding to the user request information based on the target service, the second business type and the request model; wherein the second scheduling rule is at least used to ensure that the user request information is executed in multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

[0122] In one embodiment of the present application, the scheduling unit 56 is further configured to: When it is determined that the service type is the first service type, load information of the target service is obtained, and when it is determined according to the load information that the load of the target service is greater than a preset load threshold, a target load record is generated according to the load information; Acquire multiple historical load records; wherein the multiple historical load records are load records generated at multiple previous adjacent moments of the target load record; When it is determined according to the historical load record and the target load record that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first service type is updated to the second service type.

[0123] In one embodiment of the present application, the scheduling unit 56 is further configured to: When it is determined that the service type is the first service type, a ratio calculation is performed between the response time and a preset time watershed to obtain a first time ratio; When the first time ratio is greater than the first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

[0124] In one embodiment of the present application, the scheduling unit 56 is further configured to: When it is determined that the service type is the second service type, a ratio calculation is performed between the response time and a preset time watershed to obtain a second time ratio; When the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

[0125] In one embodiment of the present application, the scheduling unit 56 is further configured to: Obtain the number of multiple running requests corresponding to each model instance in the target service, and compare the number of multiple running requests in pairs to obtain a number comparison result; In the case where it is determined that the service type is the first service type, the user request information is allocated to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance; When it is determined that the service type is the second service type, the user request information is allocated to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

[0126] In one embodiment of the present application, the scheduling unit 56 is further configured to: Obtaining the target calling frequency and historical calling frequency of the user's request information; wherein the historical calling frequency is the calling frequency of the request information obtained at the previous adjacent time when the user requested the information; A ratio is calculated based on the target call frequency and the historical call frequency to obtain a frequency ratio, and when the frequency ratio is less than a preset frequency ratio threshold, the target service is scaled down.

[0127] In one embodiment of the present application, the scheduling unit 56 is further configured to: When it is determined that the business type is the first business type, any model instance in the target service is deleted; When it is determined that the business type is the second business type, in the target service, the model instance in the service with the highest load is deleted.

[0128] For the description of the features in the embodiment corresponding to the service scheduling device, please refer to the relevant description of the embodiment corresponding to the service scheduling method, which will not be repeated here.

[0129] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned service scheduling method embodiments.

[0130] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned service scheduling method embodiments when running.

[0131] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0132] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned service scheduling method embodiments are implemented.

[0133] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned service scheduling method embodiments are implemented.

[0134] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] The above is a detailed introduction to a service scheduling method, electronic device, storage medium and computer program provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A service scheduling method, characterized in that: include: Performing request parsing processing on the acquired user request information to obtain a request field corresponding to the user request information, as well as the number of response words and the response time of the user request information; wherein the request field at least includes a request source and a request model; determining whether the request source includes the business type of the user request information, and if the request source does not include the business type of the user request information, performing a balance value calculation process according to the number of response words and the response time to obtain a target balance value; Performing business classification processing on the user request information according to the target balance value to obtain the business type of the user request information; The target scheduling rule corresponding to the user request information is determined according to the business type and the request model, and the corresponding target service is scheduled according to the target scheduling rule, and the user request information is executed through the target service.

2. The service scheduling method according to claim 1, characterized in that: The request parsing process is performed on the acquired user request information to obtain the request field corresponding to the user request information, and the number of response words and the response time of the user request information, including: Performing information parsing processing on the user request information by a preset request parsing module to obtain the request field corresponding to the user request information; Performing data extraction and processing on the user request information by calling a preset monitoring module to obtain the initial word unit number and initial response time of the user request information; Obtaining the number of historical word units and historical response time; wherein the number of historical word units is the number of response word units of the request information obtained at the previous adjacent time of the user request information, and the historical response time is the response time of the request information obtained at the previous adjacent time of the user request information; According to the historical word unit number and the historical response time, data smoothing processing is performed on the initial word unit number and the initial response time by using a preset smoothing algorithm to obtain the response word unit number and the response time.

3. The service scheduling method according to claim 1, characterized in that: After determining whether the request source includes the business type of the user request information, the method further includes: In a case where the request source includes the service type of the user request information, the service type in the request source is acquired.

4. The service scheduling method according to claim 1, characterized in that: The balance value calculation process is performed according to the number of response words and the response time to obtain the target balance value, which includes: Performing balance value calculation processing according to a preset quantity watershed, a preset time watershed, a preset balance coefficient, the number of response word units and the response time to obtain the target balance value; Among them, the preset quantity watershed is at least used to represent the quantity threshold corresponding to the number of response words, the preset time watershed is at least used to represent the time threshold corresponding to the response time, and the preset balance coefficient is a parameter used to adjust the relationship between the number of response words and the response time.

5. The service scheduling method according to claim 4, characterized in that: The performing business classification processing on the user request information according to the target balance value to obtain the business type of the user request information includes: When the target balance value is less than a preset balance value threshold, determining the service type of the user request information as a first service type; In a case where the target balance value is greater than the preset balance value threshold, determining the service type of the user request information as a second service type; wherein the number of response words of the user request information of the second service type is greater than the number of response words of the user request information of the first service type, and the response time of the user request information of the second service type is less than the response time of the user request information of the first service type; When the target balance value is equal to the preset balance value threshold, the service type of the user request information is determined as a preset service type; wherein the preset service type is the first service type or the second service type.

6. The service scheduling method according to claim 5, characterized in that: Determining the target scheduling rule corresponding to the user request information according to the service type and the request model includes: In the case where it is determined that the business type is the first business type, determining a single service corresponding to the request model, and using the single service as the target service; Determine a first scheduling rule corresponding to the user request information based on the target service, the first business type, and the request model; wherein the first scheduling rule is at least used to ensure that the user request information is executed in the single service; In the case where it is determined that the business type is the second business type, determining a plurality of services corresponding to the request model, and using the plurality of services as the target services; Determine a second scheduling rule corresponding to the user request information based on the target service, the second business type and the request model; wherein the second scheduling rule is at least used to ensure that the user request information is executed in the multiple services, and the target scheduling rule is the first scheduling rule or the second scheduling rule.

7. The service scheduling method according to claim 6, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: In the case where it is determined that the service type is the first service type, acquiring load information of the target service, and in the case where it is determined according to the load information that the load of the target service is greater than a preset load threshold, generating a target load record according to the load information; Acquire multiple historical load records; wherein the multiple historical load records are load records generated at multiple previous adjacent moments of the target load record; When it is determined according to the historical load record and the target load record that the load of the target service is greater than the preset load threshold for a continuous number of times, and when the continuous number is greater than the preset continuous threshold, the first service type is updated to the second service type.

8. The service scheduling method according to claim 6, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: When it is determined that the service type is the first service type, performing a ratio calculation process on the response time and the preset time watershed to obtain a first time ratio; When the first time ratio is greater than a first preset ratio threshold, a first model instance is created in the target service, and the user request information is executed through the first model instance, wherein the first model instance and the initial model instance are in the same service, and the initial model instance is the model instance that is executing the user request information.

9. The service scheduling method according to claim 8, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: When it is determined that the service type is the second service type, performing a ratio calculation process on the response time and the preset time watershed to obtain a second time ratio; When the second time ratio is greater than a second preset ratio threshold, a second model instance is created in the target service, and the user request information is executed through the second model instance, wherein the second model instance and the initial model instance are in different services.

10. The service scheduling method according to claim 9, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: Obtaining multiple numbers of running requests corresponding to each model instance in the target service, and performing pairwise comparison processing on the multiple numbers of running requests to obtain a number comparison result; In the case where it is determined that the service type is the first service type, allocating the user request information to the first model instance based on the number comparison result, so as to execute the user request information through the first model instance; In the case where it is determined that the business type is the second business type, the user request information is allocated to the second model instance based on the number comparison result, so as to execute the user request information through the second model instance.

11. The service scheduling method according to claim 9, characterized in that: After scheduling the corresponding target service according to the target scheduling rule and executing the user request information through the target service, the method further includes: Obtaining a target calling frequency and a historical calling frequency of the user request information; wherein the historical calling frequency is a calling frequency of the request information obtained at an adjacent time before the user request information; A ratio calculation is performed according to the target call frequency and the historical call frequency to obtain a frequency ratio, and when the frequency ratio is less than a preset frequency ratio threshold, the target service is scaled down.

12. The service scheduling method according to claim 11, characterized in that: When the frequency ratio is less than a preset frequency ratio threshold, scaling down the target service includes: In the case where it is determined that the business type is the first business type, deleting any model instance in the target service; When it is determined that the business type is the second business type, in the target service, the model instance in the service with the highest load is deleted.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the service scheduling method as claimed in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the service scheduling method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the service scheduling method according to any one of claims 1 to 12 are implemented.

Images