Medical large model service instance scheduling method and system based on delay and traffic portrait

Abstract

The application belongs to the field of artificial intelligence large model services, and provides a medical large model service instance scheduling method and system based on time delay and traffic portrait, which comprises: obtaining and storing the meta information of all large model service instances and generating a backend service list; analyzing and extracting features based on the medical request of a user to determine the request feature label of the medical request; periodically calculating the dynamic load coefficient and load state of each large model service instance according to the performance index in the service process of each large model service instance; determining all candidate large model service instances based on the request feature label and the backend service list, and determining the optimal large model service instance according to the dynamic load coefficient of each candidate large model instance and the time delay tolerance in the request feature label. The application solves the problems of uncontrollable delay and poor adaptability of instance performance fluctuation in a heterogeneous large model service cluster, significantly improves the reliability of medical business while taking into account the scheduling efficiency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence large model service technology, specifically involving a method and system for scheduling medical large model service instances based on latency and traffic profiling. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As large language models are increasingly used in medical scenarios such as assisted diagnostic report generation, medical record summarization, medical Q&A, and clinical decision support, medical institutions typically need to call various intelligent model services with different performance levels. These services are deployed locally or on different cloud platforms, forming a heterogeneous model service cluster.

[0004] Existing scheduling methods often employ fair round-robin or forwarding based solely on network layer information, failing to differentiate between latency-sensitive critical services like real-time outpatient consultations and services that tolerate longer processing times, such as inpatient research data analysis. This makes it difficult to guarantee responsiveness for time-sensitive services. Furthermore, the scheduler lacks semantic awareness of the requested task content, often assigning complex tasks to smaller models with insufficient processing capacity, impacting result quality, or assigning simple tasks to larger models, resulting in wasted computing power. In addition, the actual processing capacity of each model instance dynamically changes due to underlying hardware, concurrent load, and the model's own state. Existing solutions typically rely on static weight configuration or simple heartbeat detection, failing to accurately determine the instance's current load and actual processing capacity in real time, easily forwarding new requests to already congested nodes. Moreover, the deployment, shutdown, and performance tuning of model services heavily rely on manual intervention and policy configuration; the system lacks the ability to adaptively schedule based on traffic characteristics and model state, resulting in high maintenance costs and slow response times.

[0005] Therefore, in the existing heterogeneous model service cluster, it is impossible to achieve adaptive scheduling that takes into account business priorities, request content and real-time model status, and it is also impossible to meet the differentiated timeliness and quality requirements of medical intelligent services, resulting in high operation and maintenance complexity. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a method and system for scheduling large-scale medical model service instances based on latency and traffic profiling. This invention understands the real-time requirements and task complexity of medical requests through real-time traffic profiling, and combines this with accurate performance profiling (latency load) of backend large-scale model instances to achieve an unfair intelligent scheduling approach—that is, prioritizing high-real-time, high-priority requests to obtain high-quality, fast service resources.

[0007] According to some embodiments, the first solution of the present invention provides a method for scheduling large-scale medical model service instances based on latency and traffic profiling, which adopts the following technical solution: A method for scheduling large-scale medical model service instances based on latency and traffic profiling includes: Obtain and store the metadata of all large model service instances, and generate a backend service list; Based on the user's medical requests, feature extraction is performed to determine the request feature tags of the medical requests; Based on the performance metrics of each large model service instance during its service process, the dynamic load coefficient and load status of each large model service instance are periodically calculated. Based on request feature tags and the backend service list, all candidate large model service instances are identified, and the optimal large model service instance is determined according to the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags.

[0008] Furthermore, the step of analyzing and extracting features based on the user's medical request to determine the request feature tags of the medical request includes: Receive the user's medical request and extract the set of prompt words for the medical request; The hash value is calculated based on the set of prompt words as the request fingerprint, and the historical request profile records that match are retrieved from the pre-built vector database based on the request fingerprint. If a match is found, request feature tags are generated based on the matched historical request profile records. If a match fails, the current medical request is temporarily labeled based on a preset default feature label, which serves as the basis for scheduling the large model. At the same time, a request profile is asynchronously generated in the background based on the prompt word set, and request feature labels are generated based on the request profile.

[0009] Furthermore, if the matching fails, the current medical request is temporarily labeled based on a preset default feature label, which serves as the basis for large model scheduling; simultaneously, a request profile is asynchronously generated in the background based on the prompt word set, and request feature labels are generated based on the request profile, including: If the match fails, the current medical request is temporarily labeled based on the preset default feature labels, and the default feature labels are used as the basis for large model scheduling. Simultaneously, request profile records are asynchronously generated in the background based on the set of prompt words, and request feature tags are generated based on the request profile records.

[0010] Furthermore, the preheating of large model service instances that are launched or registered for the first time based on standard test requests, generating an initial performance baseline, includes: Based on standard test requests, large model service instances that are launched or registered for the first time are warmed up to generate an initial performance baseline; During the formal service phase, the end-to-end response latency, number of input tokens, and number of output tokens for each medical request processed by the large model service instance are continuously acquired to form a request-level performance record. Based on request-level performance records, the performance metrics of each large model service instance are calculated periodically, and the dynamic load coefficient and load status are calculated based on the performance metrics.

[0011] Furthermore, based on request-level performance records, performance metrics for each large model service instance are periodically calculated, and dynamic load coefficients and load states are calculated based on these performance metrics, including: Based on all request-level performance records of the large model service instance within the current preset period, calculate the average first-word latency, average generation latency, request throughput, average number of input tokens, and average number of output tokens as performance indicators. The average first-word delay and average generation delay are normalized to the baseline first-word delay and baseline generation delay in the initial performance baseline, respectively, to obtain the normalized first-word delay factor and the normalized generation delay factor. The request throughput is normalized by comparing it with the historical maximum request throughput in the initial performance baseline to obtain the normalized throughput utilization. The normalized first-word delay factor, the normalized generation delay factor, and the normalized throughput utilization are weighted and summed to obtain the dynamic load coefficient, and the corresponding load state is determined based on the dynamic load coefficient.

[0012] Furthermore, the process of determining all candidate large model service instances based on request feature tags and the backend service list, and determining the optimal large model service instance based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags, includes: Based on request feature tags, all candidate large model service instances are identified from the backend service list; If the latency tolerance in the request feature tag is low tolerance, then the largest model service instance with the lowest dynamic load coefficient is selected as the optimal large model service instance. If the latency tolerance in the request feature tag is high tolerance, then the large model service instance with the highest dynamic load coefficient among those exceeding the preset overload threshold is selected as the optimal large model service instance.

[0013] According to some embodiments, the second aspect of the present invention provides a medical large-scale model service instance scheduling system based on latency and traffic profiling, which adopts the following technical solution: A medical big data model service instance scheduling system based on latency and traffic profiling includes: The service list generation module is used to obtain and store the metadata of all large model service instances and generate a backend service list. The medical request feature extraction module is used to analyze and extract features based on the user's medical requests, and determine the request feature tags of the medical requests; The instance load determination module is used to periodically calculate the dynamic load coefficient and load status of each large model service instance based on the performance indicators during the service process of each large model service instance. The instance scheduling module is used to determine all candidate large model service instances based on request feature tags and the backend service list, and to determine the optimal large model service instance based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags.

[0014] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the medical large-scale model service instance scheduling method based on latency and traffic profiling as described in the first scheme above.

[0016] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the medical big data model service instance scheduling method based on latency and traffic profiling as described in the first scheme above.

[0018] According to some embodiments, a fifth aspect of the present invention provides a computer program product or computer program.

[0019] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the medical big data model service instance scheduling method based on latency and traffic profiling as described in the first embodiment above.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention incorporates the recommended model parameter magnitude into request feature tags. During scheduling, it filters candidate instance groups with matching capabilities from the service list based on this parameter magnitude. This directs complex tasks to models with large parameter magnitudes to ensure result quality, while simple tasks are directed to models with small parameter magnitudes to avoid wasting computing power. Combined with the dynamic selection of instance real-time load coefficients, the system achieves overall throughput improvement and resource utilization optimization for heterogeneous model clusters while ensuring the service quality of each business. Simultaneously, by proactively avoiding instances already overloaded, it reduces the risk of single-instance overload at the source, improving the overall stability of the service chain. After a request is forwarded to the optimal large-model service instance and processed, performance data is sent back to the collection stage, forming a closed loop of "collection—computation—scheduling—feedback". When an instance experiences performance degradation, its load coefficient will increase in the next computation cycle, and the scheduling module automatically reduces request allocation to that instance; when the instance recovers, the load coefficient drops, and request allocation automatically increases. This mechanism requires no manual intervention and effectively reduces the risk of request failures due to single-point performance degradation. In summary, this invention solves the core problems of uncontrollable latency, lack of business priority differentiation, and poor adaptability to instance performance fluctuations in heterogeneous large model service clusters through deep coupling of "request semantic awareness - real-time quantification of instance performance - differentiated adaptive scheduling". It significantly improves the reliability of medical services and user experience while taking into account scheduling efficiency. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of the medical large-scale model service instance scheduling method based on latency and traffic profiling in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] Example 1 like Figure 1 As shown, this embodiment provides a method for scheduling medical large-scale model service instances based on latency and traffic profiling. This embodiment uses the application of this method to a server as an example for illustration. It can be understood that this method can also be applied to terminals, and can also be applied to a system including terminals, servers, and systems, and can be implemented through the interaction between terminals and servers. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps: Step S1: Obtain and store the metadata of all large model service instances, and generate a backend service list; Obtain the metadata for each large model service instance. Meta information includes at least the instance's unique identifier, access endpoint (URL), authentication key, API protocol type (e.g., OpenAI compatible, Tongyi Qianwen native, etc.), model name, parameter size (e.g., 14B, 30B), supported request types, and the business line it belongs to (e.g., outpatient, inpatient, research).

[0028] After verifying and confirming the connectivity of the submitted metadata, the metadata is written to the persistent database and synchronized to the remote cache to form a list of backend services for subsequent scheduling phases to query and filter.

[0029] When an instance goes offline or is abnormally offline, update or clear the corresponding metadata record in the remote cache to ensure the real-time status of the inventory.

[0030] Step S2: Analyze and extract features based on the user's medical request to determine the request feature tags of the medical request; The specific method and process for quickly analyzing the request content and generating request feature tags after receiving a medical request is as follows: These feature tags will be used in the subsequent scheduling phase to match and filter against the metadata of model service instances to determine candidate large model service instances that meet the criteria. The specific steps are as follows: Step S2.1: Receive the user's medical request and extract the set of prompt words for the medical request; Receive the user's medical request, extract all characters or the first N characters of the prompt words in the medical request, and form a prompt word set; where N is a preset value, and in this embodiment N=50.

[0031] Step S2.2: Calculate the request fingerprint based on the prompt word set, and retrieve matching historical request profile records from the pre-built vector database according to the request fingerprint; Request fingerprint matching: Calculate the MD5 hash value of the character sequence (e.g., 50 characters) of the prompt set as the request fingerprint.

[0032] Using the request fingerprint as the query key, search in a pre-built vector database (such as Milvus) to see if there is a matching historical request profile record; The vector database stores the feature vectors of historical requests and their associated historical request profile records. These historical request profile records are in a structured format and include at least the task type, the magnitude of recommendation model parameters, the task complexity level, the latency tolerance level, and the business scenario to which they belong.

[0033] In addition, the vector database maintains a dedicated key-value mapping storage area to record the mapping relationship between the hash values ​​of historical requests and the corresponding historical request profile records. This storage area can be implemented using a high-performance key-value cache. The storage structure is as follows: Key: Request fingerprint, i.e., the hash value calculated from the set of request prompt words. Value: The structured historical request profile record corresponding to this request; Meanwhile, the vector database stores feature vectors and their associated historical request profile records. Each record in the vector database contains two fields: the feature vector of each medical request and the corresponding historical request profile record. The historical request profile records here are the same data as those in the key-value cache; the difference lies in the retrieval method—the key-value cache supports precise retrieval by hash value, while the vector database supports retrieval by vector similarity.

[0034] Specifically, the matching process: If a record with the key "request fingerprint" exists in the key-value mapping storage area of ​​the vector database, a match is considered successful. A successful match means that the first N characters of the current request's prompt are exactly the same as the first N characters of a historical request's prompt, therefore the profile data from that historical request can be directly reused. The system reads the value corresponding to that key, thus obtaining the profile data for the current request.

[0035] If the key-value mapping storage area of ​​the vector database does not contain a record with the key "request fingerprint", it is considered a no-match. In this case, a vector similarity search can be performed as a supplement: the set of prompt words for the medical request is converted into feature vectors, and historical request profile records with similarity exceeding a preset threshold are searched in the vector database. If a historical request profile record that meets the conditions is found, it is considered a successful semantic match, and the feature vector is extracted and associated with the historical request profile record; if it is still not found, a no-match is confirmed, and the asynchronous profile generation process is triggered.

[0036] The criteria for determining a successful match are as follows: Hash exact match successful: A key exists in the key-value mapping storage area that is exactly equal to the hash value of the current medical request.

[0037] Vector similarity matching successful: There are historical request profile records in the vector database whose feature vectors have a similarity to the current medical request that exceeds a preset threshold, such as a cosine similarity greater than 0.95.

[0038] If either of the two matching methods is satisfied, the match is considered successful, and the corresponding historical request profile record can be obtained.

[0039] Step S2.3: If a match is successful, generate request feature tags based on the matched historical request profile records; When a matching historical request profile record exists in the vector database, that historical request profile record is read directly. It is in a structured format and contains at least the following fields: Task type: Identifies the type of medical task this request belongs to. Optional values ​​include generation, summary, matching, tool invocation, extraction, etc. Recommended model parameter scale: The recommended model size based on task complexity, such as 7B, 14B, 30B, etc. Task complexity level: such as high, medium, low; Latency tolerance level: Indicates the sensitivity of the request to response time. For example, low latency tolerance corresponds to real-time business, and high latency tolerance corresponds to batch processing business. Business scenarios include outpatient services, inpatient services, emergency services, and scientific research.

[0040] The entire historical request profile is recorded and used as the request feature tag for the current request, which is then attached to the request context. After this feature extraction is completed, the request carries this feature tag into the subsequent scheduling stage.

[0041] Step S2.4: If the match fails, the current medical request is temporarily labeled based on the preset default feature labels, and the default feature labels are used as the basis for large model scheduling; at the same time, a request profile is asynchronously generated in the background according to the prompt word set, and request feature labels are generated based on the request profile. When no matching historical profile record exists in the vector database, the system performs the following two parallel processing steps: Step S2.4.1: (Synchronization part): If the matching fails, the current medical request is temporarily labeled based on the preset default feature label, and the default feature label is used as the basis for large model scheduling; The current medical request is temporarily labeled with a preset default feature label. The default feature label adopts a conservative strategy, for example: task type is labeled as "general", recommendation model parameter magnitude is labeled as "medium", latency tolerance is labeled as "real-time", and the business scenario is labeled as "default". The current request carrying this default label immediately enters the subsequent scheduling stage without being blocked.

[0042] Step S2.4.2: (Asynchronous part): Simultaneously generate request profile records asynchronously in the background based on the prompt word set, and generate request feature tags based on the request profile records; The specific steps for triggering an asynchronous task in the background are as follows: Step S2.4.2.1: Based on the prompt word set, use a pre-trained classification and evaluation model to classify the task and determine the request profile record of the medical request; The set of prompts for medical requests is combined with a pre-set analysis instruction template and then sent to the classification and evaluation model.

[0043] The analysis instruction template is used to guide the model to output multi-dimensional analysis results in one go. An example template is: "Please analyze the following medical requests, determine their task type (generation / summary / matching / tool ​​call / extraction), task complexity (high / medium / low), recommended model parameter level (7B / 14B / 30B / 72B), latency tolerance (low / medium / high), and business scenario (outpatient / inpatient / emergency / research), and return the results in JSON format." For example, the JSON format is: {"type": "tool call", "recommended_model": "Qwen-30B", "complexity": "high", "delay_tolerance": "low", "scenario": "outpatient"}.

[0044] In this embodiment, the classification and evaluation model adopts the Qwen-14B-Instruct model deployed on two A800 computing cards. This model has been trained to perform multi-dimensional analysis and judgment on medical requests.

[0045] The classification and evaluation model analyzes the set of prompt words for medical requests and outputs a structured request profile record. The fields are consistent with the historical request profile records and include at least the task type, the magnitude of the recommendation model parameters, the task complexity level, the latency tolerance level, and the business scenario.

[0046] Step S2.4.2.2: Vectorize the set of prompt words to obtain the feature vector of the medical request; Specifically, the embedding model is invoked to vectorize the extracted set of prompt words, generating a feature vector for the medical request. In this embodiment, the Qwen-Embedding model is used.

[0047] Step S2.4.2.3: After associating the request profile record and the request feature vector, store them in the vector database, and generate request feature labels based on the request profile record; The generated request profile record is associated with the request feature vector and asynchronously written to the vector database.

[0048] This write operation does not block the current request. Subsequently, when a request with the same or similar fingerprint arrives again, it will be directly matched and the precise request profile will be returned.

[0049] This embodiment combines precise matching of request fingerprints with similarity retrieval from a vector database, allowing for the rapid reuse of historical request profile records and significantly reducing the classification time for similar requests. The asynchronous profile generation mechanism continuously stores the analysis results of new request types into the vector database without blocking current requests, enabling the system to have self-learning capabilities—as runtime increases, the profile hit rate gradually improves, classification accuracy and efficiency are continuously optimized, and the workload of manual annotation and rule maintenance decreases accordingly.

[0050] Step S3: Based on the performance metrics of each large model service instance during its service process, periodically calculate the dynamic load coefficient and load status of each large model service instance; This step is a specific method and process for adaptive scheduling after completing the extraction of medical request features, combined with the real-time performance status of various model service instances.

[0051] In healthcare scenarios, the response process of large model services can be decomposed into two stages: the initial response stage, from request issuance to the generation of the first response token, and the output generation stage, from initial token generation to the completion of all responses. The initial response stage mainly consumes input token pre-filling processing, request queuing, and model context loading, while the output generation stage mainly consumes token-by-token decoding computation. In scenarios such as real-time outpatient consultations, users are extremely sensitive to initial token latency, which directly affects the user experience; while in batch processing scenarios such as inpatient research data analysis, overall throughput efficiency is more important. Existing technologies typically only use end-to-end total latency as a metric, failing to distinguish the contribution of the two types of latency mentioned above, resulting in insufficient perception of instance load status and difficulty in supporting differentiated scheduling decisions.

[0052] To address the aforementioned issues, this embodiment introduces an independent record of the first token response time during the performance acquisition phase. This enables refined decomposition and load attribution of the response process for each large model service instance. The specific steps are as follows: Step S3.1: Based on standard test requests, warm up the large model service instance that is launched or registered for the first time to generate an initial performance baseline; When a large model service instance completes startup or is newly registered and launched, a preheating process is first executed to obtain the initial performance baseline data of the instance in an idle state.

[0053] Step S3.1.1: Send three types of pre-defined standard test requests sequentially to the newly started or registered large model service instance, among which, The first type of standard test request is a short text request, with 50 tokens input and 200 tokens output; The second type of standard test request is a Chinese text request, with 500 input tokens and 800 output tokens; The third type of standard test request is a long text request, with 2000 input tokens and 1500 output tokens. Each type of request is sent 10 times.

[0054] Step S3.1.2: Record the timestamp of each standard test request, the timestamp of the first response token generation, the timestamp of the complete response end, and the actual number of output tokens generated; calculate the first word latency, generation latency, and token generation rate. The first-word delay is equal to the timestamp of the first response token generation minus the timestamp of the request being sent. Generation latency equals the complete response end timestamp minus the first response token generation timestamp; The token generation rate is equal to the number of output tokens divided by the generation delay.

[0055] The token generation rate is equal to the number of output tokens divided by the end-to-end response latency.

[0056] Step S3.1.3: Take the average of the median first-word latency of all classes of standard test requests as the benchmark first-word latency of the large model service instance; take the average of the median generation latency of all classes of standard test requests as the benchmark generation latency of the large model service instance; take the average of the token generation rates of all classes of standard test requests as the benchmark token processing rate. Among them, the first word latency values ​​of the 10 standard test requests for each type are sorted from smallest to largest, and the arithmetic mean of the 5th and 6th values ​​is taken as the median first word latency value of that type of standard test request. For each type of standard test request, the 10 generation latency values ​​are sorted from smallest to largest, and the arithmetic mean of the 5th and 6th values ​​is taken as the median generation latency value for that type of standard test request. The average token generation rate of all classes of standard test requests, i.e., the average of the token generation rates of all 30 measurements.

[0057] Step S3.1.4: Use the baseline first-word latency, baseline generation latency, and baseline token processing rate as the performance baseline data for this instance. Using the unique identifier of the large model service instance as the key, store the performance baseline and the historical maximum request throughput that matches the large model service instance in local memory and remote cache to form the initial performance baseline.

[0058] Among them, the historical maximum request throughput of the same model with the same hardware configuration as the instance is obtained from historical operation data and stored in the remote cache.

[0059] Step S3.2: During the formal service phase, continuously acquire the end-to-end response latency, number of input tokens, and number of output tokens for each medical request processed by the large model service instance, and form a request-level performance record; S3.2.1: During the formal service phase, for each medical request that has been processed by the large model service instance, record the arrival timestamp of the medical request, the generation timestamp of the first response token, and the completion timestamp of the complete response, and calculate the first word latency and generation latency; The first-word delay is equal to the timestamp of the first response token generation minus the timestamp of the request arrival, and the generation delay is equal to the timestamp of the complete response end minus the timestamp of the first response token generation.

[0060] S3.2.2: Obtain the number of tokens corresponding to the user input content from the medical request data sent to the large model, as the number of input tokens; and obtain the number of tokens corresponding to the generated result content from the response data returned by the large model, as the number of output tokens.

[0061] S3.2.3: Combine the request arrival timestamp, first word delay, generation delay, number of input tokens, and number of output tokens into a request-level performance record; Specifically, request-level performance records are written to a fixed-length circular queue maintained in the memory of the large model service instance. When the queue is full, the new record overwrites the oldest record, ensuring the controllability of memory usage.

[0062] Step S3.3: Based on request-level performance records, periodically calculate the performance metrics for each large model service instance, and calculate the dynamic load coefficient and load status based on the performance metrics; The system deploys a separate timed calculation task on the scheduling module side. This task performs load factor calculations for each online instance in the cluster at a fixed interval of 30 seconds. The calculation steps are as follows: Step S3.3.1: Based on all request-level performance records of the large model service instance within the current preset period, calculate the average first-word latency, average generation latency, request throughput, average number of input tokens, and average number of output tokens as performance indicators; Average first-word latency: the sum of the first-word latencies of all requests divided by the total number of requests; Average generation latency: the sum of the generation latencies of all requests divided by the total number of requests; Request throughput: Total number of requests divided by period duration, e.g., 30 seconds; Average number of input tokens: the sum of the number of input tokens for all requests divided by the total number of requests; Average number of output tokens: the sum of the number of output tokens for all requests divided by the total number of requests.

[0063] Step S3.3.2: Normalize the average first-word delay, average generation delay and the baseline first-word delay and baseline generation delay in the initial performance baseline to obtain the normalized first-word delay factor and the normalized generation delay factor. Specifically, the normalized first-word delay factor = min(current average first-word delay ÷ baseline first-word delay, upper limit value), where the upper limit value can be set to 2.0 to prevent extreme values ​​from causing calculation deviations.

[0064] Normalized generation delay factor = min(current average generation delay ÷ baseline generation delay, upper limit), and the upper limit can also be set to 2.0.

[0065] Step S3.3.3: Normalize the request throughput with the historical maximum request throughput in the initial performance baseline to obtain the normalized throughput utilization.

[0066] Specifically, normalized throughput utilization = min(current request throughput ÷ historical maximum request throughput, 1.0).

[0067] Step S3.3.4: Perform a weighted summation of the normalized first-word delay factor, the normalized generation delay factor, and the normalized throughput utilization to obtain the dynamic load coefficient, and determine the corresponding load state based on the dynamic load coefficient. Dynamic load factor = First weight × Normalized first-word delay factor + Second weight × Normalized generation delay factor + Third weight × Normalized throughput utilization The sum of the first weight, the second weight, and the third weight is 1.

[0068] In this embodiment, for outpatient real-time consultation services, priority is given to ensuring the timeliness of the first character response, with a first weight of 0.5, a second weight of 0.2, and a third weight of 0.3. For inpatient research data analysis services, greater emphasis is placed on overall generation efficiency and throughput, with a first weight of 0.2, a second weight of 0.5, and a third weight of 0.3. Different business lines can be configured with different weight combinations, and the scheduling module selects the corresponding weight for calculation based on the business line to which the request belongs. The larger the dynamic load coefficient value, the heavier the current load and the slower the response of the instance.

[0069] For example, when the overload threshold is set to 0.7, a dynamic load factor below 0.3 indicates that the instance is in a light load state, between 0.3 and 0.7 indicates a normal load state, and above 0.7 indicates a heavy load or overload state. The scheduling process avoids routing new requests to heavy load instances. The overload threshold can be selected according to the actual situation.

[0070] Understandably, after calculating the dynamic load factor in each cycle, the dynamic load factor, load status, and various performance indicators for the current cycle will be updated and stored, and the expiration time of the dynamic load factor will be set for scheduling.

[0071] Specifically, the dynamic load coefficient and various performance indicators of each large model service instance are associated with the unique identifier of the large model service instance and updated to the local memory of the large model service instance. The performance metrics include average first-word latency, average generation latency, request throughput, average number of input tokens, and average number of output tokens.

[0072] The dynamic load factor and performance metrics of each large model service instance are synchronously written to the remote cache, and an expiration time is set for scheduling. The above metrics are synchronously written to the remote cache. The key format in the remote cache is "Instance ID:realtime_status", and the values ​​use a JSON structure to store the metric values ​​and update timestamps. The expiration time is set to 60 seconds, but other expiration times can be selected as needed. Similarly, different calculation periods can be chosen. An expiration time greater than the calculation period (30 seconds) ensures that the data remains valid until the next update cycle, while also ensuring that the status information of abnormal offline instances is automatically cleared after 60 seconds.

[0073] This embodiment continuously collects the first-word latency and generation latency of completed requests from each instance, calculates the normalized latency factor and throughput utilization at fixed intervals, and generates a dynamic load coefficient. This mechanism can quickly capture instance performance fluctuations, significantly improving the response speed of resource adaptation compared to traditional solutions that rely on manual configuration of static weights or simple heartbeat detection. Instance online / offline status and performance changes are automatically detected and cached by the system, eliminating the need for frequent manual adjustments to scheduling parameters and greatly reducing operational complexity.

[0074] Step S4: Based on the request feature tags and the backend service list, determine all candidate large model service instances, and determine the optimal large model service instance based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags. Step S4.1: Based on request feature tags, determine all candidate large model service instances from the backend service list, using the following selection rules: First, using the recommended model name and parameter magnitude in the request feature tags as the main matching conditions, we search for large model service instances in the backend service list that have the same or compatible model names and match or are similar in parameter magnitude. Secondly, using the task type in the request feature tags as an auxiliary condition, we filter large model service instances that support that task type. Finally, using the business scenario in the request feature tags as an auxiliary condition, we filter out large model service instances that serve the corresponding business lines.

[0075] After the above screening, all candidate large model service instances were determined.

[0076] Based on the task type, business scenario, and number of recommendation model parameters in the request feature tags, instances that meet the conditions are accurately selected from the service list, so that requests of different complexities and business attributes are assigned appropriately—complex tasks are assigned to large models to ensure quality, and simple tasks are assigned to small models to avoid wasting computing power.

[0077] Step S4.2: Obtain the dynamic load coefficients of all candidate large model service instances. Based on the dynamic load coefficients of each candidate large model service instance and the latency tolerance in the request feature tags, determine the optimal large model service instance according to the following rules: If the latency tolerance in the request feature tag is low tolerance, then the largest model service instance with the lowest dynamic load coefficient is selected as the optimal large model service instance. If the latency tolerance in the request feature tag is high tolerance, then the large model service instance with the highest dynamic load coefficient among those exceeding the preset overload threshold is selected as the optimal large model service instance.

[0078] For example, when the overload threshold is set to 0.7, a dynamic load factor below 0.3 indicates that the instance is in a light load state, between 0.3 and 0.7 indicates a normal load state, and above 0.7 indicates a heavy load or overload state. The scheduling module avoids routing new requests to the heavy load instance.

[0079] For real-time services with low latency tolerance, prioritize instances with the lowest load and fastest response to ensure the timeliness of critical services such as outpatient consultations; for batch services with high latency tolerance, select instances with higher load from those that are not overloaded to achieve overall cluster load balancing, improve resource utilization, and solve the problem that coarse scheduling cannot take into account the differentiated business needs.

[0080] By introducing request analysis and feature extraction steps into the scheduling chain, the system achieves for the first time automatic identification and labeling of medical request task types, complexity, latency tolerance, and their respective business scenarios. During scheduling decisions, a differentiated strategy is adopted based on the latency tolerance in the request feature tags: for low-latency-tolerance requests such as real-time outpatient consultations, the instance with the lowest load and fastest response is prioritized, effectively shortening the end-to-end response latency of high-priority tasks; for high-latency-tolerance requests such as inpatient research data analysis, instances with higher loads can be selected from those not overloaded, avoiding all requests concentrating on the least busy instance. This mechanism solves the problem of traditional scheduling strategies ignoring business differences and critical requests being blocked by low-priority tasks.

[0081] Example 2 This embodiment provides a medical large-scale model service instance scheduling system based on latency and traffic profiling, including: The service list generation module is used to obtain and store the metadata of all large model service instances and generate a backend service list. The medical request feature extraction module is used to analyze and extract features based on the user's medical requests, and determine the request feature tags of the medical requests; The instance load determination module is used to periodically calculate the dynamic load coefficient and load status of each large model service instance based on the performance indicators during the service process of each large model service instance. The instance scheduling module is used to determine all candidate large model service instances based on request feature tags and the backend service list, and to determine the optimal large model service instance based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags.

[0082] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0083] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0084] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0085] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the medical large-scale model service instance scheduling method based on latency and traffic profiling as described in Embodiment 1 above.

[0086] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the medical big data model service instance scheduling method based on latency and traffic profiling as described in Embodiment 1 above.

[0087] Example 5 This embodiment provides a computer program product or computer program, including computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the medical big data model service instance scheduling method based on latency and traffic profiling described in Embodiment 1 above.

[0088] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0093] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for scheduling large-scale medical model service instances based on latency and traffic profiling, characterized in that, include: Obtain and store the metadata of all large model service instances, and generate a backend service list; Based on the user's medical requests, feature extraction is performed to determine the request feature tags of the medical requests; Based on the performance metrics of each large model service instance during its service process, the dynamic load coefficient and load status of each large model service instance are periodically calculated. Based on request feature tags and the backend service list, all candidate large model service instances are identified, and the optimal large model service instance is determined according to the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags.

2. The medical large-scale model service instance scheduling method based on latency and traffic profiling as described in claim 1, characterized in that, The step of analyzing and extracting features based on the user's medical requests to determine the request feature tags for the medical requests includes: Receive the user's medical request and extract the set of prompt words for the medical request; The hash value is calculated based on the set of prompt words as the request fingerprint, and the historical request profile records that match are retrieved from the pre-built vector database based on the request fingerprint. If a match is found, request feature tags are generated based on the matched historical request profile records. If a match fails, the current medical request is temporarily labeled based on a preset default feature label, which serves as the basis for scheduling the large model. At the same time, a request profile is asynchronously generated in the background based on the prompt word set, and request feature labels are generated based on the request profile.

3. The medical large-scale model service instance scheduling method based on latency and traffic profiling as described in claim 2, characterized in that, If the matching fails, the current medical request is temporarily labeled based on the preset default feature label, and the default feature label is used as the basis for large model scheduling. Simultaneously, a request profile is asynchronously generated in the background based on the set of prompt words, and request feature tags are generated based on the request profile, including: If the match fails, the current medical request is temporarily labeled based on the preset default feature labels, and the default feature labels are used as the basis for large model scheduling. Simultaneously, request profile records are generated asynchronously in the background based on the set of prompt words, and request feature tags are generated based on the request profile records.

4. The medical large-scale model service instance scheduling method based on latency and traffic profiling as described in claim 1, characterized in that, The process of warming up large model service instances that are launched or registered for the first time based on standard test requests to generate an initial performance baseline includes: Based on standard test requests, large model service instances that are launched or registered for the first time are warmed up to generate an initial performance baseline; During the formal service phase, the end-to-end response latency, number of input tokens, and number of output tokens for each medical request processed by the large model service instance are continuously acquired to form a request-level performance record. Based on request-level performance records, the performance metrics of each large model service instance are calculated periodically, and the dynamic load coefficient and load status are calculated based on the performance metrics.

5. The medical large-scale model service instance scheduling method based on latency and traffic profiling as described in claim 4, characterized in that, Based on request-level performance records, performance metrics for each large model service instance are periodically calculated, and dynamic load coefficients and load status are calculated based on these performance metrics, including: Based on all request-level performance records of the large model service instance within the current preset period, calculate the average first-word latency, average generation latency, request throughput, average number of input tokens, and average number of output tokens as performance indicators. The average first-word delay and average generation delay are normalized to the baseline first-word delay and baseline generation delay in the initial performance baseline, respectively, to obtain the normalized first-word delay factor and the normalized generation delay factor. The request throughput is normalized by comparing it with the historical maximum request throughput in the initial performance baseline to obtain the normalized throughput utilization. The normalized first-word delay factor, the normalized generation delay factor, and the normalized throughput utilization are weighted and summed to obtain the dynamic load coefficient, and the corresponding load state is determined based on the dynamic load coefficient.

6. The medical large-scale model service instance scheduling method based on latency and traffic profiling as described in claim 1, characterized in that, Based on request feature tags and the backend service list, all candidate large model service instances are determined. Then, based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags, the optimal large model service instance is determined, including: Based on request feature tags, all candidate large model service instances are identified from the backend service list; If the latency tolerance in the request feature tag is low tolerance, then the largest model service instance with the lowest dynamic load coefficient is selected as the optimal large model service instance. If the latency tolerance in the request feature tag is high tolerance, then the large model service instance with the highest dynamic load coefficient among those exceeding the preset overload threshold is selected as the optimal large model service instance.

7. A medical large-scale model service instance scheduling system based on latency and traffic profiling, characterized in that, include: The service list generation module is used to obtain and store the metadata of all large model service instances and generate a backend service list. The medical request feature extraction module is used to analyze and extract features based on the user's medical requests, and determine the request feature tags of the medical requests; The instance load determination module is used to periodically calculate the dynamic load coefficient and load status of each large model service instance based on the performance indicators during the service process of each large model service instance. The instance scheduling module is used to determine all candidate large model service instances based on request feature tags and the backend service list, and to determine the optimal large model service instance based on the dynamic load coefficient of each candidate large model instance and the latency tolerance in the request feature tags.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the medical large model service instance scheduling method based on latency and traffic profiling as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the medical large model service instance scheduling method based on latency and traffic profiling as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps in the medical large-scale model service instance scheduling method based on latency and traffic profiling as described in any one of claims 1-6.

Images