A dialogue message queue management system based on a large model cluster

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a dialogue message queue management system based on a large model cluster. The system comprises a message preprocessing module, a message router, a plurality of independent priority queues, a resource scheduler, a dialogue state manager and a large model cluster. The message preprocessing module receives an external dialogue request, constructs a metadata object, assigns a priority queue label, calculates a static score and records a timestamp; the message router distributes the request to the corresponding priority queue according to the label; the resource scheduler executes a scheduling strategy, calculates a dynamic priority score, realizes load balancing, and retrieves historical context information from the dialogue state manager, combines the request and the context into a processing task and assigns the processing task to the large model cluster; the large model cluster returns a model response to the resource scheduler, and the dialogue state manager is responsible for context storage and updating. The various components of the system can efficiently process dialogue message requests through close cooperation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a dialogue message queue management system based on a large model cluster. BACKGROUND

[0002] With the wide application of large language models (LLM) in intelligent dialogue, content generation and other fields, using a large model cluster to cope with high concurrency requests has become an industry standard architecture. In such a multi-tenant service scenario, the existing technology generally uses a first-in-first-out (FIFO) message queue as the core request management mechanism. In this architecture, all arriving user dialogue requests are indiscriminately sent into the same queue, and the LLM computing cluster in the backend strictly consumes and processes them according to the order of receipt, thereby realizing the buffering and preliminary allocation of requests.

[0003] However, the above-mentioned homogeneous processing strategy based on the FIFO queue is difficult to adapt to the high-level needs of differentiated services and context continuity of modern LLM services, and its inherent defects are mainly manifested as follows: it cannot distinguish the priority of requests, which leads to the fact that high-priority requests (for example, requests from VIP customers or urgent system tasks) must wait until all ordinary requests before it are processed, causing unnecessary delay and seriously affecting the experience of key users. SUMMARY

[0004] The purpose of the present application is to introduce a comprehensive management mechanism including multi-level priority queues, dialogue state managers and dynamic resource schedulers, to realize fine management and efficient use of LLM computing resources by preprocessing and prioritizing the messages entering the system and intelligently scheduling the dialogue context.

[0005] In a first aspect, the embodiments of the present application provide a dialogue message queue management system based on a large model cluster, comprising:

[0006] A message preprocessing module is configured to receive dialogue message requests and construct structured metadata objects by fusing multi-dimensional information from the client, the authentication system and the request content itself; based on the metadata objects, dynamically allocate priority queue tags for the requests by querying a configurable mapping rule table; and determine the static basic score of the request based on the metadata objects and record the entry time stamp of the request;

[0007] A message router is configured to distribute the requests into corresponding priority queues according to the priority queue tags;

[0008] A dialogue state manager is configured to store and retrieve the historical context information of the dialogue according to the dialogue identifier;

[0009] A resource scheduler is used to perform the following steps: selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy; the decision basis of the scheduling strategy is any one of the following: the static priority of the queue; the static priority, static weight and dynamic credit value of the queue; the static priority of the queue and the current number of tokens in the token bucket; the static priority of the queue, real-time pressure status, system load status and estimated task complexity;

[0010] The dynamic priority score of the requests in the target priority queue is determined by the static base score of the request and the waiting time after enqueueing, and the request with the highest dynamic priority score is identified as the request to be processed.

[0011] The historical context information of the pending request is retrieved and associated from the dialog state manager to form a processing task; a target large model instance is selected based on the load balancing and task matching principle, and the processing task is assigned to the target large model instance.

[0012] Optionally, the step of constructing a structured metadata object by integrating multi-dimensional information from the client, authentication system, and the request content itself includes:

[0013] The user database is queried using authentication credentials to obtain static business attributes related to the service contract, including user identifier, service level, and rate limiting policy.

[0014] Parse the explicit business fields in the request body and normalize the parsed fields, including mapping different task types to a standard identifier and converting priority fields of different formats into a uniform value.

[0015] The request content is analyzed by an inference engine to generate implicit data attributes, which include at least one of the following: task type inference based on text length, emergency sign recognition based on keyword matching, and task complexity score based on a prediction model.

[0016] Based on the acquired static business attributes, the normalized explicit business fields, and the implicit data attributes, a structured metadata object is generated; wherein, the structured metadata object includes: service contract identifier, task type identifier, emergency flag, task complexity score, user identifier, and rate limiting strategy.

[0017] Optionally, the message preprocessing module includes a sub-mapping module for request-to-queue mapping, the sub-mapping module including a configurable mapping rule table; the sub-mapping module is configured to perform the following steps:

[0018] The configurable mapping rule table is queried through the mapping function. Based on the service contract identifier, task type identifier, and emergency flag in the metadata object, the corresponding priority queue index is determined and used as the priority queue label.

[0019] When the combination of the current service contract identifier, task type identifier, and emergency flag is not found in the mapping rule table, a fallback mapping mechanism is triggered, and at least one of the following operations is performed: the request is assigned to a preset default priority queue; a downgraded matching operation is performed: according to the preset matching hierarchy, the service contract identifier is matched first; if no match is found, the task type identifier is matched; if no match is found, the emergency flag is matched; the request is rejected and an error message is returned to the request source.

[0020] Optionally, determining the static basic score of the request based on the metadata object includes:

[0021] Based on the service contract identifier in the metadata object, the first preset score table is queried to obtain the corresponding service contract score; based on the task type identifier in the metadata object, the second preset score table is queried to obtain the corresponding task type score; based on the emergency flag in the metadata object, the corresponding emergency flag score is determined; the service contract score, task type score, and emergency flag score are weighted and summed according to preset weights, and the result is used as the static base score.

[0022] Optionally, determining the dynamic priority score of multiple requests in the target priority queue by using the static base score of the request and the waiting time after the request is enqueued includes: obtaining the static base score from the message preprocessing module; calculating the waiting time of the request from the enqueuing timestamp; and inputting the waiting time into a configurable dynamic aging function to obtain a dynamic aging factor.

[0023] The static base score and the dynamic aging factor are weighted and summed to obtain the dynamic priority score; wherein the dynamic aging function is any one of the following functions: a linear function, wherein the dynamic aging factor is proportional to the waiting time; a bounded function, wherein the dynamic aging factor increases with the waiting time but has a preset upper limit; a nonlinear function, wherein the dynamic aging factor increases nonlinearly with the waiting time.

[0024] Optionally, selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy includes: sequentially polling the status of each queue in descending order of their static priorities; and selecting the first queue that is not empty as the target priority queue.

[0025] Optionally, the step of selecting a target priority queue from the multiple independent priority queues according to a configurable scheduling strategy includes: presetting a corresponding static weight for each queue, and setting the initial value of the dynamic credit counter of each queue to the static weight value of that queue; polling and checking all non-empty queues, comparing the current dynamic credit counter values ​​of each queue, and selecting the queue with the largest dynamic credit counter value as a candidate queue.

[0026] If multiple candidate queues exist, the queue with the highest static priority is selected as the final target priority queue. If only one candidate queue exists, that candidate queue is directly designated as the target priority queue. After retrieving the request to be processed from the target priority queue, the dynamic credit counter value of the target priority queue is subtracted from the sum of the static weights of all queues. The credit counters of all queues in the system are then incremented by their respective static weight values. Finally, the process of polling and checking all non-empty queues is returned to form a continuous scheduling loop.

[0027] Optionally, the step of selecting a target priority queue from the multiple independent priority queues according to a configurable scheduling strategy includes: associating each queue with a token bucket and configuring a corresponding token generation rate and token bucket capacity; periodically generating tokens for each queue, with the current number of tokens for each queue not exceeding its token bucket capacity; when scheduling is required, selecting qualified queues among all non-empty queues whose current number of tokens is sufficient to cover the estimated cost of the task at the head of the queue; selecting the queue with the highest static priority from the qualified queues as the target priority queue; and returning to the step of selecting qualified queues among all non-empty queues whose current number of tokens is sufficient to cover the estimated cost of the task at the head of the queue when scheduling is required, to form a continuous scheduling loop.

[0028] Specifically, after retrieving the pending request from the target priority queue, the estimated token cost of the pending request is deducted from the current token count of the target priority queue.

[0029] Optionally, selecting a target priority queue from the plurality of independent priority queues according to a configurable scheduling policy includes:

[0030] A scheduling value score is calculated for each non-empty queue. The scheduling value score is calculated by comprehensively evaluating the following multiple dimensions: the queue's static priority, the queue's real-time pressure status, the system's real-time load status, and the estimated complexity of the tasks. Among them, the queue's real-time pressure status comprehensively considers the queue length and the average waiting time of the tasks; the system's real-time load status is obtained based on the overall resource utilization of the large model cluster; and the estimated complexity of the tasks is estimated by a prediction model for the resource requirements of multiple tasks at the head of the queue.

[0031] The indicators of each dimension are normalized to eliminate differences in units; based on the normalized indicators of each dimension, the final scheduling value score of each queue is calculated according to the preset scheduling value score; the queue with the highest scheduling value score is selected as the target priority queue.

[0032] Optionally, the selection of the target large model instance based on load balancing and task matching principles includes:

[0033] Monitor the real-time running status of all instances in the large model cluster, and obtain the current load indicators, available hardware resource information, and loaded model capability information of each instance;

[0034] Based on the specific requirements of the task to be processed, the matching degree between each instance and the task is evaluated, and a matching degree score is generated; based on the acquired real-time running status, the load score of each instance is calculated, where the lower the current load index and the more available hardware resources, the higher the load score of the instance; the matching degree score and the load score of each instance are combined and weighted according to preset weights to generate a comprehensive evaluation score; the instance with the highest comprehensive evaluation score is selected as the target large model instance.

[0035] The various components of this system collaborate closely to form a highly efficient dialogue message processing pipeline. When a user request arrives at the system, the message preprocessing module first performs in-depth analysis and feature extraction. Then, the message router performs precise classification and distribution. Priority queues act as buffer layers to smooth traffic fluctuations. Finally, the resource scheduler optimizes the scheduling and execution of requests based on intelligent algorithms, while the dialogue state manager ensures the continuity and consistency of the dialogue throughout the process. This architectural design enables the system to achieve low latency, high throughput, and high quality of service in high-concurrency, multi-tenant scenarios, providing reliable infrastructure support for large-scale dialogue services. Through flexible and configurable scheduling strategies and adaptive resource management mechanisms, the system can dynamically adjust according to the needs of different business scenarios, demonstrating excellent adaptability and scalability. Attached Figure Description

[0036] Figure 1 A schematic diagram illustrating the core components of the system and the data flow between each component, provided in an embodiment of the present invention;

[0037] Figure 2 A flowchart illustrating a strategy for managing a dialogue message queue is provided in this embodiment of the invention.

[0038] Figure 3 This is a schematic diagram of the structure of a dialogue message queue management system based on a large model cluster, provided in an embodiment of the present invention. Detailed Implementation

[0039] With the development of artificial intelligence technology, large language models (LLMs) are increasingly widely used in fields such as natural language processing and intelligent dialogue. In commercial applications, large model clusters are typically used to provide services to handle concurrent requests from a massive number of users.

[0040] Currently, in multi-tenant, high-concurrency service scenarios, large-scale clusters generally face the following challenges: When a large number of dialogue requests flood into the system in a short period of time, traditional service architectures typically use First-In, First-Out (FIFO) message queues to cache and process these requests. However, this simple queue management strategy has a significant drawback: indiscriminate queuing. FIFO queues treat all requests equally, failing to differentiate between the priorities of different users or tasks. This results in high-priority requests (e.g., requests from VIP customers or urgent system tasks) having to wait for all ordinary requests preceding them to be processed, causing unnecessary delays and severely impacting the experience of critical users.

[0041] This invention aims to address the technical problems of high response latency, unfair resource allocation, inability to guarantee a superior user experience, and easy loss of dialogue context caused by the use of traditional message queues in existing large-scale model clusters when handling high-concurrency dialogue requests. This invention provides a dialogue message queue management strategy and system based on large-scale model clusters. It introduces a comprehensive management mechanism including multi-level priority queues, a dialogue state manager, and a dynamic resource scheduler. By preprocessing and prioritizing messages entering the system, and combining this with intelligent scheduling based on dialogue context, it achieves fine-grained management and efficient utilization of LLM computing resources.

[0042] The main objectives of this invention include: First, optimizing resource allocation. Dynamically allocating LLM computing resources based on request priority ensures high-priority tasks are executed first. Second, improving response speed. Significantly reducing the average waiting time and response time for high-priority and interactive dialogue requests. Third, ensuring dialogue continuity. Effectively maintaining the context information of multi-turn dialogues through a dialogue state manager ensures that requests within the same dialogue are processed in an orderly manner. Fourth, providing flexible priority management. Supporting dynamic priority allocation of requests based on multiple dimensions such as user level and task type to meet the needs of different business scenarios. Fifth, improving system stability and throughput. In high-concurrency scenarios, preventing system overload and improving overall service capabilities through intelligent scheduling and load balancing.

[0043] To ensure clarity, the overall technical solution of this invention will be described in detail below, taking into account the system architecture diagram provided by this invention. Figure 1This is a schematic diagram of the overall architecture of a dialogue message queue management system based on a large model cluster, as described in this invention. The diagram illustrates the connections and data flow between the message preprocessing module, message router, multi-level priority queue, resource scheduler, dialogue state manager, and large model (LLM) instance cluster.

[0044] The core components of this system and the data flow between each component, such as Figure 1 As shown, the message preprocessing module is the system's data entry point. It is responsible for receiving raw dialogue requests from different clients (Web applications, mobile apps, and API calls). Its core function is to parse the request metadata and assign a priority tag to each message request according to preset rules. The criteria for priority determination may include, but are not limited to: user identification: distinguishing between VIP users, regular users, internal test users, etc.; API key levels corresponding to different paid service levels; and task types distinguishing between real-time interactive dialogues, asynchronous batch text generation tasks, etc.

[0045] Requests containing metadata such as "urgent" are considered requests. After processing, each request object will contain the original information and a clear priority label (e.g., priority: HIGH).

[0046] Message Router: This module's function is relatively simple and straightforward. It receives priority-tagged message requests from the message preprocessing module and accurately distributes them to the corresponding priority queues based on their priority tags. For example, requests labeled HIGH are sent to the high-priority queue, while those labeled MEDIUM go to the medium-priority queue.

[0047] Multi-level priority queues: This is the core data structure of this invention. The system sets up multiple independent queues, each corresponding to a priority level. A typical implementation can be divided into three levels:

[0048] High-priority queues are used to handle requests requiring the fastest response, such as real-time conversations from VIP paying users or API calls in critical business processes. Medium-priority queues handle routine interaction requests from regular users, ensuring basic service quality. Low-priority queues handle non-real-time, delayable background tasks, such as batch data analysis and text report generation. Tasks in this queue are typically processed when server resources are idle.

[0049] The resource scheduler is the brain of this invention, responsible for intelligently allocating requests in the queues to the backend Large Model (LLM) instance cluster. Its working logic is as follows: the scheduler does not strictly follow a pattern of processing the next queue only when a queue is empty, but instead employs weighted round-robin or other more complex strategies. For example, the scheduler can pull tasks from different queues in a ratio of "processing 5 high-priority requests, then 2 medium-priority requests, and finally 1 low-priority request." This ensures the priority of high-priority tasks while preventing low-priority tasks from being completely "starved."

[0050] State awareness and load balancing: The scheduler monitors the health and load of each instance in the LLM instance cluster in real time. When assigning new tasks, it selects the idle instance with the lowest current load to process them, thereby achieving effective load balancing and maximizing the cluster's throughput.

[0051] Collaboration with the conversation state manager: Before scheduling a request, the scheduler queries the conversation state manager based on the conversation_id in the request to obtain the complete historical context of the conversation.

[0052] The conversation state manager is responsible for persistently storing and managing the context information of each individual conversation. It can be implemented using a key-value database (such as Redis) or an in-memory database. The key is a globally unique `conversation_id`, and the value stores the conversation's historical messages, user information, session state, etc.

[0053] Collaborative Work: When the resource scheduler is ready to send a request (e.g., a user's follow-up question) to an LLM instance, it first retrieves the corresponding historical conversation record from the state manager, carrying the conversation_id. Then, it sends this historical record along with the current user request to the LLM instance. After the LLM processes and generates a response, the system updates the conversation state manager with both the request and response, thus maintaining context continuity. This ensures that even if consecutive requests for the same conversation are assigned to different LLM instances, consistent and coherent responses are obtained.

[0054] As described above, the dialogue message queue management system based on a large model cluster provided by this invention has the following operational characteristics: First, clear data flow. Requests flow according to the path of "preprocessing → routing → queuing → scheduling → processing → updating". Second, clearly defined component responsibilities. Each module collaborates to form a complete processing chain. Third, end-to-end state maintenance. The dialogue state manager ensures the continuity of multi-turn dialogues. Fourth, dynamic resource allocation. The resource scheduler intelligently allocates computing resources based on the system state. Fifth, priority guarantee. A multi-level queue mechanism ensures that high-priority requests are processed promptly. This embodiment demonstrates the complete processing flow of the system from request reception to the final response. The arrows between components clearly indicate the data flow and control relationships, reflecting the system's technical advantages in ensuring service quality and dialogue continuity in high-concurrency scenarios.

[0055] Furthermore, to ensure clarity in the description of the solution, the embodiments of the present invention will be further explained below in conjunction with a flowchart of the strategy for managing dialogue message queues. For example... Figure 2 This is a flowchart illustrating the strategy for managing dialogue message queues according to the present invention. The diagram details the complete processing steps of a user dialogue request, from its entry into the system, through preprocessing, routing, enqueueing, scheduling, LLM processing, until the final response is returned. A complete dialogue request processing flow is as follows:

[0056] Step 1: Request Reception and Preprocessing. The user initiates a request through the client, containing user credentials, conversation_id, and message content. The request first reaches the message preprocessing module, which appends a priority tag (e.g., priority: MEDIUM) to the request based on information such as user credentials.

[0057] Step 2: Routing and Enqueuing. The message router reads the MEDIUM priority tag of the request and then puts the request object into the medium priority queue to await processing.

[0058] Step 3: The scheduler pulls requests. The resource scheduler, based on its internal weighted scheduling algorithm, polls the medium-priority queue and retrieves one or more requests from it.

[0059] Step 4: Context Acquisition and Instance Allocation. For each retrieved request, the scheduler resolves its `conversation_id`. It queries the conversation state manager to obtain all historical conversation contexts associated with that `conversation_id`. Simultaneously, the scheduler queries the current state of the LLM cluster and selects the optimal (e.g., the most idle) LLM instance.

[0060] Step 5: LLM Processing. The scheduler merges the acquired dialogue context with the current user request, sending it as a complete processing task to the selected LLM instance. The LLM instance performs reasoning based on the complete context information and generates a response.

[0061] Step 6: State Update and Result Return. The LLM instance returns the generated response to the system. The system first updates the conversation state manager with the user request and LLM response for use in the next conversation. Then, the final response is returned to the user client over the network.

[0062] This invention, through an innovative architecture combining a multi-level priority queue with a dialogue state manager, and unified intelligent coordination by a resource scheduler, successfully solves the three core pain points of traditional message queue systems in high-concurrency scenarios: head-of-line blocking, context loss, and uneven resource allocation. This system significantly reduces the response latency of high-priority tasks, ensures the continuity and accuracy of multi-turn dialogues, and dynamically optimizes the resource utilization of large model clusters. Thus, while guaranteeing a superior user experience, it comprehensively improves the system's throughput, stability, and service quality.

[0063] After a detailed introduction to the core components of this system, the data flow between each component, and the flowchart of the dialogue message queue management strategy, the following will elaborate on a dialogue message queue management system based on a large model cluster provided by an embodiment of the present invention. Figure 3 As shown in the figure, an embodiment of the present invention provides a dialogue message queue management system based on a large model cluster, which may include:

[0064] The message preprocessing module 310 is used to receive dialogue message requests and construct a structured metadata object by integrating multi-dimensional information from the client, authentication system and the request content itself; based on the metadata object, it dynamically assigns priority queue labels to the request by querying a configurable mapping rule table; and determines the static base score of the request based on the metadata object and records the request's enqueue timestamp.

[0065] Message router 320 is used to distribute requests to the corresponding priority queues based on priority queue labels;

[0066] Dialogue state manager 330 is used to store and retrieve historical context information of a dialogue based on a dialogue identifier;

[0067] Resource scheduler 340 is used to perform the following steps: select a target priority queue from multiple independent priority queues according to a configurable scheduling policy; the decision basis for the scheduling policy is any of the following: the static priority of the queue; the static priority, static weight and dynamic credit value of the queue; the static priority of the queue and the current number of tokens in the token bucket; the static priority of the queue, real-time stress status, system load status and estimated task complexity;

[0068] The dynamic priority score of the requests in the target priority queue is determined by the static base score of the request and the waiting time after enqueueing, and the request with the highest dynamic priority score is identified as the request to be processed.

[0069] The historical context information of the pending requests is retrieved and associated from the dialog state manager to form processing tasks; a target large model instance is selected based on load balancing and task matching principles, and processing tasks are assigned to the target large model instance.

[0070] like Figure 3 As shown in the figure, the present invention provides a dialogue message queue management system based on a large model cluster. It adopts a layered design and a modular architecture. The components communicate with each other through clearly defined interfaces, and together they form an efficient and reliable dialogue message processing platform.

[0071] The message preprocessing module 310 serves as the system's front-end entry point and the first gatekeeper for request processing, undertaking crucial request parsing and preprocessing functions. This module adopts a multi-stage processing pipeline architecture, which specifically includes: a metadata construction unit, a queue mapping unit, and a static scoring unit.

[0072] The metadata construction unit builds a complete request profile by integrating information from three different dimensions: explicit parameters obtained from the client (such as task type identifiers and user-defined tags); user attributes queried from the authentication system (including service level, permission configuration, and rate limiting policies); and implicit features obtained through intelligent analysis of the request content itself (such as identifying urgency through natural language processing and inferring task complexity through text length analysis). This multi-source information, after standardization, is organized into a structured metadata object, providing a comprehensive and accurate data foundation for subsequent scheduling decisions.

[0073] The queue mapping unit, based on the constructed metadata object, dynamically maps requests to priority queues by querying a configurable mapping rule table. This mapping rule table supports matching based on a combination of conditions across multiple dimensions, including service contracts, task types, and urgency flags, and allows system administrators to flexibly adjust mapping strategies according to actual business needs. For example, real-time chat requests from enterprise VIP users can be mapped to the highest priority queue, while batch processing tasks from free users can be mapped to the lowest priority queue.

[0074] The static scoring unit is responsible for calculating a static base score for each request based on metadata objects. This score comprehensively considers multiple factors such as the user's service level, task type characteristics, and urgency, and derives a quantified priority indicator through a preset weighted calculation formula. Simultaneously, this unit accurately records the timestamp of each request entering the system, providing a time benchmark for subsequent calculations of waiting time and for implementing dynamic priority adjustments.

[0075] The message router 320, serving as the system's traffic distribution hub, employs a lightweight and highly efficient design. This component receives categorized requests from the message preprocessing module and performs precise routing based on the priority queue labels they carry.

[0076] The router maintains direct connection channels with each priority queue internally, ensuring that requests are delivered to the target queue with minimal latency. To ensure system stability, the router also implements traffic monitoring and overload protection mechanisms. When a severe backlog is detected in a priority queue, the routing policy can be temporarily adjusted to prevent the congestion of a single queue from affecting the overall system performance.

[0077] The Dialogue State Manager 330 is a core component ensuring dialogue continuity, employing a distributed architecture to support high concurrency. This manager uses dialogue identifiers to create an index, maintaining a complete state record for each independent dialogue session.

[0078] The context storage engine employs optimized data serialization formats and compression algorithms to efficiently store information such as dialogue history, user preferences, and session metadata. Storage supports automatic expiration and cleanup mechanisms to avoid unlimited resource consumption. A real-time retrieval interface provides the resource scheduler with low-latency context query services. When the scheduler is ready to process a dialogue request, it can quickly retrieve the complete history of the dialogue through this interface, ensuring that large models can generate accurate responses based on sufficient context information. A state synchronization mechanism ensures data consistency in a distributed environment. When multiple requests access the same dialogue context simultaneously, the manager uses locking mechanisms and version control to guarantee the atomicity and consistency of data operations.

[0079] Resource scheduler 340 is the system's intelligent decision-making center, employing a multi-level scheduling strategy and an adaptive resource allocation algorithm. This scheduler achieves refined resource management through the following three core stages:

[0080] (1) Queue Selection Phase. The scheduler selects the most suitable target queue from multiple independent priority queues according to the scheduling policy configured in the system. The system supports four advanced scheduling policies, each based on a different combination of scheduling factors: a strict priority policy based on the static priority of queues, ensuring that high-priority queues have absolute priority; an enhanced weighted round-robin policy based on static weights and dynamic credit values, which ensures both priority and fairness; a fair queue policy based on the token bucket algorithm, which provides guaranteed processing bandwidth for each queue; and a dynamic adaptive policy based on multi-dimensional real-time indicators, which comprehensively considers factors such as queue pressure, system load, and task complexity to achieve the globally optimal scheduling decision.

[0081] (2) Request Selection Phase. After determining the target priority queue, the scheduler selects specific requests to be processed from the queue using a dynamic priority algorithm. This algorithm combines the static base score of the request with the waiting time factor after enqueuing, and ensures that the priority of each request increases appropriately with the increase of waiting time through the dynamic priority score calculation formula, effectively preventing the "starvation" phenomenon of low-priority requests. The scheduler selects the request with the highest score as the request to be processed, which takes into account both the inherent importance of the request and the fairness of waiting.

[0082] (3) Task Assignment Phase. The scheduler retrieves the complete historical context information of the pending requests from the dialog state manager and combines this context with the current request to form a complete processing task. Then, based on load balancing and task matching principles, it selects the most suitable target instance from the large model cluster. The load balancing algorithm considers the current load rate, resource utilization, network latency, and other indicators of each instance; the task matching algorithm evaluates the instance's professional capabilities, model version, hardware configuration, and suitability for the current task. Finally, the scheduler assigns the processing task to the selected optimal instance and monitors the execution status of the task.

[0083] The various components of this system collaborate closely to form a highly efficient dialogue message processing pipeline. When a user request arrives at the system, the message preprocessing module first performs in-depth analysis and feature extraction. Then, the message router performs precise classification and distribution. Priority queues act as buffer layers to smooth traffic fluctuations. Finally, the resource scheduler optimizes the scheduling and execution of requests based on intelligent algorithms, while the dialogue state manager ensures the continuity and consistency of the dialogue throughout the process. This architectural design enables the system to achieve low latency, high throughput, and high quality of service in high-concurrency, multi-tenant scenarios, providing reliable infrastructure support for large-scale dialogue services. Through flexible and configurable scheduling strategies and adaptive resource management mechanisms, the system can dynamically adjust according to the needs of different business scenarios, demonstrating excellent adaptability and scalability.

[0084] exist Figure 3 Based on the illustrated embodiment, as one implementation of this invention, a structured metadata object is constructed by integrating multi-dimensional information from the client, the authentication system, and the request content itself. This may include the following steps, namely steps a1 to a4:

[0085] Step a1: Query the user database using authentication credentials to obtain static business attributes related to the service contract. The static business attributes include user identifier, service level, and rate limiting policy.

[0086] Specifically, the system queries the user database using authentication credentials (such as API Key, user token, or session identifier) ​​in the request to obtain static business attributes related to the service contract. In practice, the system first extracts authentication credentials from the Authorization header or authentication parameters of the request, and then initiates a query request to the user center or authentication service. After verifying the validity of the credentials, the system retrieves the user's static business attributes from the user configuration database, billing system, or service level management system. These attributes include, but are not limited to, the user's unique identifier, service level (such as S_0 internal system, S_1 enterprise VIP, S_2 regular paying user, etc.), and the corresponding rate limiting policy. These static attributes provide the basic service framework and permission boundaries for user requests.

[0087] Step a2: Parse the explicit business fields in the request body and normalize the parsed fields, including mapping different task types to a standard identifier and converting priority fields of different formats into a uniform value.

[0088] Specifically, the system performs deep parsing and standardization transformation on explicit business fields in the request body. During implementation, the system first parses the HTTP request headers, URL query parameters, and request body content (such as a JSON-formatted request body) to extract the explicitly provided business parameters from the client. Then, the system performs normalization processing: for task type fields, the system uses a pre-defined mapping table to uniformly convert different expressions into a standard identifier. For example, it maps different expressions such as "batch_summarization," "summary," and "text summary" to the standard task type identifier "T_2." For priority fields, the system identifies various priority representations (such as "high" / "low," numerical priorities 1-10, urgency flags, etc.) and converts them into a unified numerical representation to ensure consistency in subsequent processing.

[0089] Step a3: Analyze the request content through the inference engine to generate implicit data attributes. The implicit data attributes include at least one of the following: task type inference based on text length, emergency sign recognition based on keyword matching, and task complexity score based on prediction model.

[0090] Specifically, the inference engine performs multi-dimensional intelligent analysis of the request content to generate implicit data attributes that reflect the inherent characteristics of the request. Specific implementations include: task type inference based on text length, automatically inferring the task type by calculating the number of characters or tags in the request prompt text and combining it with preset threshold rules; for example, automatically identifying it as a long text batch task when the text length exceeds 4000 characters; emergency flag recognition based on keyword matching, using a multi-pattern matching algorithm to identify emergency-related keywords (such as "emergency," "URGENT," "ASAP," etc.) in the request content and dynamically setting emergency status flags; and task complexity scoring based on a prediction model, using a pre-trained lightweight machine learning model (such as a gradient boosting decision tree or a small neural network) to analyze the request text, predict the task's computational complexity and resource requirements, and output a normalized score between 0 and 1, providing a quantitative basis for subsequent resource scheduling.

[0091] Step a4: Based on the acquired static business attributes, normalized explicit business fields, and the implicit data attributes, a structured metadata object is generated. This structured metadata object includes at least: a service contract identifier, a task type identifier, an urgency flag, a task complexity score, a user identifier, and a rate limiting policy.

[0092] Based on the processing results of the first three steps, the system fuses multi-source data to generate a complete structured metadata object. During implementation, the system integrates the static business attributes obtained through authentication, the normalized explicit business fields, and the implicit data attributes generated by intelligent inference, encapsulating them according to a predefined data pattern to form a standardized metadata object. This object contains at least the following core fields: service contract identifier, clearly identifying the user's service level and business permissions; task type identifier, providing standardized task classification information; emergency flag, indicating the urgent processing needs of the request; and task complexity score, providing a quantitative estimate of computing resource requirements. The final generated metadata object provides a comprehensive and accurate data foundation for subsequent priority determination, queue allocation, and resource scheduling. Through the collaborative work of these four steps, this embodiment of the invention achieves a complete transformation process from multi-dimensional information fusion to a structured metadata object, significantly improving the system's adaptability to complex business scenarios and the accuracy of scheduling decisions.

[0093] Based on the above embodiments, as one implementation of the present invention, the message preprocessing module includes a sub-mapping module for requesting to a queue, the sub-mapping module including a configurable mapping rule table; the sub-mapping module is configured to perform the following steps, namely steps b1 and b2:

[0094] Step b1: Query the configurable mapping rule table through the mapping function, and determine the corresponding priority queue index based on the service contract identifier, task type identifier and emergency flag in the metadata object, which serves as the priority queue label.

[0095] The mapping submodule queries the configurable mapping rule table through the mapping function MapToQueue(ServiceTier, TaskType, EmergencyFlag) and determines the priority queue index based on three core dimensions in the metadata object. An example of the mapping rule table structure is shown below:

[0096]

[0097] After receiving the metadata object containing the service contract identifier "S_1", the task type identifier "T_0" and the emergency flag "true", the mapping function precisely matches the combination in the rule table, finds the corresponding queue index 0, which is the highest priority queue, and outputs this index as the priority queue label.

[0098] Step b2: When the combination of the current service contract identifier, task type identifier, and emergency flag is not found in the mapping rule table, a fallback mapping mechanism is triggered, and at least one of the following operations is performed: the request is assigned to a preset default priority queue; a downgraded matching operation is performed: according to the preset matching hierarchy, the service contract identifier is matched first; if no match is found, the task type identifier is matched; if no match is found, the emergency flag is matched; the request is rejected and an error message is returned to the request source.

[0099] Specifically, when the complete combination of the current service contract identifier, task type identifier, and emergency flag is not found in the mapping rule table, the system triggers a rollback mapping mechanism to ensure that the request can be processed appropriately.

[0100] The first approach is the default queue allocation strategy. For unknown request combinations that cannot be matched, the system allocates them to a preset default priority queue. The default queue is usually set as a medium-priority queue or a low-priority queue, which ensures that requests can be processed while preventing unknown requests from occupying high-priority resources.

[0101] The second method is a downgraded matching process. The system performs downgraded matching according to a preset three-level matching hierarchy. First-level matching: Prioritizes matching the service contract identifier, ignoring the task type and urgency flag. For example, for the combination (S_1, T_3, true), the (S_1, *, *) rule is searched first. Second-level matching: If no match is found for the service contract identifier, the task type identifier is matched, ignoring the service contract and urgency flag. For example, the (*, T_3, *) rule is searched. Third-level matching: If neither of the first two levels matches, only the urgency flag is matched. For example, the (*, *, true) rule is searched.

[0102] The third approach is a request rejection mechanism. Specifically, when all the above matching strategies fail, the system rejects the request and returns standardized error information to the request source, including an error code, error description, and suggested action. For example, it might return: "Error code: MAP_001, Description: Unable to map the request to the appropriate processing queue, Suggestion: Contact the system administrator to update the mapping rule table."

[0103] This embodiment achieves the following technical advantages through multi-dimensional mapping and intelligent fallback mechanisms: By combining queries based on service contracts, task types, and urgency flags, it ensures that requests are assigned to the most suitable priority queue. The multi-level fallback mechanism guarantees system stability when facing unknown request combinations, preventing request loss or system anomalies due to mapping failures. Furthermore, the configurable mapping rule table supports dynamic updates, enabling the system to quickly adapt to changes in business needs without modifying core code. Additionally, recording mapping failures provides data support for system optimization and rule table improvement. Through this embodiment, the system can achieve intelligent and accurate request-to-queue mapping in complex multi-tenant, multi-business scenarios, laying a solid foundation for subsequent priority scheduling.

[0104] Based on the above embodiments, as one implementation of the present invention, determining the static basic score of a request based on a metadata object may include the following steps:

[0105] Step c1: Based on the service contract identifier in the metadata object, query the first preset score table to obtain the corresponding service contract score.

[0106] Specifically, the system queries the first preset score table based on the service contract identifier in the metadata object to obtain the corresponding service contract score. The first preset score table is pre-configured based on the user's payment level and service level agreement (SLA), for example: internal system / administrator (S_0): 100 points; enterprise VIP customer (S_1): 90 points; ordinary paying user (S_2): 70 points; free / trial user (S_3): 50 points.

[0107] In practice, the system uses the service contract identifier as a key value to perform a precise matching query in the first preset score table, ensuring that users of different service levels receive a base score corresponding to their payment level.

[0108] Step c2: Based on the task type identifier in the metadata object, query the second preset score table to obtain the corresponding task type score.

[0109] The system queries the second preset score table based on the task type identifier in the metadata object to obtain the corresponding task type score. The second preset score table is pre-configured based on the computational complexity and real-time requirements of the task. For example: real-time interactive dialogue (T_0): 85 points; asynchronous code generation (T_1): 75 points; batch document summarization (T_2): 60 points; offline data analysis (T_3): 40 points.

[0110] During implementation, the system uses task type identifiers as query conditions to retrieve the corresponding scores from the second preset score table, ensuring that tasks of different business natures receive appropriate priority weights.

[0111] Step c3: Determine the corresponding emergency flag score based on the emergency flag in the metadata object.

[0112] The system determines the corresponding emergency flag score based on the emergency flag in the metadata object. This determination process uses binary logic: when the emergency flag is true, it is assigned a score of 50; when the emergency flag is false, it is assigned a score of 0. In practice, the system can complete this score calculation through simple conditional checks, providing a score boost for requests with urgent processing needs.

[0113] Step c4: The service contract score, task type score, and emergency flag score are weighted and summed according to preset weights, and the result is used as the static base score.

[0114] Specifically, the system weights and sums the service contract score, task type score, and emergency flag score according to preset weights, and the result is used as the final static base score. The weights corresponding to the service contract score, task type score, and emergency flag score can be set according to actual circumstances, and this embodiment of the invention does not impose specific limitations on this. For example, a weight configuration example is as follows: Service contract weight: 0.5; Task type weight: 0.3; Emergency flag weight: 0.2. The specific calculation formula is: Static base score = (Service contract score × 0.5) + (Task type score × 0.3) + (Emergency flag score × 0.2). For example, a real-time interactive dialogue initiated by an enterprise VIP user (S_1: 90 points) (T_0: 85 points) and marked as emergency (50 points) will have a static base score calculated as: (90 × 0.5) + (85 × 0.3) + (50 × 0.2) = 45 + 25.5 + 10 = 80.5 points.

[0115] This embodiment achieves objective and consistent calculation of static base scores through a standardized score table and weight configuration, specifically including the following beneficial effects: First, quantitative evaluation. Abstract concepts such as user service level, task type, and urgency are transformed into concrete numerical values, providing an accurate basis for priority ranking. Second, flexible configuration. Through an adjustable score table and weight coefficients, the system can quickly adjust the priority calculation strategy according to changes in business needs. Third, multi-dimensional integration. It comprehensively considers three key dimensions: user value, task characteristics, and urgency, avoiding priority deviations caused by a single factor. Fourth, efficient calculation: Simple table lookup and weighted calculation ensure excellent performance even in high-speed processing scenarios. Through this embodiment, the system can generate a static base score for each request that accurately reflects its business importance and processing priority, providing reliable basic data support for subsequent dynamic priority adjustments and resource scheduling.

[0116] Based on the above embodiments, as an implementation of the present invention, determining the dynamic priority scores of multiple requests in the target priority queue by using the static base score of the request and the waiting time after the request is enqueued may include the following steps:

[0117] Step d1: Obtain the static base score from the message preprocessing module.

[0118] The resource scheduler obtains a pre-calculated static base score from the message preprocessing module. This static base score, generated during the preprocessing stage before request queuing, contains a weighted sum of the service contract score, task type score, and urgency flag score. The system quickly retrieves the corresponding static base score from the preprocessing result cache using the request identifier, avoiding redundant calculations and improving processing efficiency.

[0119] Step d2: Calculate the waiting time for the request since the enqueue timestamp.

[0120] The system calculates the precise waiting time for a request from its enqueue timestamp. Specifically, the system obtains the current system timestamp and calculates the difference between it and the enqueue timestamp recorded by the message preprocessing module: Waiting time (t_wait) = Current time - Enqueue timestamp. The time precision is down to the millisecond level, ensuring the accuracy and fairness of the calculation. For requests with long waiting times, the system automatically handles time overflow boundary cases.

[0121] Step d3: Input the waiting time into the configurable dynamic aging function to obtain the dynamic aging factor.

[0122] Step d4 involves weighted summation of the static base score and the dynamic aging factor to obtain the dynamic priority score. The dynamic aging function can be any of the following: a linear function, where the dynamic aging factor is proportional to the waiting time; a bounded function, where the dynamic aging factor increases with waiting time but has a preset upper limit; or a non-linear function, where the dynamic aging factor increases non-linearly with waiting time.

[0123] Specifically, the calculated waiting time is input into a configurable dynamic aging function to obtain the dynamic aging factor. The system supports three types of dynamic aging functions, which can be flexibly configured according to business needs:

[0124] The first type of dynamic aging function is a linear function, dynamic aging factor = k × t_wait, where k is the aging coefficient, which can be determined according to the actual situation, for example, k = 0.1. This function ensures that the aging factor increases uniformly over time, and is suitable for scenarios with high fairness requirements.

[0125] The second type of dynamic aging function is a bounded function, where the dynamic aging factor is min(k×t_wait, F_max), and F_max is a preset upper limit, for example, F_max=20. In this case, the values ​​of k and F_max can be determined according to the actual situation, and this embodiment of the invention does not impose specific limitations on them. This function prevents low-priority tasks from infinitely inflating their scores due to long waiting times, and avoids unreasonably surpassing newly entered highest-priority tasks.

[0126] The third type of dynamic aging function is a non-linear function, such as a logarithmic function or a piecewise function, which rapidly increases the score in the early stage of aging and slows down the rate of increase in the later stage to achieve more precise control.

[0127] Finally, the static base score and the dynamic aging factor are weighted and summed to obtain the final dynamic priority score. For example, the calculation formula can be: Dynamic Priority Score = w1 × Static Base Score + w2 × Dynamic Aging Factor. Here, w1 and w2 are preset weighting coefficients, and their values ​​can be configured according to actual conditions. Typically, w1 + w2 = 1 is set. For example, configuring w1 = 0.7 and w2 = 0.3 emphasizes the dominant role of the static base score while also considering the fairness of waiting time.

[0128] For example, suppose a request has a static base score of 80 points and has been waiting for 50 time units. The system is configured to use a linear aging function (k=0.1). The steps to calculate the dynamic priority score are: get the static base score: 80 points; calculate the waiting time: 50 units; calculate the dynamic aging factor: 0.1×50=5 points; weighted summation: 0.7×80+0.3×5=56+1.5=57.5 points.

[0129] This embodiment achieves the following technical effects through dynamic priority score calculation: First, it prevents task starvation. A dynamic aging factor ensures that the priority of requests waiting for a long time gradually increases, preventing low-priority tasks from never being processed. Second, it balances efficiency and fairness. It considers both the inherent static priority of requests and the fairness of waiting time, achieving a good balance between system efficiency and user fairness. Third, it is flexible and configurable. It supports multiple aging functions and weight configurations, allowing scheduling strategies to be adjusted according to different business scenarios. Fourth, it is computationally efficient. Simple arithmetic operations ensure excellent performance even in high-speed scheduling scenarios. Through the implementation of this embodiment, the system can intelligently adjust the processing priority of requests in high-concurrency scenarios, ensuring both timely processing of high-priority tasks and the fairness and efficiency of system resource allocation.

[0130] Next, the four configurable scheduling strategies used in this embodiment of the invention for selecting a target priority queue from multiple independent priority queues will be described in detail.

[0131] As one implementation of this invention, selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy may include the following steps: sequentially polling the status of each queue in descending order of their static priorities; and selecting the first queue that is not empty as the target priority queue.

[0132] In this implementation, the resource scheduler selects a target priority queue from multiple independent priority queues based on a configurable scheduling strategy. Specifically, a strict priority scheduling strategy is employed, implemented through the following steps: The resource scheduler sequentially polls and checks the status of each priority queue according to a preset order of static priority from high to low. The system maintains a list of priority queues, for example: [highest priority queue, high priority queue, medium priority queue, low priority queue], and strictly traverses and checks according to this order. During each scheduling decision, the scheduler starts from the head of the list and sequentially queries the current status of each queue, including key indicators such as whether the queue is empty and the number of requests awaiting processing in the queue.

[0133] During the polling process, the scheduler selects the first non-empty queue as the target priority queue. Specifically, the scheduler starts checking from the highest priority queue: if the highest priority queue is not empty (containing pending requests), it is immediately designated as the target priority queue, and the checking of subsequent queues is terminated; if the highest priority queue is empty, the next priority queue (such as a higher priority queue) is checked; and so on, until the first non-empty queue is found; if all queues are empty, the scheduler enters a waiting state until a new request arrives.

[0134] The strict priority scheduling strategy adopted in this embodiment has the following technical advantages: First, it guarantees high-priority requests. Requests in high-priority queues are guaranteed to have absolute priority over low-priority queues, meeting the low-latency requirements of VIP users and urgent tasks. Second, it is simple and efficient: the algorithm logic is simple and clear, with high execution efficiency, suitable for high-concurrency scenarios. Third, it is highly predictable: the scheduling behavior is completely predictable, facilitating system performance analysis and capacity planning. This implementation method is particularly suitable for scenarios with strict Service Level Agreement (SLA) requirements, ensuring that high-priority users receive stable, high-quality service. Through the implementation of this embodiment, the system can efficiently and reliably select the queue that needs the highest priority processing in a multi-priority queue environment, laying the foundation for subsequent request scheduling and processing.

[0135] As another implementation of this invention, selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy may include the following steps:

[0136] Each queue is pre-set with a corresponding static weight, and the initial value of the dynamic credit counter for each queue is set to the static weight value of that queue.

[0137] Poll all non-empty queues, compare the current dynamic credit counter values ​​of each queue, and select the queue with the largest dynamic credit counter value as the candidate queue.

[0138] If multiple candidate queues exist, the queue with the highest static priority is selected as the final target priority queue. If only one candidate queue exists, that queue is directly designated as the target priority queue. Specifically, after retrieving a request from the target priority queue, the dynamic credit counter value of the target priority queue is subtracted from the sum of the static weights of all queues. Furthermore, the credit counters of all queues in the system are incremented by their respective static weight values. Finally, the process returns to the step of polling and checking all non-empty queues to form a continuous scheduling loop.

[0139] In this implementation, the resource scheduler employs an enhanced weighted round-robin strategy to select a target priority queue from multiple independent priority queues. Intelligent queue selection is achieved through the following steps: The system presets a corresponding static weight for each priority queue, reflecting the queue's relative importance in resource allocation. For example, a high-priority queue can be configured with a weight of 5, a medium-priority queue with a weight of 3, and a low-priority queue with a weight of 1. Simultaneously, the system sets the initial value of each queue's dynamic credit counter to the queue's static weight value, establishing an initial credit balance state.

[0140] The resource scheduler polls and checks non-empty queues, comparing the current dynamic credit counter values ​​of each queue. The system identifies the queue with the largest dynamic credit counter value as the primary candidate queue. When multiple queues have the same credit counter value, and all are at the maximum value, the system initiates a tie-breaking mechanism, selecting the queue with the highest static priority from the tied queues as the final target priority queue; if only one queue has the largest credit counter value, that queue is directly designated as the target priority queue.

[0141] After retrieving a request from the target priority queue, the system performs a credit update operation: subtracting the sum of the static weights of all queues from the dynamic credit counter value of the target priority queue. This operation significantly lowers the priority of that queue, providing scheduling opportunities for other queues. Simultaneously, the system increments the credit counter of each queue by its corresponding static weight value, ensuring that the credit value of each queue grows steadily over time. After completing the credit update, the system returns to the step of polling and checking all non-empty queues, forming a continuous, adaptive scheduling loop.

[0142] The enhanced weighted round-robin strategy adopted in this embodiment has the following technical advantages: First, it balances fairness and efficiency. The credit mechanism ensures that each queue receives processing opportunities according to its weight ratio, avoiding the "starvation" phenomenon that may occur in low-priority queues in strict priority strategies, and better reflecting the differences in importance between different queues than a simple round-robin strategy. Second, it offers dynamic adaptability. The dynamic changes in the credit counter enable the system to automatically adjust the scheduling frequency according to the actual load of each queue, achieving load-sensitive resource allocation.

[0143] Third, priority guarantee. By selecting the queue with the highest static priority in the event of a tie, the inherent service level remains effective based on the weighted allocation. Fourth, stability and convergence: The credit update mechanism ensures that the actual processing ratio of each queue strictly follows its static weight in long-term operation, exhibiting excellent mathematical convergence. This implementation is particularly suitable for multi-tenant service scenarios that require balancing fairness and priority guarantee, ensuring timely processing of high-priority tasks while providing basic service quality guarantees for all users. Through the implementation of this embodiment, the system can achieve intelligent, adaptive, fair, and efficient queue selection in complex multi-priority queue environments, providing reliable scheduling guarantees for large-scale, high-concurrency dialogue services.

[0144] Based on the above embodiments, as one implementation of the present invention, selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy includes:

[0145] Associate each queue with a token bucket and configure the corresponding token generation rate and token bucket capacity.

[0146] Tokens are generated periodically for each queue, and the current number of tokens in each queue does not exceed the capacity of its token bucket.

[0147] When scheduling is required, select eligible queues from all non-empty queues whose current token count is sufficient to cover the estimated cost of the task at the head of the queue.

[0148] Select the queue with the highest static priority from the eligible queues as the target priority queue. Return to the execution step of filtering out eligible queues in all non-empty queues whose current token count is sufficient to cover the estimated cost of the task at the head of the queue when scheduling is required, to form a continuous scheduling loop. After retrieving the pending request from the target priority queue, deduct the estimated token cost of the pending request from the current token count of the target priority queue.

[0149] In this implementation, the resource scheduler employs a token bucket fair queuing strategy to select a target priority queue from multiple independent priority queues. The system associates each priority queue with an independent token bucket and configures two key parameters for each token bucket: the token generation rate (r_tokens / second) and the token bucket capacity (C_max). The token generation rate determines the long-term average processing bandwidth the queue can obtain, while the token bucket capacity controls the maximum number of tokens the queue can accumulate during idle periods to handle sudden traffic surges. For example, a higher token generation rate (e.g., 10 tokens / second) and a larger bucket capacity (e.g., 100 tokens) can be configured for high-priority queues to ensure they receive sufficient resource supply.

[0150] The system periodically generates tokens for each queue using a high-frequency timer. Within each update cycle Δt, the token count of queue Qi is updated according to the following rule: current token count = min(current token count + r_i × Δt, C_max). This mechanism ensures that the token count of each queue grows steadily at a preset rate, without exceeding its bucket capacity limit, thus achieving long-term precise control over queue resource usage.

[0151] When the resource scheduler needs to select a task, it first filters out all eligible queues that simultaneously meet two conditions: the queue is not empty and the queue currently has enough tokens to cover the estimated cost of the task at its head. The token cost of a task is determined by either querying a pre-defined cost table based on task metadata (such as task type, input text length, etc.) or by calling a lightweight cost prediction model for real-time estimation. This filtering step ensures that only queues with sufficient "resource credit" are eligible to participate in this scheduling competition.

[0152] From all eligible queues, the system selects the queue with the highest static priority as the target priority queue. This selection mechanism ensures that, given sufficient resources, high-priority queues still enjoy absolute priority. After retrieving a request from the target priority queue, the system immediately deducts the estimated token cost of the request from the queue's current token count, completing real-time resource settlement. After completing this scheduling step, the system immediately returns to the eligible queue selection step, forming a continuous scheduling loop. This design enables the system to respond quickly to changes in resource status, providing timely services to eligible queues, while effectively preventing any single queue from excessively consuming system resources through the token deduction mechanism.

[0153] The token bucket fair queuing strategy adopted in this embodiment has the following significant advantages: First, quantifiable quality of service assurance: By precisely controlling the token generation rate, the system can provide a guaranteed minimum service bandwidth for each queue, meeting the requirements of different Service Level Agreements (SLAs). Second, flexible burst processing capability: The token bucket capacity parameter allows queues to accumulate tokens during idle periods, thereby burst processing backlogged tasks when needed, improving the system's adaptability to traffic fluctuations. Third, efficient resource utilization: Through a token deduction mechanism based on actual costs, the system achieves fine-grained measurement and allocation of computing resources, avoiding inefficient resource use. Fourth, a combination of fairness and priority: While ensuring that each queue receives basic resource guarantees, the importance differences between different queues are still maintained through static priorities. This implementation is particularly suitable for multi-tenant scenarios requiring strict resource isolation and predictable quality of service, providing fair basic service guarantees for all users while ensuring the performance of high-priority tasks.

[0154] Based on the above embodiments, as one implementation of the present invention, selecting a target priority queue from multiple independent priority queues according to a configurable scheduling strategy includes:

[0155] A scheduling value score is calculated for each non-empty queue. This score is derived by comprehensively evaluating multiple dimensions, including the queue's static priority, its real-time stress state, the system's real-time load state, and the estimated task complexity. The queue's real-time stress state considers both queue length and average task waiting time. The system's real-time load state is based on the overall resource utilization of the large model cluster. The estimated task complexity is calculated using a predictive model to estimate the resource requirements of multiple tasks at the head of the queue.

[0156] The indicators for each dimension are normalized to eliminate differences in units. Based on the normalized indicators, the scheduling value score for each queue is calculated according to the preset scheduling value score; the queue with the highest scheduling value score is selected as the target priority queue.

[0157] In this implementation, the resource scheduler uses a dynamic adaptive scheduling strategy to select a target priority queue from multiple independent priority queues. The system calculates a comprehensive scheduling value score for each non-empty queue, which is obtained by integrating real-time indicators from four key dimensions:

[0158] First, the queue's static priority dimension ( The system considers the inherent static priority identifier of the queue, which is a fundamental element of scheduling decisions. High-priority queues receive higher base scores in this dimension, ensuring that the system always respects the preset service level classification.

[0159] Second, the real-time pressure status dimension of the queue ( The system comprehensively considers queue length and average task waiting time to assess queue pressure. The specific calculation is: Queue Pressure = w3 × Queue Length + w4 × Average Waiting Time. Here, w3 and w4 are preset weighting coefficients. This mechanism ensures that queues with severe backlogs or excessively long waiting times receive appropriate priority increases.

[0160] Third, the system's real-time load status dimension ( The overall resource utilization of the large model cluster is evaluated. The system monitors key indicators such as CPU, GPU utilization, and memory usage of all LLM instances in real time to calculate the overall system load level: System Load = Current Used Resources / Total Available Resources. When the system load is high, the scheduler tends to select tasks with lower computational complexity to improve throughput; when the load is low, it focuses more on service quality.

[0161] Fourth, the dimension of task estimation complexity ( The system estimates the resource requirements of multiple tasks in the queue using a predictive model. It employs lightweight machine learning models (such as regression models based on features like task type and input text length) to predict the relative computational cost of tasks and outputs a normalized complexity score.

[0162] The throughput tendency of computing systems ( Judgment method: This is a heuristic function used to determine whether to handle complex or simple tasks for the fastest turnaround under the current system load.

[0163] The calculation formula is:

[0164] The system normalizes the original indicators for each dimension to eliminate dimensional differences. A min-max normalization system is used to scale each indicator to the [0,1] interval: Normalized value = (Original value - Minimum value) / (Maximum value - Minimum value). This process ensures that indicators with different dimensions are compared and weighted on the same scale.

[0165] Finally, queue Scheduling value It is derived from the following formula: The weights in the formula can be configured according to the actual application scenario.

[0166] The dynamic adaptive scheduling strategy adopted in this embodiment has the following technical advantages: First, multi-dimensional intelligent decision-making. It comprehensively considers queue attributes, system status, and task characteristics to achieve more comprehensive and intelligent scheduling decisions. Second, real-time environmental adaptability. By continuously monitoring system load and queue pressure, the scheduling strategy is dynamically adjusted to adapt to constantly changing workloads. Third, fine-grained resource control: Based on task complexity prediction, it achieves fine-grained allocation of computing resources, improving resource utilization efficiency. Fourth, business goal alignment: Configurable weight coefficients enable the scheduling strategy to flexibly adapt to different business goals and performance requirements. This implementation method is particularly suitable for complex scenarios with large workload fluctuations and diverse task characteristics, maximizing overall system throughput and resource utilization while ensuring high-priority service quality.

[0167] Based on the above embodiments, as one implementation of the present invention, selecting a target large model instance based on load balancing and task matching principles may include the following steps:

[0168] The system monitors the real-time running status of all instances in the large model cluster, obtaining current load metrics, available hardware resource information, and loaded model capabilities for each instance. Based on the specific requirements of the task to be processed, it evaluates the matching degree between each instance and the task, generating a matching degree score. Based on the obtained real-time running status, it calculates the load score for each instance; instances with lower current load metrics and more abundant available hardware resources receive higher load scores. The matching degree score and load score of each instance are combined and weighted according to preset weights to generate a comprehensive evaluation score. The instance with the highest comprehensive evaluation score is selected as the target large model instance.

[0169] In this implementation, the resource scheduler selects target large model instances based on load balancing and task matching principles. Optimal instance selection is achieved through the following intelligent evaluation mechanism: the system collects real-time runtime status data of all instances in the large model cluster via a distributed monitoring agent. Monitoring data includes: current load metrics: GPU utilization, memory utilization, number of currently processed tasks, and request processing rate; available hardware resources: remaining GPU memory capacity, number of available CPU cores, and network bandwidth utilization; and model capability information: loaded model types, model versions, supported inference capabilities, and domain tags. This monitoring data is updated at a second-level frequency, providing an accurate real-time status profile for instance selection.

[0170] Furthermore, the system evaluates the matching degree between each instance and the task from multiple dimensions based on the specific requirements of the task to be processed: The first dimension is model capability matching. Specifically, it checks whether the model loaded by the instance supports the inference capabilities required by the task, such as code generation, text summarization, multi-turn dialogue, and other professional functions. The second dimension is resource requirement matching. Specifically, based on the estimated complexity of the task, it assesses whether the hardware resources of the instance meet the task execution requirements, especially the video memory capacity and computing power. The third dimension is performance requirement matching: for latency-sensitive tasks, instances with low network latency and fast processing speed are prioritized; for computationally intensive tasks, instances with strong computing power are selected. The matching degree score uses a standardized score of 0-1, calculated through a preset matching rule table and similarity algorithm.

[0171] Based on real-time monitoring data, the system calculates the load score for each instance: Load Score = w1 × (1 - GPU utilization) + w2 × (Available VRAM ratio) + w3 × (1 - Current number of tasks / Maximum concurrency). Here, w1, w2, and w3 are preset weighting coefficients to ensure that instances with lower current load metrics and more available hardware resources receive higher load scores. The system weights and merges the matching score and load score according to preset weights, resulting in a comprehensive evaluation score = α × Matching Score + β × Load Score; where α and β are configurable weighting coefficients, satisfying α + β = 1. Weight configuration supports dynamic adjustment; for example, during peak business periods, the load score weight (β value) can be increased to ensure system stability; when prioritizing service quality, the matching score weight (α value) can be increased to guarantee task execution effectiveness.

[0172] The system selects the instance with the highest overall evaluation score as the target large model instance. When multiple instances have similar scores, the system supports several selection strategies: Performance priority: Select the instance with higher hardware configuration; Energy efficiency priority: Select the instance with better energy efficiency ratio; Cost priority: Select the instance with lower operating costs.

[0173] The instance selection system used in this embodiment has the following technical advantages: First, multi-objective optimization. It simultaneously considers task matching degree and system load, maintaining system stability while ensuring task execution effect.

[0174] Second, real-time adaptability. Based on monitoring data updated in seconds, it ensures that selection decisions always reflect the latest system status. Third, refined resource utilization. Through precise matching evaluation, it ensures that tasks are assigned to the most suitable instance for execution, improving resource utilization efficiency. Fourth, flexible configurability. It supports dynamic adjustment of weight coefficients to adapt to the needs of different business scenarios and performance goals. Fifth, fault tolerance and degradation. When the optimal instance is unavailable, the system automatically selects the second-best instance to ensure service continuity. This implementation is particularly suitable for heterogeneous large-scale cluster environments, maximizing the overall throughput and resource utilization of the cluster while ensuring service quality, providing reliable instance selection guarantees for multi-tenant, multi-business scenarios.

[0175] As another implementation of this invention, the resource scheduler is further configured to trigger the dialogue state manager to update the dialogue context after receiving the processing result returned by the target large model instance.

[0176] After the resource scheduler receives the processing result returned by the target large model instance, it first performs integrity verification and format checks on the result. The system confirms that the result data contains necessary fields, such as the dialogue identifier, the generated response content, and the processing status indicator, and verifies the integrity and validity of the data to avoid context update errors caused by data anomalies. After successful verification, the resource scheduler immediately sends a context update request to the dialogue state manager. This request uses a standardized data format and contains the following key information:

[0177] Dialogue identifier: uniquely identifies the current dialogue session; User original request: includes request content and metadata;

[0178] Model-generated response: The complete response content returned by the large model instance; Timestamp information: A precise record of the request and response times; Session status flags: Indicate whether the session has ended or requires special handling.

[0179] Upon receiving an update request, the dialogue state manager performs atomic context update operations: it appends the current user request and the corresponding model response as a complete dialogue round to the history of the dialogue; it updates metadata such as the last active time of the session, the dialogue round count, and the cumulative interaction duration; it verifies whether the updated context data conforms to predefined business rules and integrity constraints; and it persists the updated dialogue context to the storage system to ensure that the data is not lost.

[0180] After the context update is complete, the dialog state manager returns an update confirmation message to the resource scheduler. If an exception occurs during the update process, the system executes the following procedures: First, retry mechanism: For temporary errors, the system automatically performs a limited number of retries; Second, degradation handling: When persistent storage is unavailable, the system temporarily stores the context data in the cache, and synchronizes it after storage is restored; Third, error reporting: For unrecoverable errors, the system generates a detailed error report and notifies the operations and maintenance personnel.

[0181] This implementation is suitable for real-time interactive scenarios with high requirements for dialogue coherence. It ensures that the context of multi-turn dialogues remains up-to-date and accurate, providing users with a coherent and intelligent dialogue experience. By tightly coupling context updates with the reception of processing results, the system achieves efficient management and reliable maintenance of the dialogue state.

Claims

1. A dialogue message queue management system based on a large model cluster, characterized in that, include: The message preprocessing module is used to receive dialogue message requests and construct structured metadata objects by integrating multi-dimensional information from the client, authentication system and the request content itself. Based on metadata objects, priority queue labels are dynamically assigned to requests by querying a configurable mapping rule table; The system determines the static base score of the request based on the metadata object and records the request's queuing timestamp; wherein, the structured metadata object includes: service contract identifier, task type identifier, urgency flag, task complexity score, user identifier, and rate limiting policy; the service contract identifier is used to identify the user's service level and business permissions; A message router is used to distribute requests to the corresponding priority queues based on priority queue labels. The dialogue state manager is used to store and retrieve historical context information of a dialogue based on the dialogue identifier; The resource scheduler performs the following steps: The target priority queue is selected from multiple independent priority queues according to a configurable scheduling strategy; the decision basis of the scheduling strategy is any one of the following: (1) the static priority of the queue; (2) the static priority, static weight and dynamic credit value of the queue; (3) the static priority of the queue and the current number of tokens in the token bucket; (4) the static priority of the queue, the real-time pressure status of the queue, the system load status and the estimated complexity of the task. The dynamic priority score of the requests in the target priority queue is determined by the static base score of the request and the waiting time after enqueueing, and the request with the highest dynamic priority score is identified as the request to be processed. The historical context information of the pending requests is retrieved and associated from the dialog state manager to form pending tasks; a target large model instance is selected based on the principles of load balancing and task matching, and the pending tasks are assigned to the target large model instance.

2. The system according to claim 1, characterized in that, The process of constructing a structured metadata object by integrating multi-dimensional information from the client, authentication system, and the request content itself includes: By querying the user database using authentication credentials, static business attributes related to the service contract are obtained, including user identifier, service level, and rate limiting policy. Parse the explicit business fields in the request body and normalize the parsed fields, including mapping different task types to a standard identifier and converting priority fields of different formats into a uniform value. The request content is analyzed by an inference engine to generate implicit data attributes, which include at least one of the following: task type inference based on text length, emergency sign recognition based on keyword matching, and task complexity score based on a prediction model. Based on the obtained static business attributes, the normalized explicit business fields, and the implicit data attributes, a structured metadata object is generated.

3. The system according to claim 2, characterized in that, The message preprocessing module includes a sub-mapping module for request-to-queue mapping, the sub-mapping module including a configurable mapping rule table; the sub-mapping module is configured to perform the following steps: The configurable mapping rule table is queried through the mapping function. Based on the service contract identifier, task type identifier, and emergency flag in the metadata object, the corresponding priority queue index is determined and used as the priority queue label. When the combination of the current service contract identifier, task type identifier, and emergency flag is not found in the mapping rule table, a rollback mapping mechanism is triggered, and any of the following operations are performed: (1) Assign the request to the preset default priority queue; (2) Perform downgraded matching operation: According to the preset matching hierarchy, prioritize matching service contract identifiers. If no match is found, continue matching task type identifiers. If no match is found, continue matching emergency identifiers. (3) Reject the request and return an error message to the requesting source.

4. The system according to claim 2, characterized in that, Determining the static base score of the request based on the metadata object includes: Based on the service contract identifier in the metadata object, query the first preset score table to obtain the corresponding service contract score; Based on the task type identifier in the metadata object, query the second preset score table to obtain the corresponding task type score; Based on the emergency flag in the metadata object, determine the corresponding emergency flag score; The service contract score, task type score, and emergency flag score are weighted and summed according to preset weights, and the result is used as the static base score.

5. The system according to claim 4, characterized in that, The process of determining the dynamic priority score of a request in the target priority queue based on the static base score and the waiting time after enqueueing includes: Obtain the static basic score from the message preprocessing module; Calculate the waiting time for the request since the enqueue timestamp; The waiting time is input into a configurable dynamic aging function to obtain the dynamic aging factor; The static base score and the dynamic aging factor are weighted and summed to obtain the dynamic priority score; The dynamic aging function is any one of the following functions: A linear function, wherein the dynamic aging factor is proportional to the waiting time; A bounded function, wherein the dynamic aging factor increases with the waiting time but has a preset upper limit; A nonlinear function, wherein the dynamic aging factor increases nonlinearly with the waiting time.

6. The system according to claim 1, characterized in that, The step of selecting a target priority queue from multiple independent priority queues according to a configurable scheduling policy includes: The status of each queue is polled sequentially in descending order of static priority. The first non-empty queue is selected as the target priority queue.

7. The system according to claim 1, characterized in that, The step of selecting a target priority queue from multiple independent priority queues according to a configurable scheduling policy includes: Each queue is pre-set with a corresponding static weight, and the initial value of the dynamic credit counter for each queue is set to the static weight value of that queue. Poll all non-empty queues, compare the current dynamic credit counter values ​​of each queue, and select the queue with the largest dynamic credit counter value as the candidate queue. If there are multiple candidate queues, select the queue with the highest static priority from among them as the final target priority queue; if there is only one candidate queue, directly determine that candidate queue as the target priority queue. Specifically, after retrieving the request to be processed from the target priority queue, the dynamic credit counter value of the target priority queue is subtracted from the sum of the static weights of all queues; and the dynamic credit counter of all queues in the system is increased by their respective static weight values. It then returns to perform the step of polling and checking all non-empty queues to form a continuous scheduling loop.

8. The system according to claim 1, characterized in that, The step of selecting a target priority queue from multiple independent priority queues according to a configurable scheduling policy includes: Associate each queue with a token bucket and configure the corresponding token generation rate and token bucket capacity; Tokens are generated periodically for each queue, and the current number of tokens in each queue does not exceed the capacity of its token bucket. When scheduling is required, select all non-empty queues that have enough current tokens to cover the estimated cost of their head task. Select the queue with the highest static priority from the qualified queues as the target priority queue; return to the step of filtering out qualified queues in all non-empty queues when scheduling is required, where the current number of tokens is sufficient to cover the estimated cost of the task at the head of the queue, to form a continuous scheduling loop; Specifically, after retrieving the pending request from the target priority queue, the estimated token cost of the pending request is deducted from the current token count of the target priority queue.

9. The system according to claim 1, characterized in that, The step of selecting a target priority queue from multiple independent priority queues according to a configurable scheduling policy includes: A scheduling value score is calculated for each non-empty queue. The scheduling value score is calculated by comprehensively evaluating the following multiple dimensions: the queue's static priority, the queue's real-time pressure status, the system's real-time load status, and the estimated complexity of the tasks. Among them, the queue's real-time pressure status comprehensively considers the queue length and the average waiting time of the tasks; the system's real-time load status is obtained based on the overall resource utilization of the large model cluster; and the estimated complexity of the tasks is estimated by a prediction model based on the resource requirements of multiple tasks at the head of the queue. The indicators of each dimension are normalized to eliminate differences in units; Based on the normalized indicators of each dimension, the scheduling value score is calculated to obtain the final scheduling value score for each queue. Select the queue with the highest scheduling value score as the target priority queue.

10. The system according to any one of claims 1 to 9, characterized in that, The selection of target large model instances based on load balancing and task matching principles includes: Monitor the real-time running status of all instances in the large model cluster, and obtain the current load indicators, available hardware resource information, and loaded model capability information of each instance; Based on the specific requirements of the task to be processed, assess the matching degree between each instance and the task, and generate a matching degree score; Based on the acquired real-time running status, the load score of each instance is calculated. The lower the current load index and the more available hardware resources an instance has, the higher its load score. The matching score and load score of each instance are combined and weighted according to preset weights to generate a comprehensive evaluation score; The instance with the highest overall evaluation score is selected as the target large model instance.

Images