Traffic management method, electronic device, storage medium and computer program product

Abstract

The invention discloses a traffic management method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of traffic management and control, and the method comprises the steps: receiving a large model service request, and judging whether the large model service request is allowed to be processed or not based on the large model service request and a preset traffic record library; selecting a target large model from the preset large models based on the traffic weights of the preset large models under the condition that the large model service request is allowed to be processed; and under the condition that the target large model is idle, distributing the large model service request to the target large model for processing, recording the actual consumption flow of the large model service request, and updating the preset flow record library based on the actual consumption flow. According to the method and the device, the technical problem of poor management effect in the current gateway flow management based on the request times is solved.

Description

Technical Field

[0001] This application relates to the field of traffic management technology, and in particular to a traffic management method, electronic device, storage medium and computer program product. Background Technology

[0002] With the widespread application of large-scale modeling technology across various industries, its service traffic is increasingly flooding existing network infrastructures. In current technologies, network gateways, as key components of traffic management, typically employ rate limiting mechanisms based on QPS (Queries Per Second) and load balancing strategies based on request counts to ensure the stability of backend services. However, large-scale modeling service requests are characterized by vastly different resource consumption per request and long processing times. A predictable correlation no longer exists between the actual computational resource consumption and the simple number of requests, making it impossible to accurately quantify the actual computational resource consumption of a single request and difficult to identify and manage high-resource-consuming requests. Therefore, current gateway traffic management based on request counts suffers from poor management effectiveness.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a traffic management method, electronic device, storage medium, and computer program product, which aims to solve the technical problem of poor management effect in current gateway traffic management based on the number of requests.

[0005] To achieve the above objectives, this application proposes a traffic management method, which includes: Receive large model service requests and determine whether to allow processing the large model service requests based on the large model service requests and a preset traffic record database; If the large model service request is allowed to be processed, the target large model is selected from the preset large models based on the traffic weight of each preset large model; When the target large model is idle, the large model service request is allocated to the target large model for processing, the actual traffic consumed by the large model service request is recorded, and the preset traffic record database is updated based on the actual traffic consumed.

[0006] In one embodiment, the step of determining whether to allow processing the large model service request based on the large model service request and a preset traffic record database includes: The corresponding user identifier is determined based on the large model service request. Query the user's cumulative data consumption within a preset time period in the preset data consumption record database, and compare the cumulative data consumption with the preset data limit. If the cumulative traffic consumption is less than the preset traffic limit, it is determined that the large model service request can be processed. If the accumulated traffic consumption exceeds the preset traffic limit, it is determined that the large model service request cannot be processed.

[0007] In one embodiment, the step of selecting the target large model from the preset large models based on the traffic weights of each preset large model includes: Calculate the traffic weights of each preset large model in the current period and generate random numbers, wherein the random numbers are integers from zero to the range corresponding to the sum of the traffic weights; The preset large models are combined into a large model sequence and the large model sequence is traversed. For any candidate large model in the large model sequence, it is determined whether the random number is less than the traffic weight corresponding to the candidate large model. If the random number is less than the traffic weight corresponding to the candidate large model, the candidate large model is taken as the target large model for the current period, and the traversal ends. If the random number is not less than the traffic weight corresponding to the candidate large model, then the random number is subtracted from the traffic weight corresponding to the candidate large model to obtain a new random number. The next large model of the candidate large model is taken as the new candidate large model, and the step of determining whether the random number is less than the traffic weight corresponding to the candidate large model is returned to be executed until the target large model is found or the sequence of large models is traversed.

[0008] In one embodiment, the step of calculating the traffic weights corresponding to each preset large model in the current period includes: According to the preset collection cycle, obtain the cumulative delay time and cumulative resource consumption of each preset large model in the historical processing process; For any of the preset large models, the incremental average latency of the preset large model in the current period is calculated based on the total latency of the preset large model up to the previous period, the total resource consumption of the preset large model up to the previous period, the cumulative latency of the preset large model, and the cumulative resource consumption of the preset large model. The traffic weight of the preset large model in the current period is determined based on the incremental average delay, wherein the incremental average delay is inversely proportional to the traffic weight.

[0009] In one embodiment, the step of determining the traffic weight of the preset large model in the current period based on the incremental average delay includes: Obtain the incremental average delay corresponding to each preset large model, calculate the reciprocal of each incremental average delay, and obtain each basic weight value; By summing up the aforementioned basic weight values, the total global basic weight is obtained; The ratio of the basic weight value corresponding to the preset large model to the sum of the global basic weights is used as the traffic weight corresponding to the preset large model.

[0010] In one embodiment, the step of allocating the large model service request to the target large model for processing when the target large model is idle further includes: Determine whether there are any idle connections in the connection pool of the target large model; If there are idle connections in the connection pool, the target large model is determined to be idle, wherein the idle connection is used to transmit the large model service request; If there are no idle connections in the connection pool, determine whether the number of connections in the connection pool has reached the preset connection limit. If the number of connections has not reached the preset connection limit, the target large model is determined to be idle, and the idle connection is created in the connection pool. If the number of connections reaches the preset connection limit, it is determined that the target large model is not idle, and the large model service request is rejected.

[0011] In one embodiment, the step of recording the actual traffic consumed by the large model service request includes: Obtain the response data stream returned by the target large model in processing the large model service request; Identify the format identification information in the response data stream, and identify the response type of the response data stream based on the format identification information; If the response type is a non-streaming response, information representing traffic consumption is synchronously extracted from the preset structured fields of the response data stream as the actual traffic consumed by the large model service request. When the response type is a streaming response, when processing continuous data segments of the response data stream, a target data segment carrying total resource consumption identification information is searched from the continuous data segments, and information representing traffic consumption is extracted from the target data segment as the actual traffic consumption of the large model service request.

[0012] Furthermore, to achieve the above objectives, this application also proposes a traffic management system, which includes: The request and response module is used to receive large model service requests and determine whether to allow processing the large model service request based on the large model service request and a preset traffic record database. The model filtering module is used to select a target large model from the preset large models based on the traffic weight of each preset large model, provided that the large model service request can be processed. The processing feedback module is used to allocate the large model service request to the target large model for processing when the target large model is idle, record the actual traffic consumed by the large model service request, and update the preset traffic record database based on the actual traffic consumed.

[0013] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the traffic management method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the traffic management method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the traffic management method described above.

[0016] This application provides a traffic management method, which includes: receiving a large model service request, and determining whether to allow processing the large model service request based on the large model service request and a preset traffic record database; if processing the large model service request is allowed, selecting a target large model from the preset large models based on the traffic weight of each preset large model; if the target large model is idle, allocating the large model service request to the target large model for processing, recording the actual traffic consumed by the large model service request, and updating the preset traffic record database based on the actual traffic consumed.

[0017] This application receives large-model service requests and determines whether to allow processing based on a pre-set traffic record database. By introducing the traffic record database, it reflects historical cumulative resource consumption, transforming the basis for admission decisions from a simple "request frequency" to a quantifiable "historical resource consumption." This allows for the accurate identification and blocking of users or requests with excessively high historical cumulative resource consumption that may threaten system stability, achieving effective control over high-resource-consuming requests. When processing is allowed, a target large-model is selected based on the traffic weights of each pre-set large model. The weights dynamically reflect the real-time pressure on the large model rather than static request counts, solving the problem of uneven load distribution and intelligently scheduling traffic to the large model with the least pressure, improving gateway response speed and stability. Requests are assigned to the target large model for processing, and the actual traffic consumption is recorded and the record database is updated. By continuously monitoring and accumulating real resource consumption data, the challenge of quantifying resource consumption is solved, achieving refined end-to-end monitoring and dynamic policy adjustment. Compared to related solutions that cannot accurately quantify the actual computational resource consumption of a single request and struggle to identify and manage high-resource-consuming requests, this solution shifts from request-based management to refined management based on actual resource consumption. This effectively solves the problem of poor gateway traffic management in large-scale model scenarios and improves system resource utilization and service reliability. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the traffic management method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the traffic management method of this application; Figure 3 This is an example diagram of resource consumption data acquisition provided in Embodiment 2 of this application; Figure 4 This is a flowchart illustrating the overall architecture of the traffic management method provided in Embodiment 2 of this application. Figure 5 This is a flowchart of the rate limiting module provided in Embodiment 2 of this application; Figure 6 This is a flowchart of the routing module provided in Embodiment 2 of this application; Figure 7This is a schematic diagram of the module structure of the traffic management system according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the traffic management method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the first embodiment described herein is merely used to explain the technical solution of this application and is not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of the first embodiment of this application is: receiving a large model service request, and determining whether to allow processing the large model service request based on the large model service request and a preset traffic record database; if processing the large model service request is allowed, selecting a target large model from the preset large models based on the traffic weight of each preset large model; if the target large model is idle, allocating the large model service request to the target large model for processing, recording the actual traffic consumed by the large model service request, and updating the preset traffic record database based on the actual traffic consumed.

[0025] In the first embodiment, for ease of description, the following description uses the traffic management system as the execution subject.

[0026] As a key component of traffic management, existing network gateways typically employ rate limiting mechanisms based on QPS (Queries Per Second) and load balancing strategies based on the number of requests to ensure the stability of backend services. However, large-scale service requests are characterized by significant differences in resource consumption per request and long processing times. There is no predictable correlation between the actual computational resource consumption and the simple number of requests, making it impossible to accurately quantify the actual computational resource consumption of a single request and difficult to identify and manage high-resource-consuming requests.

[0027] This application provides a solution that receives large-model service requests and determines whether to allow processing based on a preset traffic record database. By introducing the traffic record database, it reflects historical cumulative resource consumption, transforming the basis for admission decisions from a simple "request frequency" to a quantifiable "historical resource consumption." This allows for the accurate identification and blocking of users or requests with excessively high historical cumulative resource consumption that may threaten system stability, achieving effective control over high-resource-consuming requests. When processing is allowed, a target large model is selected based on the traffic weights of each preset large model. The weights dynamically reflect the real-time pressure of the large model rather than static request counts, solving the problem of uneven load distribution and intelligently scheduling traffic to the large model with the least pressure, improving gateway response speed and stability. Requests are assigned to the target large model for processing, and the actual traffic consumption is recorded and the record database is updated. By continuously monitoring and accumulating real resource consumption data, the problem of quantifying resource consumption is solved, achieving refined end-to-end monitoring and dynamic policy adjustment. Compared to related solutions that cannot accurately quantify the actual computational resource consumption of a single request and struggle to identify and manage high-resource-consuming requests, this solution shifts from request-based management to refined management based on actual resource consumption. This effectively solves the problem of poor gateway traffic management in large-scale model scenarios and improves system resource utilization and service reliability.

[0028] It should be noted that the executing entity in the first embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, or other electronic device, or a system, application, or program capable of implementing the above functions. The first embodiment and the following embodiments will be described using a traffic management system as an example.

[0029] All actions involving the acquisition of signals, information, or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the relevant device.

[0030] Based on this, embodiments of this application provide a traffic management method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the traffic management method of this application.

[0031] In this embodiment, the traffic management method includes steps S01 to S03: Step S01: Receive a large model service request and determine whether to allow processing the large model service request based on the large model service request and the preset traffic record database. It should be noted that a large model service request refers to a network request initiated by the client to access and utilize the large-scale computing power of the backend. The default traffic log is a database used for persistent storage and querying of historical data associated with traffic consumption; it can be a Redis instance, which records the amount of tokens generated when users utilize large models.

[0032] Additionally, it's important to note that in large-scale model applications, tokens are used as the unit of measurement for resource consumption. A token is the smallest unit of text processing in a large model; it can be a character, word, or punctuation mark. Therefore, tokens can represent the bandwidth consumed during large-scale model applications; in other words, bandwidth can be represented by tokens. A single request can input or output a large number of tokens. The larger the number of tokens, the more computing resources are required.

[0033] Understandably, because the QPS mechanism can only count request frequency, while the resource consumption (such as the number of tokens processed) of a single request in a large model varies greatly, the number of requests is severely decoupled from the actual resource cost. Step S01, when receiving a large model service request, makes an admission decision based on a pre-set traffic record database, elevating the basis for admission judgment from the vague "request frequency" to the quantifiable "total historical resource consumption," overcoming the shortcomings of existing technologies that solely rely on QPS for rate limiting. This step, by querying and maintaining a traffic record database reflecting historical accumulated actual resource consumption, can accurately identify and block users or requests whose accumulated resource consumption is close to or exceeds the quota, thereby achieving effective identification and control of high resource consumption requests.

[0034] Step S02: If it is allowed to process large model service requests, select the target large model from the preset large models based on the traffic weight of each preset large model. It's important to note that the traffic weight is a dynamically calculated value for each available pre-defined large model. This value determines the probability that a new incoming request will be assigned to that pre-defined large model. The shorter the processing latency of a pre-defined large model, the higher its traffic weight. The target large model is the one ultimately selected from multiple available pre-defined large models to handle the current large model service request.

[0035] Understandably, traditional resource scheduling strategies rely on request counts or simple connection states, which fail to detect the actual computational pressure differences caused by the processing of different token quantities in the backend large models. This can easily lead to uneven load distribution among the preset large models. Step S02, after the large model service request is admitted, selects the target large model based on the traffic weight of each preset large model. By introducing traffic weight as the core scheduling parameter, the scheduling decision can dynamically respond to the actual load level of the backend large models, directing large model service requests to the large models with relatively stronger processing capabilities (or lower pressure). This optimizes the overall utilization of computing resources and alleviates the problems of local overload and global response delay caused by uneven load distribution.

[0036] Step S03: When the target large model is idle, the large model service request is allocated to the target large model for processing, the actual traffic consumed by the large model service request is recorded, and the preset traffic record database is updated based on the actual traffic consumed.

[0037] It should be noted that actual traffic consumption refers to the total traffic actually consumed by a large model service request after it has been fully processed. This data is obtained by asynchronously parsing the response content returned by the preset large model and is represented by the number of resource consumption identifiers (such as Tokens).

[0038] Understandably, after confirming that the target large model is idle, step S03 allocates the large model service request to the model for execution and records the actual traffic consumed by this request. Subsequently, the traffic record database is updated, thus constructing a closed-loop management link of execution-measurement-update. This makes the traffic record database, which serves as the core decision-making basis, a learning database that can dynamically and accurately reflect the real traffic consumption cost. This ensures that the admission judgment in step S01 and the weight calculation in step S02 can be iterated and optimized based on the latest and real system operation data, so that the decision-making of the entire management system can continuously maintain accuracy and adaptability.

[0039] In one feasible implementation, step S01, the step of determining whether to allow processing of large model service requests based on large model service requests and a preset traffic record database, includes steps A01 to A04: Step A01: Determine the corresponding user identifier based on the large model service request; It should be noted that the user identifier is an information feature extracted from the large model service request, used to uniquely identify and trace the entity that initiated the large model service request, such as a user ID.

[0040] Additionally, it should be noted that when extracting user identifiers, user identity can be identified through various conditions such as IP address, request route, and request headers. For example, user identifiers can be extracted from the source IP address of the large model service request, the request path (such as / v1 / chat / completions), or specific headers (such as X-API-Key).

[0041] Step A02: Query the user's cumulative data consumption within a preset time period in the preset data consumption record database, and compare the cumulative data consumption with the preset data limit. It should be noted that the cumulative traffic consumption is the total traffic actually consumed by all large model requests served by a user within a preset time (e.g., 1 minute), which is bound to the user's identifier. It is measured by the number of tokens, divided into total tokens, question tokens, and answer tokens. For example, user A's total large model requests in the past minute have consumed 500 tokens. The preset traffic limit is the maximum total traffic threshold allowed to be consumed within a preset time.

[0042] Step A03: If the accumulated traffic consumption is less than the preset traffic limit, determine that the large model service request can be processed. Step A04: If the accumulated traffic consumption exceeds the preset traffic limit, it is determined that large model service requests are not allowed to be processed.

[0043] For example, if user A has consumed 5200 tokens in the past minute, when a new large model service request arrives, the query shows that the cumulative consumption is 5200 tokens. If the result is compared with the upper limit of 5000 tokens, and 5200 > 5000, then it is determined that this request cannot be processed and the processing link is disconnected.

[0044] In this implementation, by extracting user identifiers from large-scale service requests, a foundation for personalized management is laid. By querying the user's cumulative traffic consumption within a preset period and comparing it with a preset upper limit, the traditional extensive model of QPS rate limiting, which only counts the number of requests, is changed. The basis for admission decisions is changed from request frequency to measurable cumulative resource consumption (token quantity). Based on the comparison results, the judgment is executed, which can accurately identify and block user requests whose historical resource consumption has approached or exceeded the quota, thereby effectively managing requests with huge single resource consumption and high cumulative cost.

[0045] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2In step S02, the step of selecting the target large model from the preset large models based on the traffic weights of each preset large model includes steps S11 to S14: Step S11: Calculate the traffic weights of each preset large model in the current period and generate random numbers, where the random numbers are integers from zero to the range corresponding to the sum of each traffic weight; It should be noted that the current period is a fixed time window for sampling performance indicators and recalculating traffic weights based on a preset collection frequency (e.g., every 2 seconds). Within this period, the traffic weights of each preset large model remain unchanged, ensuring that load balancing decisions are based on comparable data within the same time period. The random number is an integer uniformly generated within the interval [0, the sum of all traffic weights]. Its randomness ensures that traffic is distributed according to weight proportions, avoiding allocation deviations caused by fixed order or state.

[0046] Step S12: Combine the preset large models into a large model sequence and traverse the large model sequence. For any candidate large model in the large model sequence, determine whether the random number is less than the traffic weight corresponding to the candidate large model. It should be noted that the large model sequence refers to a list of all preset large models arranged in a preset order (such as configuration order or preset large model identifier sorting). This sequence defines a fixed order for the algorithm to traverse, ensuring that each large model is evaluated sequentially in each selection. The candidate large model is the currently detected large model during the traversal of the large model sequence. The initial candidate large model is the first preset large model in the large model sequence. If it is not selected, the next preset large model is used as the new candidate large model.

[0047] Step S13: If the random number is less than the traffic weight corresponding to the candidate large model, the candidate large model is taken as the target large model for the current period, and the traversal ends. Step S14: If the random number is not less than the traffic weight corresponding to the candidate large model, subtract the traffic weight corresponding to the candidate large model from the random number to obtain a new random number. Take the next large model of the candidate large model as the new candidate large model, and return to execute the step of judging whether the random number is less than the traffic weight corresponding to the candidate large model, until the target large model is found or the large model sequence is traversed.

[0048] For example, suppose there are three candidate large models A, B, and C, with corresponding traffic weights of 50, 30, and 20 (totaling 100), respectively. The system generates a random integer, such as 42. The traversal starts with candidate large model A: the random number 42 is less than the weight of A, 50, so candidate large model A is immediately selected as the target large model, and the traversal ends. If the generated random number is 75, then: first, it is compared with the weight (50) of candidate large model A. 75 is not less than 50, so the random number is updated to 75-50=25 and moved to candidate large model B; then, it is compared with the weight (30) of candidate large model B. 25 is less than 30, so candidate large model B is selected. This process ensures that the probability of candidate large models A, B, and C being selected is consistent with their traffic weight ratios (50%, 30%, 20%), thereby directing more traffic to the candidate large model with less pressure.

[0049] In this implementation, the traffic weights of each preset large model are calculated and quantified into comparable values. At the same time, a random number within the total weight range is generated, and the large model sequence is sequentially traversed to accumulate and compare weights. This ensures that the probability of each candidate large model being selected is proportional to its weight. The scheduling decision for new requests is transformed from the traditional simple polling or random allocation based on the number of requests to an intelligent probability distribution that is dynamically associated with the real-time traffic weights of candidate large models. This solves the problem of uneven pressure and low resource utilization among candidate large models caused by the inability of traditional load balancing strategies to perceive the real computing load. It realizes dynamic traffic distribution based on the real load state and optimizes the overall system throughput and response performance.

[0050] In one feasible implementation, step S11, which calculates the traffic weights of each preset large model in the current period, includes steps B01 to B03: Step B01: Obtain the cumulative delay time and cumulative resource consumption of each preset large model in the historical processing according to the preset collection cycle; It should be noted that the preset collection cycle is a pre-configured fixed time interval used to periodically collect performance metrics of each preset large model, for example, two seconds per cycle. The historical processing process is the complete process of each preset large model handling all client requests from the start of its service until the current collection time, encompassing all request records received, processed, and responded to during this period. Cumulative latency refers to the total time consumed by each preset large model in the historical processing process, from the start of processing each request to returning a complete response, reflecting the total working time of the corresponding preset large model. Cumulative resource consumption is a measure of the total computing resources consumed by each preset large model in the historical processing process to handle all completed requests, i.e., the total number of tokens processed.

[0051] Step B02: For any one of the preset large models, calculate the incremental average latency of the preset large model in the current period based on the total latency of the preset large model up to the previous period, the total resource consumption of the preset large model up to the previous period, the cumulative latency of the preset large model, and the cumulative resource consumption of the preset large model. It should be noted that the total latency and total resource consumption up to the previous period refer to the cumulative latency and cumulative resource consumption recorded by the preset large model at the end of the previous period in the current period, respectively. The incremental average latency is based on the difference between two adjacent acquisition periods and is used to characterize the performance of the preset large model in the current period.

[0052] For example, the formula for calculating the incremental average delay is:

[0053] in, For the first The incremental average latency of a pre-defined large model, The total delay up to the previous period. To accumulate delay time, This represents the total resource consumption up to the previous period. This is to accumulate resource consumption. The incremental average latency reflects the real-time processing speed and pressure level of the preset large model within the most recent time window. The smaller the value, the higher the processing efficiency and the lighter the pressure.

[0054] Additionally, it should be noted that by calculating the incremental average latency instead of using the global cumulative average, the influence of outdated historical data can be eliminated, capturing the true fluctuations in the performance of each preset large model within the most recent sampling period.

[0055] Step B03: Determine the traffic weight of the preset large model in the current period based on the incremental average delay, wherein the incremental average delay is inversely proportional to the traffic weight.

[0056] In this embodiment, by periodically collecting accumulated latency and resource consumption, the abstract computational pressure is transformed into quantifiable historical time-series data. By calculating the incremental average latency, the interference of long-term historical accumulated data on the current instantaneous state is eliminated, and the true unit processing efficiency of each preset large model in the latest period is accurately captured. The traffic weight is determined based on the inverse ratio of this incremental latency, so that the weight value can directly and sensitively reflect the current processing capacity of each preset large model. The decision basis for load balancing is changed from the number of requests that is unrelated to the actual computational load to a dynamic performance indicator that is directly linked to the instantaneous processing efficiency of the preset large model. This solves the problem of uneven load caused by the inability of traditional strategies to perceive the actual pressure.

[0057] In one feasible implementation, step B03, the step of determining the traffic weight of the preset large model in the current period based on the incremental average delay, includes steps B11 to B13: Step B11: Obtain the incremental average delay corresponding to each preset large model, calculate the reciprocal of each incremental average delay, and obtain the basic weight values. It should be noted that the base weight value is the reciprocal of the incremental average latency, used to characterize the processing efficiency of the preset large model in the latest statistical period. The lower the incremental average latency (the higher the efficiency), the larger its reciprocal, i.e., the larger the base weight value. Therefore, the base weight value directly and positively reflects the processing capability advantage of the preset model in the most recent time period. For example, if the incremental average latency of model A is 0.2 seconds, its base weight value is 5; and the latency of model B is 0.5 seconds, its base weight value is 2. This indicates that the processing efficiency of model A in the current period is quantified as a value of 5, which is higher than the value of 2 for model B.

[0058] Step B12: Sum the values ​​of each basic weight to obtain the total global basic weight; It should be noted that the global basic weight sum is the arithmetic sum of the basic weight values ​​of all preset large models participating in load balancing within the same period.

[0059] Step B13: The ratio of the basic weight value corresponding to the preset large model to the sum of the global basic weights is used as the traffic weight corresponding to the preset large model.

[0060] For example, the formula for calculating traffic weight is:

[0061] in, For the first Traffic weights for a pre-defined large model For the first The basic weight values ​​corresponding to the preset large model. That is, the first The inverse of the incremental average latency corresponding to each preset large model. This is the sum of the global basic weights. .

[0062] In this embodiment, by taking the reciprocal, the latency of the characterization processing time is converted into a basic weight representing the efficiency, realizing an intuitive mapping that the lower the latency, the higher the weight. Through summation and ratio operations, all basic weights are normalized to ensure that the sum of each weight is a fixed value, thereby forming a stable and comparable probability distribution system. This allows the subsequent weighted random scheduling to accurately allocate traffic based on the relative performance differences of each preset large model, solving the problem of uneven load caused by the inability of traditional request-based strategies to perceive and respond to differences in real computing pressure.

[0063] In one feasible implementation, before step S03, which involves allocating the large model service request to the target large model for processing when the target large model is idle, steps B21 to B25 are included: Step B21: Determine whether there are any idle connections in the connection pool of the target large model; It's important to note that the connection pool is a collection of reusable network connections maintained by the system for each pre-defined large model. This is a software architectural component used to cache and manage established connections to a specific backend pre-defined large model. The purpose is to avoid the overhead of repeatedly establishing connections for each new request (such as TCP three-way handshakes and TLS negotiation), thereby significantly improving processing efficiency. Idle connections refer to connections in the connection pool that are currently not occupied by any requests and are in an available state. When a large model service request needs to be processed, the system prioritizes reusing existing idle connections in the connection pool.

[0064] Step B22: If there are idle connections in the connection pool, determine that the target large model is idle. The idle connection is used to transmit large model service requests. Step B23: If there are no idle connections in the connection pool, determine whether the number of connections in the connection pool has reached the preset connection limit. It should be noted that the preset connection limit is the maximum allowed number of connections in the connection pool corresponding to each preset large language model. This is used to prevent a single preset large language model from occupying system connection resources without limit. For example, even if a large number of requests are directed to model A, once the number of connections in its connection pool reaches the preset threshold (e.g., 100), the system will no longer create new connections for it, and subsequent requests will be rejected. This ensures that other models (such as model B and model C) still have sufficient connection resources available, avoiding the risk of global service paralysis due to overload of a single model.

[0065] Step B24: If the number of connections has not reached the preset connection limit, determine that the target large model is idle and create an idle connection in the connection pool. Step B25: If the number of connections reaches the preset connection limit, determine that the target large model is not idle and refuse to accept large model service requests.

[0066] It should be noted that, for example, when a large model service request needs to be routed to the target large model, the system first accesses its corresponding connection pool. If there is a connection marked as idle in the connection pool, the connection is directly reused for request transmission. If there is no idle connection in the connection pool, the current number of connections in the connection pool is queried and compared with the preset connection limit. If the number of connections is less than the preset connection limit, a new connection is created, placed in the connection pool and marked as idle, and then used to transmit the large model service request. If the preset connection limit has been reached, it is determined that the target large model has no available connection resources, the large model service request is rejected, and an error response is returned to the client.

[0067] In this implementation, by comparing the connection pool status with the preset upper limit, idle connections are reused first to improve efficiency. When there are insufficient connections but the upper limit has not been reached, new connections are created elastically. When the upper limit has been reached, requests are rejected decisively. Strict hard limits on the number of concurrent connections are implemented for each large model, realizing resource isolation between different models. This solves the risk that traditional gateways, due to the lack of model connection management, may cause a single high-load model to monopolize connection resources and cause global service instability.

[0068] In one feasible implementation, step S03, recording the actual traffic consumed by the large model service request, includes steps B31-B34: Step B31: Obtain the response data stream returned by the target large model processing large model service request; It should be noted that the response data stream refers to the raw data sequence returned to the gateway after the target large model has processed the large model service request. It carries the processing result of the large model service request and includes response headers and response bodies. For example, for a text generation request, this data stream contains the text content generated by the model and related metadata information.

[0069] Step B32: Identify the format identification information in the response data stream, and identify the response type of the response data stream based on the format identification information; It should be noted that the response type refers to the organization and transmission format of the response data stream, which can be determined based on the identification information in the response data stream (such as the Content-Type field in the HTTP response header).

[0070] Step B33: In the case of a non-streaming response, information representing traffic consumption is synchronously extracted from the preset structured fields of the response data stream as the actual traffic consumed by the large model service request. It's important to note that a non-streaming response is a response type characterized by the target large model encapsulating and returning the complete response result all at once after processing the entire large model service request. The gateway can synchronously read and parse the entire response body. Predefined structured fields are pre-agreed key-value pair paths or locations within the non-streaming response's data body used to store specific information. For example, in a JSON response body, there might be a field named "usage" containing subfields such as "prompt_tokens" and "completion_tokens." These subfields are the predefined structured fields, and the gateway extracts information according to these agreed-upon paths. Traffic consumption refers to the quantifiable metrics of the actual computing resources consumed in processing large model service requests, which can include the number of input tokens, the number of output tokens, and the total number of tokens.

[0071] Step B34: In the case of a streaming response, when processing continuous data segments of the response data stream, the target data segment carrying the total resource consumption identifier is searched from the continuous data segments, and the information representing the traffic consumption is extracted from the target data segment as the actual traffic consumption of the large model service request.

[0072] It's important to note that streaming response is another type of response. Its characteristic is that during the processing of large-scale service requests, the target large model continuously returns partial results (or intermediate results) to the client in batches as data fragments until completion. Continuous data fragments refer to relatively independent blocks of data continuously returned by the backend target large model during the streaming response process. Total resource consumption identifier information refers to metadata within certain data fragments of the streaming response used to summarize and report the total resource consumption (i.e., traffic consumption) of this request. For example, a JSON object containing a "usage" field recording complete token usage. The target data fragment refers to a specific fragment within the many continuous data fragments of the streaming response that specifically carries the total resource consumption identifier information. When processing the streaming response, the gateway needs to continuously parse the continuous data fragments and identify this target data fragment in order to extract the final resource consumption.

[0073] For example, to aid in understanding the technical concept or principles of this application, please refer to Figure 3 , Figure 3 Example graphs showing resource consumption data acquisition are provided. Figure 3As shown, the system first receives the response body. It then creates a BackupBodyReader instance to simultaneously write to the cache and store the BackupBodyReader in the CmbLImProxyReader context. This ensures that the original data stream continues to be transmitted to the client without interruption, guaranteeing a good user experience. After the response is completed, the system triggers an asynchronous execution process: it retrieves the BackupBodyReader from the CmbLImProxyReader, reads the cached response body, and then parses and extracts the Token information. Subsequently, the system stores the Token information in the IlmTokens context, which is then used by other plugins (such as the rate limiting module) to retrieve Token information from IlmTokens. This decouples response transmission from resource consumption information collection, accurately assessing backend resource consumption while completely avoiding its impact on client response latency.

[0074] In this embodiment, by adaptively identifying the response type, synchronous or streaming processing strategies are adopted accordingly to extract structured resource consumption information from specific fields or data fragments in the response body. This breaks through the limitation of traditional gateways that only record the number of requests but cannot know the internal resource cost of the request, and realizes accurate and lossless collection of the actual computational consumption (such as the number of tokens) of a single request for a large model service.

[0075] For example, to aid in understanding the technical concept or principles of this application, please refer to Figure 4 , Figure 4 A flowchart illustrating the overall architecture of the traffic management method is provided. Figure 4 As shown, the process is clearly divided into a request phase and a response phase. In the request phase, requests from clients first reach the rate limiting module for resource consumption-based admission judgment. If approved, the request is passed to the routing module, which dynamically selects the target backend model based on the real-time pressure of the pre-set backend model and forwards the request. In the response phase, after the selected backend model processes the request, it generates a response and returns it to the response module. The response module is responsible for asynchronously parsing the response content to obtain the actual resource consumption data. This data is then fed back to update the system status. Finally, the complete response result is returned to the client along the link, thus forming a complete closed loop from request admission, intelligent routing, backend processing to response parsing and feedback.

[0076] Specifically, such as Figure 5 As shown, Figure 5A flowchart for the rate limiting module is provided. In the request phase, the system first receives a large model service request, then extracts the user identifier from the request and queries the user's cumulative traffic consumption based on this identifier. Next, the query result is compared with a preset traffic limit to determine if it exceeds the threshold. If not, the system proceeds to call the preset large model from the backend; if it exceeds the threshold, the response is stopped immediately, and no further processing is performed. In the response phase, the system obtains information representing traffic consumption from the response data stream and extracts the specific value of the actual traffic consumed. This value is then used to update the preset traffic record database to ensure that the cumulative records accurately reflect the latest consumption. Finally, a response is returned to the client to conclude the process.

[0077] Specifically, such as Figure 6 As shown, Figure 6 A flowchart for the routing module is provided. First, the incremental average latency of each preset large model is periodically calculated to prepare real-time performance metrics for subsequent decisions. When a request reaches the routing stage, the system immediately calculates the traffic weight of each preset large model based on the incremental average latency and selects the target large model from among them according to the traffic weight. After selecting the target large model, the system checks if the target large model is idle. If the result is no, the request is directly rejected; if the result is yes, it attempts to reuse an idle connection. If no reusable idle connection is found, it further checks if the number of connections has reached the preset connection limit. If the determination is yes, the request is also rejected; if the determination is no, a new idle connection is created to handle the current request.

[0078] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the traffic management method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0079] This application also provides a traffic management system; please refer to [reference needed]. Figure 7 The traffic management system includes: The request and response module 10 is used to receive large model service requests and determine whether to allow processing of large model service requests based on the large model service requests and the preset traffic record database. The model filtering module 20 is used to select the target large model from each preset large model based on the traffic weight of each preset large model, while allowing the processing of large model service requests; The processing feedback module 30 is used to allocate large model service requests to the target large model for processing when the target large model is idle, record the actual traffic consumed by the large model service requests, and update the preset traffic record database based on the actual traffic consumed.

[0080] The traffic management system provided in this application, employing the traffic management method described in the above embodiments, can solve the technical problem of poor management performance in current gateway traffic management based on request counts. Compared with the prior art, the beneficial effects of the traffic management system provided in this application are the same as those of the traffic management method provided in the above embodiments, and other technical features in the traffic management system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0081] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the traffic management method in Embodiment 1 above.

[0082] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, and PADs (Portable Application Description: Tablet computers), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0083] like Figure 8As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0084] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0085] The electronic device provided in this application, employing the traffic management method described in the above embodiments, can solve the technical problem of poor management performance in current gateway traffic management based on request counts. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the traffic management method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0086] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0087] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0088] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the traffic management method in the above embodiments.

[0089] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0090] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0091] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the traffic management device: receives a large model service request and determines whether to allow processing the large model service request based on the large model service request and a preset traffic record database; if processing the large model service request is allowed, selects a target large model from the preset large models based on the traffic weight of each preset large model; if the target large model is idle, allocates the large model service request to the target large model for processing, records the actual traffic consumed by the large model service request, and updates the preset traffic record database based on the actual traffic consumed.

[0092] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0095] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described traffic management method, which can solve the technical problem of poor management effect in current gateway traffic management based on the number of requests. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the traffic management method provided in the above embodiments, and will not be repeated here.

[0096] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the traffic management method described above.

[0097] The computer program product provided in this application can solve the technical problem of poor management effect in current gateway traffic management based on the number of requests. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the traffic management method provided in the above embodiments, and will not be repeated here.

[0098] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A traffic management method, characterized in that, The traffic management method includes: Receive large model service requests and determine whether to allow processing the large model service requests based on the large model service requests and a preset traffic record database; If the large model service request is allowed to be processed, the target large model is selected from the preset large models based on the traffic weight of each preset large model; When the target large model is idle, the large model service request is allocated to the target large model for processing, the actual traffic consumed by the large model service request is recorded, and the preset traffic record database is updated based on the actual traffic consumed.

2. The traffic management method as described in claim 1, characterized in that, The step of determining whether to allow processing the large model service request based on the large model service request and the preset traffic record database includes: The corresponding user identifier is determined based on the large model service request. Query the user's cumulative data consumption within a preset time period in the preset data consumption record database, and compare the cumulative data consumption with the preset data limit. If the cumulative traffic consumption is less than the preset traffic limit, it is determined that the large model service request can be processed. If the accumulated traffic consumption exceeds the preset traffic limit, it is determined that the large model service request cannot be processed.

3. The traffic management method as described in claim 1, characterized in that, The step of selecting the target large model from the preset large models based on the traffic weights of each preset large model includes: Calculate the traffic weights of each preset large model in the current period and generate random numbers, wherein the random numbers are integers from zero to the range corresponding to the sum of the traffic weights; The preset large models are combined into a large model sequence and the large model sequence is traversed. For any candidate large model in the large model sequence, it is determined whether the random number is less than the traffic weight corresponding to the candidate large model. If the random number is less than the traffic weight corresponding to the candidate large model, the candidate large model is taken as the target large model for the current period, and the traversal ends. If the random number is not less than the traffic weight corresponding to the candidate large model, then the random number is subtracted from the traffic weight corresponding to the candidate large model to obtain a new random number. The next large model of the candidate large model is taken as the new candidate large model, and the step of determining whether the random number is less than the traffic weight corresponding to the candidate large model is returned to be executed until the target large model is found or the sequence of large models is traversed.

4. The traffic management method as described in claim 3, characterized in that, The steps for calculating the traffic weights of each preset large model in the current period include: According to the preset collection cycle, obtain the cumulative delay time and cumulative resource consumption of each preset large model in the historical processing process; For any of the preset large models, the incremental average latency of the preset large model in the current period is calculated based on the total latency of the preset large model up to the previous period, the total resource consumption of the preset large model up to the previous period, the cumulative latency of the preset large model, and the cumulative resource consumption of the preset large model. The traffic weight of the preset large model in the current period is determined based on the incremental average delay, wherein the incremental average delay is inversely proportional to the traffic weight.

5. The traffic management method as described in claim 4, characterized in that, The step of determining the traffic weight of the preset large model in the current period based on the incremental average delay includes: Obtain the incremental average delay corresponding to each preset large model, calculate the reciprocal of each incremental average delay, and obtain each basic weight value; By summing up the aforementioned basic weight values, the total global basic weight is obtained; The ratio of the basic weight value corresponding to the preset large model to the sum of the global basic weights is used as the traffic weight corresponding to the preset large model.

6. The traffic management method as described in claim 1, characterized in that, Before the step of allocating the large model service request to the target large model for processing when the target large model is idle, the method further includes: Determine whether there are any idle connections in the connection pool of the target large model; If there are idle connections in the connection pool, the target large model is determined to be idle, wherein the idle connection is used to transmit the large model service request; If there are no idle connections in the connection pool, determine whether the number of connections in the connection pool has reached the preset connection limit. If the number of connections has not reached the preset connection limit, the target large model is determined to be idle, and the idle connection is created in the connection pool. If the number of connections reaches the preset connection limit, it is determined that the target large model is not idle, and the large model service request is rejected.

7. The traffic management method as described in claim 1, characterized in that, The step of recording the actual traffic consumed by the large model service request includes: Obtain the response data stream returned by the target large model in processing the large model service request; Identify the format identification information in the response data stream, and identify the response type of the response data stream based on the format identification information; If the response type is a non-streaming response, information representing traffic consumption is synchronously extracted from the preset structured fields of the response data stream as the actual traffic consumed by the large model service request. When the response type is a streaming response, when processing continuous data segments of the response data stream, a target data segment carrying total resource consumption identification information is searched from the continuous data segments, and information representing traffic consumption is extracted from the target data segment as the actual traffic consumption of the large model service request.

8. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the traffic management method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the traffic management method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the traffic management method as described in any one of claims 1 to 7.

Images